docs: record Phase 21 (windowed streaming) as landed; note Direction A to B pivot

Move Phase 21 from PLAN to COMPLETED with the as-built record, and annotate
the spec that Direction B shipped after WASM fetch buffering defeated A.
This commit is contained in:
daniel-c-harvey
2026-06-24 16:05:30 -04:00
parent c1e6930c70
commit 036ee1f78e
3 changed files with 47 additions and 113 deletions
+26
View File
@@ -6,6 +6,32 @@ Newest entries at the top. Group by phase/wave header (mirroring `PLAN.md` / `CM
--- ---
## Phase 21 — Windowed Streaming Buffer (landed 2026-06-24)
**Landed:** 2026-06-24 on `streaming-overhaul`.
- **What:** Bounded client memory for long audio streams. Playing a 1 GB+ DJ Mix (the Phase 9 `Mix` medium — a single long track) no longer accumulates the full decoded PCM in the browser; instead the player holds only a sliding forward window and discards what has already played. Four waves, all landed. Public listener site only (`DeepDrftPublic.Client` player stack + `DeepDrftPublic` TypeScript audio interop); no CMS, no API endpoint, no schema change.
- **Why:** The `PlaybackScheduler` held an `AudioBuffer[]` it never evicted — both decode paths (`StreamDecoder`/`IFormatDecoder` for WAV/MP3/FLAC, and `OggDemuxer``OpusStreamDecoder` for Opus) pushed into it without limit. Decoded PCM is larger than the source (Web Audio is 32-bit float per sample/channel — a 16-bit stereo WAV roughly doubles once decoded; a low-data Opus mix decodes to the same float footprint regardless of how few compressed bytes arrived), so a 1 GB WAV or Opus mix could accumulate ~2 GB of retained float in the browser. The Opus path also had a second upstream accumulation locus: the WebCodecs `AudioDecoder` work queue and `decodedQueue: AudioData[]`.
- **Shape (by wave):**
- **21.1 — Partial eviction in `PlaybackScheduler` (cold-start; load-bearing; shared by both paths).** Drop already-played buffers while keeping position/index/time-anchor bookkeeping exact against a buffer array that no longer begins at absolute time 0. Written once, serves both decode paths (they `addBuffer` identically). This is the hardest correctness work in the phase. No refill yet.
- **21.2 — Back-pressure (one fill signal, two throttle sites).** Bound the unplayed region by stopping production above a high-water mark and resuming below low-water, driven by `PlaybackScheduler.evaluateProductionPause()` — a single shared scheduler-fill signal (OQ6/OQ7 resolved: shared controller, per-path hook). **21.2a**`StreamingAudioPlayerService` stops fetching the next segment above high-water, polling `evaluateProductionPause()` at 100 ms cadence until the fill drains below low-water (serves both paths). **21.2b** — the Opus demux/decode feed is additionally stopped when the same signal is set, so the WebCodecs decode queue and `decodedQueue` do not balloon behind a paused segment loop (Opus only; no WAV analogue). Together with 21.1 this bounds both the played and unplayed sides on both formats.
- **21.3 — Seek-back-past-window refill + clean failure recovery.** When a backward seek lands earlier than the retained tail, the existing seek-beyond-buffer path is reused pointed at the earlier offset (whichever resolver the active path ships: `IFormatDecoder`/`StreamDecoder` for WAV; the live `resolveOpusByteOffset` + `OpusStreamDecoder.reinitializeForRangeContinuation` for Opus). Minimal AC6 refill-failure handling added: `RecoverFromFailedRefill` clears the scheduler and surfaces a user-visible error rather than leaving the player wedged.
- **21.4 — Validation + Direction A→B pivot (the critical wave — see as-built divergence below).** Network-memory bounding confirmed in Daniel's browser run: the segmented approach delivers ~4 MB segments pacing with playback, with the browser holding ~one segment of raw bytes rather than the full artifact.
- **As-built divergence — Direction A→B pivot (important).** The spec recommended **Direction A** (sliding window on one open-ended forward stream, relying on pausing `ReadAsync` / the segment loop to backpressure the socket) with **Direction B** (discrete bounded `Range: bytes=start-end` segments, each fetched only when the scheduler drains below low-water) held as the documented fallback "if Direction A's back-pressure proves leaky in practice."
**21.4 browser validation proved it leaky.** On Blazor WASM, the browser `fetch` API buffers the entire HTTP response body regardless of how slowly the application reads it: pausing reads bounded the *decode* but not the *network download*, so the entire ~970 MB body accumulated in browser memory even though the application only decoded a window of it. Direction A's core assumption — that pausing `ReadAsync` closes the TCP flow-control window before the browser caches the whole body — does not hold in the WASM `fetch` runtime.
**We pivoted to Direction B.** The forward stream now issues sequential bounded `bytes=cursor-(cursor+SegmentSizeBytes-1)` Range requests (`SegmentSizeBytes = 4 MB`), each fetched only after the scheduler drains below low-water — via `RunSegmentedStreamAsync` in `StreamingAudioPlayerService`. Because each 4 MB request is fully consumed before the next is issued, the browser holds at most ~one segment of raw bytes regardless of file size. The decode-side windowing (21.1/21.2) pairs with it: the segment loop's segment-gate replaces the raw `ReadAsync` pause as the production throttle (21.2a), and the Opus decode-ahead throttle (21.2b) hooks the same `evaluateProductionPause()` signal. Seek and refill converge on the same segmented loop. Direction B is recorded as the shipped approach; Direction A is recorded as tried-in-validation and found insufficient for the WASM `fetch` runtime.
- **What Phase 21 does NOT include:** the full AC matrix re-run (Opus seek-storm, visualizer, rapid-seek concurrency under the new segmented loop) beyond the network-memory bounding confirmed in 21.4. Those acceptance criteria remain the validation baseline for any follow-on work touching this seam.
- **Design memo:** `product-notes/phase-21-windowed-streaming-buffer.md` (note: the spec recommended Direction A; the as-built pivot to Direction B is annotated at the top of that doc and in §3.2/§3.3).
---
## Phase 18 — Opus Low-Data Streaming (landed 2026-06-23) ## Phase 18 — Opus Low-Data Streaming (landed 2026-06-23)
_Note: Two distinct efforts share the "Phase 18" label — phase numbers are organisational, not sequential. See also **Phase 18 — Theme / Dark-Mode Remediation (landed 2026-06-19)** below._ _Note: Two distinct efforts share the "Phase 18" label — phase numbers are organisational, not sequential. See also **Phase 18 — Theme / Dark-Mode Remediation (landed 2026-06-19)** below._
+2 -109
View File
@@ -449,116 +449,9 @@ See `COMPLETED.md` for Phase 18 — dual-format lossless + Opus delivery, includ
--- ---
## Phase 21 — Windowed Streaming Buffer (bounded client memory for long streams) ## Phase 21 — Windowed Streaming Buffer (Completed)
Bound the **client memory** a playing track consumes to a small, configurable forward window — See `COMPLETED.md` for Phase 21 — bounded client memory for long streams, including the as-built Direction A→B pivot — which landed on `streaming-overhaul` (2026-06-24).
**independent of total stream length** — so a 1 GB+ DJ MIX (Phase 9 `Mix` medium: a single long track)
plays without the whole decoded PCM accumulating in the browser. **Public listener site only**
(`DeepDrftPublic.Client` player stack + `DeepDrftPublic` TypeScript audio interop); no CMS, no API
endpoint, no schema change.
**Phase 18 (Opus Low-Data Streaming) has landed — Phase 21 is the next pickup, reconciled to the as-built
two-decode-path reality.** Phase 18 left **two** decode paths feeding the one `PlaybackScheduler`:
WAV/MP3/FLAC via `StreamDecoder`/`IFormatDecoder`, and Opus via a **WebCodecs `AudioDecoder`** pipeline
(`OggDemuxer` → `OpusStreamDecoder`, the `IStreamingDecoder` seam — *not* `IFormatDecoder`; per-segment
`decodeAudioData` was tried and replaced). Windowing must bound **both**. The accurate index-driven Opus
seek the original spec assumed Phase 21 would build is **already live** (`resolveOpusByteOffset` over the
precomputed seek index → Range fetch → `reinitializeForRangeContinuation` with frame-accurate lead-trim);
Phase 21 **reuses** it for window-miss refills rather than building it. Opus seek is VBR-safe and
**accurate**, not approximate (the earlier "approximate page interpolation" framing is corrected).
The network path already streams in adaptive 1664 KB chunks. The accumulation is on the **decode
side**, and now has two faces. The shared one: `PlaybackScheduler` holds an `AudioBuffer[]` it **never
evicts** — both paths `addBuffer` into it, nothing is removed. Decoded PCM is larger than the source
(Web Audio is 32-bit float per sample/channel — a 16-bit stereo WAV roughly doubles once decoded), so a
1 GB WAV becomes ~2 GB of retained float; **a low-data Opus mix decodes to the same ~2 GB once played**,
so its small transfer does not spare it. The Opus-only second face: the WebCodecs decode queue +
`decodedQueue` accumulate upstream of the scheduler too. The fix: hold only a sliding forward window plus
a small back-retain, discard already-played buffers, and refill on demand — with back-pressure on the C#
read loop (both paths) **and** on the Opus demux/decode feed (Opus only).
**Architectural spine — a sliding window keyed on playback position, built as a generalization of the
landed seek-beyond-buffer paths.** The Phase 4 HTTP `Range: bytes=X-` → 206 primitive already does every
plumbing primitive the window needs, with a WAV branch and an Opus branch, both live: discard-buffers-keep-
offset via `clearForSeek`/`setPlaybackOffset` (shared); fetch-from-offset via `TrackMediaClient` (shared,
now with `?format=`); decode-header-less-body via `StreamDecoder.reinitializeForRangeContinuation` (WAV) /
`OpusStreamDecoder.reinitializeForRangeContinuation` (Opus); time→byte via `IFormatDecoder.calculateByteOffset`
(WAV) / `resolveOpusByteOffset` over the seek index (Opus) — just triggered manually and one-shot. The
genuinely new mechanisms: **partial eviction** on the shared scheduler (one implementation, both paths),
and **back-pressure** — on the C# read loop (both paths) **and additionally on the Opus WebCodecs
decode-ahead** (`decodeQueueSize` + `decodedQueue`, Opus only, since throttling the socket alone doesn't
bound the async decoder's queues). Recommended **Direction A** (sliding window on the existing single
forward stream); **Direction B** (discrete Range-fetched segments — the HLS/DASH/MSE-eviction analogue)
held as the documented fallback; **Direction C** (adopt MSE) **rejected (OQ5 = NO, Daniel 2026-06-23)** —
the bespoke Web Audio graph is a deliberate long-term commitment, and the compressed-delivery move that
would have justified MSE was met instead by **Phase 18 (Opus) feeding the same bespoke graph** through the
WebCodecs `IStreamingDecoder` seam. Direction A is the permanent destination, not a stopgap MSE would
retire.
**Invariants that must hold (the §3.5 seam contract).** Reuse each path's Range/seek machinery, don't fork
it; playback-start latency at parity; neither decoder seam's contract forked — eviction is shared at the
scheduler (zero format branches), back-pressure is seam-aware (the one place the two paths diverge);
read-only playback (no new control); the single-writer decoder discipline holds for **both** decoders
(stricter for the stateful Opus `AudioDecoder` — a stale `push` racing a reset+reconfigure corrupts
inter-frame state). The **Mix visualizer is provably unaffected** — it renders from the preprocessed
per-track high-res datum (Phase 10/12), never from live decoded PCM, so evicting played buffers cannot
starve it. The 1 GB mix is both the canonical case *and* the proof the eviction is safe.
**Interaction with deferred Phase 1 features (same seam):** windowing should land **before** preload
(1.3) — it makes preload of long tracks memory-safe by construction (a staged next-track decoder inherits
the bounded scheduler); it makes crossfade (1.4) between two long mixes affordable (the overlap doubles
the *window*, not the track); it adds a minor "don't evict the final window before the gapless boundary"
care point for 1.5. It **enlarges the error surface** (1.6): windowed refill issues mid-stream fetches
the listener didn't initiate, one of which can fail deep into a 1 GB mix — so the *cheap* half of 1.6
(clean refill-failure handling, no wedged player) is folded into this phase's acceptance criteria, not
left fully to 1.6.
Full design, the three directions with SOLID/road-not-taken rationale, use cases, acceptance criteria,
the open-question set, and the wave decomposition: `product-notes/phase-21-windowed-streaming-buffer.md`.
Sequenced as four waves. `21.1 → 21.2 → 21.3`, with `21.4` validating the whole. **21.1 is the cold-start
prerequisite and the load-bearing change** — independent of the open questions (window *sizes* are
parameters fed in later).
Decomposition is **by concern** (eviction → back-pressure → seek-back refill → validate), not by format —
eviction is genuinely shared, so a path-split would duplicate the hardest work; the one path-divergent
concern (back-pressure) carries a two-track split *inside* its wave.
- **21.1 — Partial eviction in `PlaybackScheduler` (cold-start; load-bearing; SHARED by both paths).** Drop
already-played buffers while keeping the position/index/time-anchor bookkeeping exact against a buffer
array that no longer begins at absolute time 0 (today `getCurrentPosition`/`playFromPosition`/the schedule
loop all assume `buffers[0]` is the track start). The hardest correctness work in the phase. Written once,
serves both decode paths (they `addBuffer` identically). No refill yet. **Independent of the open
questions — can begin immediately.**
- **21.2 — Back-pressure (two tracks, one fill signal).** Bound the unplayed region by throttling
production above a high-water mark and resuming below low-water, driven by the scheduler's
decoded-lookahead fill. **21.2a** — stop `ReadAsync` on the C# loop (serves both paths; *sufficient* for
WAV). **21.2b** — additionally stop the Opus demux/decode feed so the WebCodecs decode queue +
`decodedQueue` don't balloon behind a throttled socket (Opus only; no WAV analogue). Together with 21.1
this bounds both played and unplayed sides on both formats (AC1 + AC1-Opus). Routes through the existing
single-loop cancellation discipline. **Depends on 21.1.**
- **21.3 — Seek-back-past-window refill (one concern, per-path resolver).** When a backward seek lands
earlier than the retained tail, refetch via the existing seek-beyond-buffer path pointed at the earlier
offset, using whichever resolver the active path already ships (`IFormatDecoder`/`StreamDecoder` for WAV;
the live `resolveOpusByteOffset` + `OpusStreamDecoder.reinitializeForRangeContinuation` for Opus); plus
the minimal clean refill-failure handling (the 1.6 adjacency). Mostly reuse of the landed seek paths.
**Depends on 21.1 + 21.2.**
- **21.4 — Validation against the 1 GB target, BOTH formats (acceptance).** Memory profiling (bounded under
1 GB as WAV *and* as Opus, plus the Opus upstream queues), latency parity, edge-to-edge playback, the
seek matrix, induced refill failure, visualizer-running, rapid-seek concurrency (incl. an Opus
seek-storm). Largely measurement; breaks are tuning fixes in 21.1's anchor math, 21.2's water-marks, or
21.2b's Opus decode-ahead bound. **Depends on 21.121.3.**
**Dependency shape:** `21.1 → 21.2 → 21.3 → 21.4`; 21.1 is the only cold-start wave. **Phase-level
prerequisite: Phase 18 (Opus) has landed** (`COMPLETED.md`) — windowing is built against both formats.
**Open questions for Daniel (spec §6):** window-size policy axis (time-based window + memory guard —
recommended); seek-back-past-window re-buffer acceptable (recommend yes, symmetric to forward); a hard
total in-flight memory cap as a guard rail (recommend yes); window everything vs. only long tracks
(recommend everything — one path, short tracks never hit a refill). **New staff-engineer architecture
calls (spec §6):** OQ6 — one window controller for both paths or two (recommend shared controller + two
back-pressure hooks); OQ7 — drive the Opus decode-ahead bound from the single scheduler-fill signal
(recommended). **OQ5 (adopt MSE) — RESOLVED NO (Daniel 2026-06-23):** the bespoke graph stays by deliberate
choice. None block 21.1.
--- ---
@@ -1,9 +1,24 @@
# Phase 21 — Windowed Streaming Buffer (bounded client memory for long streams) # Phase 21 — Windowed Streaming Buffer (bounded client memory for long streams)
Product spec. Status: **design / framing — reconciled to as-built Phase 18 (two decode paths); Product spec. Status: **LANDED 2026-06-24 on `streaming-overhaul`.** See `COMPLETED.md` for the full
implementation-ready pending Daniel's open-question calls (OQ1OQ4 product; OQ6OQ7 staff-engineer as-built record. Author: product-designer. Date: 2026-06-23 (reconciliation pass after Phase 18 landed).
architecture).** Author: product-designer. Date: 2026-06-23 (reconciliation pass after Phase 18 landed).
**No code has been written by this doc.** > **AS-BUILT NOTE — Direction A→B pivot (2026-06-24).** This spec recommended **Direction A** (sliding
> window on one open-ended forward stream, pausing `ReadAsync`/the segment loop to backpressure the
> socket) and held **Direction B** (discrete bounded `Range: bytes=start-end` segments, §3.2) as the
> documented fallback. **21.4 browser validation proved Direction A insufficient for Blazor WASM:** the
> browser `fetch` API buffers the entire HTTP response body regardless of read pace — pausing reads
> bounded the *decode* but not the *network download*, so the whole ~970 MB body accumulated in browser
> memory even with the application decoding only a window of it. **We shipped Direction B.** The forward
> stream now issues sequential 4 MB bounded Range requests (`SegmentSizeBytes = 4 MB`), fetched via
> `RunSegmentedStreamAsync` in `StreamingAudioPlayerService`, each issued only after
> `PlaybackScheduler.evaluateProductionPause()` clears below low-water. Browser holds ~one 4 MB segment
> of raw bytes; 21.4 confirmed network-memory bounding in Daniel's browser run. The decode-side windowing
> (21.1/21.2) is unchanged and pairs with Direction B; seek/refill converge on the same segmented loop
> via `RecoverFromFailedRefill`. Direction A is recorded as tried-in-validation and found insufficient
> for the WASM `fetch` runtime. Sections §3.2–§3.3 below retain the original A vs. B vs. C analysis as
> the decision record; **Direction B is what shipped.**
Surface: **public listener site only** (`DeepDrftPublic.Client` player stack + `DeepDrftPublic` Surface: **public listener site only** (`DeepDrftPublic.Client` player stack + `DeepDrftPublic`
TypeScript audio interop). No CMS (`DeepDrftManager`) change. No data-model or schema change. The one TypeScript audio interop). No CMS (`DeepDrftManager`) change. No data-model or schema change. The one
server touch is **reuse, not new surface**: the existing `DeepDrftAPI` HTTP `Range: bytes=X-` server touch is **reuse, not new surface**: the existing `DeepDrftAPI` HTTP `Range: bytes=X-`