diff --git a/COMPLETED.md b/COMPLETED.md index b46130b..34e0ee5 100644 --- a/COMPLETED.md +++ b/COMPLETED.md @@ -6,6 +6,32 @@ Newest entries at the top. Group by phase/wave header (mirroring `PLAN.md` / `CM --- +## Phase 21 — Windowed Streaming Buffer (landed 2026-06-24) + +**Landed:** 2026-06-24 on `streaming-overhaul`. + +- **What:** Bounded client memory for long audio streams. Playing a 1 GB+ DJ Mix (the Phase 9 `Mix` medium — a single long track) no longer accumulates the full decoded PCM in the browser; instead the player holds only a sliding forward window and discards what has already played. Four waves, all landed. Public listener site only (`DeepDrftPublic.Client` player stack + `DeepDrftPublic` TypeScript audio interop); no CMS, no API endpoint, no schema change. + +- **Why:** The `PlaybackScheduler` held an `AudioBuffer[]` it never evicted — both decode paths (`StreamDecoder`/`IFormatDecoder` for WAV/MP3/FLAC, and `OggDemuxer` → `OpusStreamDecoder` for Opus) pushed into it without limit. Decoded PCM is larger than the source (Web Audio is 32-bit float per sample/channel — a 16-bit stereo WAV roughly doubles once decoded; a low-data Opus mix decodes to the same float footprint regardless of how few compressed bytes arrived), so a 1 GB WAV or Opus mix could accumulate ~2 GB of retained float in the browser. The Opus path also had a second upstream accumulation locus: the WebCodecs `AudioDecoder` work queue and `decodedQueue: AudioData[]`. + +- **Shape (by wave):** + - **21.1 — Partial eviction in `PlaybackScheduler` (cold-start; load-bearing; shared by both paths).** Drop already-played buffers while keeping position/index/time-anchor bookkeeping exact against a buffer array that no longer begins at absolute time 0. Written once, serves both decode paths (they `addBuffer` identically). This is the hardest correctness work in the phase. No refill yet. + - **21.2 — Back-pressure (one fill signal, two throttle sites).** Bound the unplayed region by stopping production above a high-water mark and resuming below low-water, driven by `PlaybackScheduler.evaluateProductionPause()` — a single shared scheduler-fill signal (OQ6/OQ7 resolved: shared controller, per-path hook). **21.2a** — `StreamingAudioPlayerService` stops fetching the next segment above high-water, polling `evaluateProductionPause()` at 100 ms cadence until the fill drains below low-water (serves both paths). **21.2b** — the Opus demux/decode feed is additionally stopped when the same signal is set, so the WebCodecs decode queue and `decodedQueue` do not balloon behind a paused segment loop (Opus only; no WAV analogue). Together with 21.1 this bounds both the played and unplayed sides on both formats. + - **21.3 — Seek-back-past-window refill + clean failure recovery.** When a backward seek lands earlier than the retained tail, the existing seek-beyond-buffer path is reused pointed at the earlier offset (whichever resolver the active path ships: `IFormatDecoder`/`StreamDecoder` for WAV; the live `resolveOpusByteOffset` + `OpusStreamDecoder.reinitializeForRangeContinuation` for Opus). Minimal AC6 refill-failure handling added: `RecoverFromFailedRefill` clears the scheduler and surfaces a user-visible error rather than leaving the player wedged. + - **21.4 — Validation + Direction A→B pivot (the critical wave — see as-built divergence below).** Network-memory bounding confirmed in Daniel's browser run: the segmented approach delivers ~4 MB segments pacing with playback, with the browser holding ~one segment of raw bytes rather than the full artifact. + +- **As-built divergence — Direction A→B pivot (important).** The spec recommended **Direction A** (sliding window on one open-ended forward stream, relying on pausing `ReadAsync` / the segment loop to backpressure the socket) with **Direction B** (discrete bounded `Range: bytes=start-end` segments, each fetched only when the scheduler drains below low-water) held as the documented fallback "if Direction A's back-pressure proves leaky in practice." + + **21.4 browser validation proved it leaky.** On Blazor WASM, the browser `fetch` API buffers the entire HTTP response body regardless of how slowly the application reads it: pausing reads bounded the *decode* but not the *network download*, so the entire ~970 MB body accumulated in browser memory even though the application only decoded a window of it. Direction A's core assumption — that pausing `ReadAsync` closes the TCP flow-control window before the browser caches the whole body — does not hold in the WASM `fetch` runtime. + + **We pivoted to Direction B.** The forward stream now issues sequential bounded `bytes=cursor-(cursor+SegmentSizeBytes-1)` Range requests (`SegmentSizeBytes = 4 MB`), each fetched only after the scheduler drains below low-water — via `RunSegmentedStreamAsync` in `StreamingAudioPlayerService`. Because each 4 MB request is fully consumed before the next is issued, the browser holds at most ~one segment of raw bytes regardless of file size. The decode-side windowing (21.1/21.2) pairs with it: the segment loop's segment-gate replaces the raw `ReadAsync` pause as the production throttle (21.2a), and the Opus decode-ahead throttle (21.2b) hooks the same `evaluateProductionPause()` signal. Seek and refill converge on the same segmented loop. Direction B is recorded as the shipped approach; Direction A is recorded as tried-in-validation and found insufficient for the WASM `fetch` runtime. + +- **What Phase 21 does NOT include:** the full AC matrix re-run (Opus seek-storm, visualizer, rapid-seek concurrency under the new segmented loop) beyond the network-memory bounding confirmed in 21.4. Those acceptance criteria remain the validation baseline for any follow-on work touching this seam. + +- **Design memo:** `product-notes/phase-21-windowed-streaming-buffer.md` (note: the spec recommended Direction A; the as-built pivot to Direction B is annotated at the top of that doc and in §3.2/§3.3). + +--- + ## Phase 18 — Opus Low-Data Streaming (landed 2026-06-23) _Note: Two distinct efforts share the "Phase 18" label — phase numbers are organisational, not sequential. See also **Phase 18 — Theme / Dark-Mode Remediation (landed 2026-06-19)** below._ diff --git a/PLAN.md b/PLAN.md index ef5bbf3..8760a71 100644 --- a/PLAN.md +++ b/PLAN.md @@ -449,116 +449,9 @@ See `COMPLETED.md` for Phase 18 — dual-format lossless + Opus delivery, includ --- -## Phase 21 — Windowed Streaming Buffer (bounded client memory for long streams) +## Phase 21 — Windowed Streaming Buffer (Completed) -Bound the **client memory** a playing track consumes to a small, configurable forward window — -**independent of total stream length** — so a 1 GB+ DJ MIX (Phase 9 `Mix` medium: a single long track) -plays without the whole decoded PCM accumulating in the browser. **Public listener site only** -(`DeepDrftPublic.Client` player stack + `DeepDrftPublic` TypeScript audio interop); no CMS, no API -endpoint, no schema change. - -**Phase 18 (Opus Low-Data Streaming) has landed — Phase 21 is the next pickup, reconciled to the as-built -two-decode-path reality.** Phase 18 left **two** decode paths feeding the one `PlaybackScheduler`: -WAV/MP3/FLAC via `StreamDecoder`/`IFormatDecoder`, and Opus via a **WebCodecs `AudioDecoder`** pipeline -(`OggDemuxer` → `OpusStreamDecoder`, the `IStreamingDecoder` seam — *not* `IFormatDecoder`; per-segment -`decodeAudioData` was tried and replaced). Windowing must bound **both**. The accurate index-driven Opus -seek the original spec assumed Phase 21 would build is **already live** (`resolveOpusByteOffset` over the -precomputed seek index → Range fetch → `reinitializeForRangeContinuation` with frame-accurate lead-trim); -Phase 21 **reuses** it for window-miss refills rather than building it. Opus seek is VBR-safe and -**accurate**, not approximate (the earlier "approximate page interpolation" framing is corrected). - -The network path already streams in adaptive 16–64 KB chunks. The accumulation is on the **decode -side**, and now has two faces. The shared one: `PlaybackScheduler` holds an `AudioBuffer[]` it **never -evicts** — both paths `addBuffer` into it, nothing is removed. Decoded PCM is larger than the source -(Web Audio is 32-bit float per sample/channel — a 16-bit stereo WAV roughly doubles once decoded), so a -1 GB WAV becomes ~2 GB of retained float; **a low-data Opus mix decodes to the same ~2 GB once played**, -so its small transfer does not spare it. The Opus-only second face: the WebCodecs decode queue + -`decodedQueue` accumulate upstream of the scheduler too. The fix: hold only a sliding forward window plus -a small back-retain, discard already-played buffers, and refill on demand — with back-pressure on the C# -read loop (both paths) **and** on the Opus demux/decode feed (Opus only). - -**Architectural spine — a sliding window keyed on playback position, built as a generalization of the -landed seek-beyond-buffer paths.** The Phase 4 HTTP `Range: bytes=X-` → 206 primitive already does every -plumbing primitive the window needs, with a WAV branch and an Opus branch, both live: discard-buffers-keep- -offset via `clearForSeek`/`setPlaybackOffset` (shared); fetch-from-offset via `TrackMediaClient` (shared, -now with `?format=`); decode-header-less-body via `StreamDecoder.reinitializeForRangeContinuation` (WAV) / -`OpusStreamDecoder.reinitializeForRangeContinuation` (Opus); time→byte via `IFormatDecoder.calculateByteOffset` -(WAV) / `resolveOpusByteOffset` over the seek index (Opus) — just triggered manually and one-shot. The -genuinely new mechanisms: **partial eviction** on the shared scheduler (one implementation, both paths), -and **back-pressure** — on the C# read loop (both paths) **and additionally on the Opus WebCodecs -decode-ahead** (`decodeQueueSize` + `decodedQueue`, Opus only, since throttling the socket alone doesn't -bound the async decoder's queues). Recommended **Direction A** (sliding window on the existing single -forward stream); **Direction B** (discrete Range-fetched segments — the HLS/DASH/MSE-eviction analogue) -held as the documented fallback; **Direction C** (adopt MSE) **rejected (OQ5 = NO, Daniel 2026-06-23)** — -the bespoke Web Audio graph is a deliberate long-term commitment, and the compressed-delivery move that -would have justified MSE was met instead by **Phase 18 (Opus) feeding the same bespoke graph** through the -WebCodecs `IStreamingDecoder` seam. Direction A is the permanent destination, not a stopgap MSE would -retire. - -**Invariants that must hold (the §3.5 seam contract).** Reuse each path's Range/seek machinery, don't fork -it; playback-start latency at parity; neither decoder seam's contract forked — eviction is shared at the -scheduler (zero format branches), back-pressure is seam-aware (the one place the two paths diverge); -read-only playback (no new control); the single-writer decoder discipline holds for **both** decoders -(stricter for the stateful Opus `AudioDecoder` — a stale `push` racing a reset+reconfigure corrupts -inter-frame state). The **Mix visualizer is provably unaffected** — it renders from the preprocessed -per-track high-res datum (Phase 10/12), never from live decoded PCM, so evicting played buffers cannot -starve it. The 1 GB mix is both the canonical case *and* the proof the eviction is safe. - -**Interaction with deferred Phase 1 features (same seam):** windowing should land **before** preload -(1.3) — it makes preload of long tracks memory-safe by construction (a staged next-track decoder inherits -the bounded scheduler); it makes crossfade (1.4) between two long mixes affordable (the overlap doubles -the *window*, not the track); it adds a minor "don't evict the final window before the gapless boundary" -care point for 1.5. It **enlarges the error surface** (1.6): windowed refill issues mid-stream fetches -the listener didn't initiate, one of which can fail deep into a 1 GB mix — so the *cheap* half of 1.6 -(clean refill-failure handling, no wedged player) is folded into this phase's acceptance criteria, not -left fully to 1.6. - -Full design, the three directions with SOLID/road-not-taken rationale, use cases, acceptance criteria, -the open-question set, and the wave decomposition: `product-notes/phase-21-windowed-streaming-buffer.md`. - -Sequenced as four waves. `21.1 → 21.2 → 21.3`, with `21.4` validating the whole. **21.1 is the cold-start -prerequisite and the load-bearing change** — independent of the open questions (window *sizes* are -parameters fed in later). - -Decomposition is **by concern** (eviction → back-pressure → seek-back refill → validate), not by format — -eviction is genuinely shared, so a path-split would duplicate the hardest work; the one path-divergent -concern (back-pressure) carries a two-track split *inside* its wave. - -- **21.1 — Partial eviction in `PlaybackScheduler` (cold-start; load-bearing; SHARED by both paths).** Drop - already-played buffers while keeping the position/index/time-anchor bookkeeping exact against a buffer - array that no longer begins at absolute time 0 (today `getCurrentPosition`/`playFromPosition`/the schedule - loop all assume `buffers[0]` is the track start). The hardest correctness work in the phase. Written once, - serves both decode paths (they `addBuffer` identically). No refill yet. **Independent of the open - questions — can begin immediately.** -- **21.2 — Back-pressure (two tracks, one fill signal).** Bound the unplayed region by throttling - production above a high-water mark and resuming below low-water, driven by the scheduler's - decoded-lookahead fill. **21.2a** — stop `ReadAsync` on the C# loop (serves both paths; *sufficient* for - WAV). **21.2b** — additionally stop the Opus demux/decode feed so the WebCodecs decode queue + - `decodedQueue` don't balloon behind a throttled socket (Opus only; no WAV analogue). Together with 21.1 - this bounds both played and unplayed sides on both formats (AC1 + AC1-Opus). Routes through the existing - single-loop cancellation discipline. **Depends on 21.1.** -- **21.3 — Seek-back-past-window refill (one concern, per-path resolver).** When a backward seek lands - earlier than the retained tail, refetch via the existing seek-beyond-buffer path pointed at the earlier - offset, using whichever resolver the active path already ships (`IFormatDecoder`/`StreamDecoder` for WAV; - the live `resolveOpusByteOffset` + `OpusStreamDecoder.reinitializeForRangeContinuation` for Opus); plus - the minimal clean refill-failure handling (the 1.6 adjacency). Mostly reuse of the landed seek paths. - **Depends on 21.1 + 21.2.** -- **21.4 — Validation against the 1 GB target, BOTH formats (acceptance).** Memory profiling (bounded under - 1 GB as WAV *and* as Opus, plus the Opus upstream queues), latency parity, edge-to-edge playback, the - seek matrix, induced refill failure, visualizer-running, rapid-seek concurrency (incl. an Opus - seek-storm). Largely measurement; breaks are tuning fixes in 21.1's anchor math, 21.2's water-marks, or - 21.2b's Opus decode-ahead bound. **Depends on 21.1–21.3.** - -**Dependency shape:** `21.1 → 21.2 → 21.3 → 21.4`; 21.1 is the only cold-start wave. **Phase-level -prerequisite: Phase 18 (Opus) has landed** (`COMPLETED.md`) — windowing is built against both formats. -**Open questions for Daniel (spec §6):** window-size policy axis (time-based window + memory guard — -recommended); seek-back-past-window re-buffer acceptable (recommend yes, symmetric to forward); a hard -total in-flight memory cap as a guard rail (recommend yes); window everything vs. only long tracks -(recommend everything — one path, short tracks never hit a refill). **New staff-engineer architecture -calls (spec §6):** OQ6 — one window controller for both paths or two (recommend shared controller + two -back-pressure hooks); OQ7 — drive the Opus decode-ahead bound from the single scheduler-fill signal -(recommended). **OQ5 (adopt MSE) — RESOLVED NO (Daniel 2026-06-23):** the bespoke graph stays by deliberate -choice. None block 21.1. +See `COMPLETED.md` for Phase 21 — bounded client memory for long streams, including the as-built Direction A→B pivot — which landed on `streaming-overhaul` (2026-06-24). --- diff --git a/product-notes/phase-21-windowed-streaming-buffer.md b/product-notes/phase-21-windowed-streaming-buffer.md index d4604f7..c91ef60 100644 --- a/product-notes/phase-21-windowed-streaming-buffer.md +++ b/product-notes/phase-21-windowed-streaming-buffer.md @@ -1,9 +1,24 @@ # Phase 21 — Windowed Streaming Buffer (bounded client memory for long streams) -Product spec. Status: **design / framing — reconciled to as-built Phase 18 (two decode paths); -implementation-ready pending Daniel's open-question calls (OQ1–OQ4 product; OQ6–OQ7 staff-engineer -architecture).** Author: product-designer. Date: 2026-06-23 (reconciliation pass after Phase 18 landed). -**No code has been written by this doc.** +Product spec. Status: **LANDED 2026-06-24 on `streaming-overhaul`.** See `COMPLETED.md` for the full +as-built record. Author: product-designer. Date: 2026-06-23 (reconciliation pass after Phase 18 landed). + +> **AS-BUILT NOTE — Direction A→B pivot (2026-06-24).** This spec recommended **Direction A** (sliding +> window on one open-ended forward stream, pausing `ReadAsync`/the segment loop to backpressure the +> socket) and held **Direction B** (discrete bounded `Range: bytes=start-end` segments, §3.2) as the +> documented fallback. **21.4 browser validation proved Direction A insufficient for Blazor WASM:** the +> browser `fetch` API buffers the entire HTTP response body regardless of read pace — pausing reads +> bounded the *decode* but not the *network download*, so the whole ~970 MB body accumulated in browser +> memory even with the application decoding only a window of it. **We shipped Direction B.** The forward +> stream now issues sequential 4 MB bounded Range requests (`SegmentSizeBytes = 4 MB`), fetched via +> `RunSegmentedStreamAsync` in `StreamingAudioPlayerService`, each issued only after +> `PlaybackScheduler.evaluateProductionPause()` clears below low-water. Browser holds ~one 4 MB segment +> of raw bytes; 21.4 confirmed network-memory bounding in Daniel's browser run. The decode-side windowing +> (21.1/21.2) is unchanged and pairs with Direction B; seek/refill converge on the same segmented loop +> via `RecoverFromFailedRefill`. Direction A is recorded as tried-in-validation and found insufficient +> for the WASM `fetch` runtime. Sections §3.2–§3.3 below retain the original A vs. B vs. C analysis as +> the decision record; **Direction B is what shipped.** + Surface: **public listener site only** (`DeepDrftPublic.Client` player stack + `DeepDrftPublic` TypeScript audio interop). No CMS (`DeepDrftManager`) change. No data-model or schema change. The one server touch is **reuse, not new surface**: the existing `DeepDrftAPI` HTTP `Range: bytes=X-`