diff --git a/CLAUDE.md b/CLAUDE.md index 2b55748..32deeba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -77,6 +77,18 @@ The player is not fetch-then-play: - **Decoded-queue bound (`PlaybackScheduler`):** `evictPlayedBuffers()` discards already-played `AudioBuffer`s, capping the scheduler's forward fill to a 96 MB ceiling (Phase 21.2). Decoded PCM is larger than source (Web Audio uses 32-bit float — a 16-bit stereo WAV roughly doubles; Opus decodes to the same float footprint). - **Network bound (segmented fetch, Phase 21 Direction B):** The forward stream is fetched as sequential bounded `bytes=cursor-{cursor+4MB-1}` Range requests via `RunSegmentedStreamAsync`; the next segment is fetched only after `DrainBackpressureAsync` confirms the scheduler is below low-water. Because each segment is fully consumed before the next is issued, the browser holds at most ~one segment of raw bytes. A per-chunk drain inside the segment loop (gated on `_streamingPlaybackStarted` so it cannot deadlock first-audio) additionally prevents high-density codecs (e.g. Opus, where a 4 MB segment is ~100 s of audio) from decoding the whole segment eagerly ahead of the playhead before the inter-segment gate runs. +**Playback stability invariants (streaming-stabilization arc):** +- **Back-pressure water marks:** forward fill 60s high / 30s low (`PlaybackScheduler.DEFAULT_FORWARD_HIGH_WATER_SECONDS` / `...LOW_WATER_SECONDS`). Production pauses on `lookahead ≥ high OR decoded bytes > 96 MB ceiling`, whichever first. The time window is a jitter cushion for Opus's async decode ramp; the byte cap is the hard OOM guarantee and is unchanged. +- **Genuine end-of-playback:** `PlaybackScheduler` uses a `streamComplete` flag (set by `setStreamComplete`) combined with an `underrun_` park/resume state to distinguish a drained-but-still-streaming queue (startup/underrun gap → park, resume on refill) from a truly finished track (stream complete AND queue drained → `finishPlayback()`, the single genuine-end path). Prevents the false `onPlaybackEnded` that previously fired when the Opus WebCodecs queue momentarily drained during the decode ramp. +- **Rebuffer hysteresis:** Opus playback start and underrun-resume are gated on `hasMinimumPlaybackLead()` (1s decoded lead, `DEFAULT_MIN_PLAYBACK_LEAD_SECONDS`); WAV keeps `hasMinimumBuffers(6)`. `streamComplete` overrides the gate so a short tail still plays out. `StreamingAudioPlayerService.TryStartPlaybackAsync()` force-starts after `MarkStreamCompleteAsync` when the threshold was never crossed (ultra-short track protection). +- **Opus `AudioContext` pre-aligned to 48 kHz** in `AudioPlayer.initializeStreaming` before any bytes flow, eliminating the mid-decode context teardown that previously OOM'd the tab under software rendering. + +**Visualizer / decode contention (decodePressure + hwAccel):** +- `DeepDrftPublic/Interop/audio/decodePressure.ts` exports a shared `DecodePressureSignal` singleton. The audio pipeline (`OpusStreamDecoder` yield-cap events, `PlaybackScheduler` underrun-parking events) calls `report()` on sustained lag; the visualizer calls `isUnderPressure()` each rAF frame. The signal engages only after ≥ 5 reports within 2500 ms with a 1s minimum hold (hysteresis); a lone startup-ramp blip never engages. +- Under pressure, `WaveformVisualizer.ts` throttles its rAF loop to ~15 fps (`PRESSURE_THROTTLE_FRAME_MS = 1000/15`), cutting main-thread WebGL software-render + physics cost so WebCodecs decode recovers. A no-op under HW accel. +- `DeepDrftPublic/Interop/visualizer/hwAccel.ts` probes `WEBGL_debug_renderer_info.UNMASKED_RENDERER_WEBGL` for software-renderer signatures (SwiftShader, llvmpipe, softpipe, Microsoft Basic Render, etc.). Absence of the debug extension is treated as accelerated — lava is disabled only on positive evidence. On first interactive render, `WaveformVisualizerControlState.ApplyCapabilityDefault(bool)` applies a one-time scoped default: `LavaEnabled = false` (the expensive subsystem) when no HW accel is detected, `WaveformEnabled` stays on. Guarded by `_capabilityDefaultApplied`; never overrides an explicit user toggle. +- **Known limitation / deferred escalation:** HW-accel-off Opus playback is made usable by defaulting lava off and throttling under pressure. If starvation recurs (e.g. waveform-only path also proves too costly, or a software renderer slips the probe), the documented next step is moving Opus WebCodecs decode off the main thread (Web Worker / AudioWorklet) so it stops competing with main-thread rendering. Not a current implementation — a recorded fallback. + Keep this seam clean — it is the most architecturally load-bearing part of the playback path. ### Theming and dark mode diff --git a/COMPLETED.md b/COMPLETED.md index 8fdfaad..0470bde 100644 --- a/COMPLETED.md +++ b/COMPLETED.md @@ -40,6 +40,22 @@ Newest entries at the top. Group by phase/wave header (mirroring `PLAN.md` / `CM Together with the existing decoded-side 96 MB ceiling (Phase 21.2) and the segmented fetch (Phase 21 Direction B), these complete the three-layer memory bound: raw queue, decoded queue, and network. +### Phase 21 — Post-landing: streaming stabilization (landed 2026-06-26) + +**Landed:** 2026-06-26 on `streaming-overhaul`. Resolves playback instability that surfaced specifically with Opus + hardware acceleration off: the Opus underrun/false-end cycle, visualizer starvation of WebCodecs decode, and an OOM path from mid-decode `AudioContext` teardown. All changes are TypeScript-only (`DeepDrftPublic/Interop/audio/` + `DeepDrftPublic/Interop/visualizer/`) plus the C# service that calls them; no API, no schema, no CMS change. + +1. **Back-pressure water marks widened to 60s high / 30s low.** `PlaybackScheduler.ts` `DEFAULT_FORWARD_HIGH_WATER_SECONDS` / `DEFAULT_FORWARD_LOW_WATER_SECONDS` doubled from 30/15. The 96 MB decoded-byte ceiling is unchanged and remains the hard OOM bound; production pauses on `lookahead ≥ high OR bytes > cap`, whichever first. The wider time window absorbs per-packet WebCodecs decode jitter (Opus 48 kHz stereo ≈ 0.37 MB/s → 60 s ≈ 23 MB — well within the cap) without false back-pressure on the Opus decode ramp. + +2. **End-of-playback gated on genuine completion (`streamComplete` + `underrun_`).** `PlaybackScheduler` now distinguishes a drained-but-still-streaming queue (startup/underrun gap) from a truly finished track (stream complete AND queue drained). `streamComplete` (set by `setStreamComplete`, called from `AudioPlayer.markStreamComplete`) is the discriminator. When the queue drains while `streamComplete` is false the scheduler parks in `underrun_` state (anchors frozen, `isActive_` false) and resumes from `scheduleNewBuffers` when buffers arrive. `finishPlayback()` is the single genuine-end path, reached only when both conditions hold simultaneously. Previously a drained Opus queue during the WebCodecs startup ramp fired `onPlaybackEnded`, nulling Duration and advancing the queue. + +3. **Rebuffer hysteresis — 1s decoded-lead gate.** `hasMinimumPlaybackLead()` gates Opus playback START and underrun RESUME (`DEFAULT_MIN_PLAYBACK_LEAD_SECONDS = 1.0`), preventing the decode-ramp thrash (resume on a 20 ms Opus packet → drain immediately → re-park → repeat). `streamComplete` overrides the gate so a short tail plays out rather than parking forever. WAV keeps its `hasMinimumBuffers(6)` buffer-count start gate unchanged — large synchronous WAV segments rarely underrun and that gate's character must not change. **Complete-without-start force-start:** `StreamingAudioPlayerService.TryStartPlaybackAsync()` is called after `MarkStreamCompleteAsync` when `_streamingPlaybackStarted` is still false, preventing an ultra-short Opus track (total audio below the 1s lead) from sitting loaded-but-not-playing. + +4. **Opus `AudioContext` pre-aligned to 48 kHz.** `AudioPlayer.initializeStreaming` calls `contextManager.recreateWithSampleRate(OPUS_SAMPLE_RATE)` when the content type resolves as Opus, before any bytes flow. The context is already at 48 kHz when the decoder's own lazy `recreateWithSampleRate` check runs, so it short-circuits (no-op). Eliminates the mid-decode context teardown-and-rebuild that previously double-decoded the stream and OOM'd the tab under HW accel off. + +5. **`decodePressure` singleton + `WaveformVisualizer` auto-throttle (~15 fps under pressure).** `DeepDrftPublic/Interop/audio/decodePressure.ts` exports a process-wide `DecodePressureSignal` singleton shared between the audio pipeline (producer: `OpusStreamDecoder` yield-cap events and `PlaybackScheduler` underrun-parking events) and the visualizer (consumer). Engages only on sustained stress (≥ 5 `report()` calls within a 2500 ms window) and holds for a 1000 ms minimum dwell before releasing, suppressing lone startup-ramp blips. `WaveformVisualizer.ts` checks `decodePressure.isUnderPressure()` each rAF frame; when engaged it skips frames so at most one renders per `PRESSURE_THROTTLE_FRAME_MS` (`1000/15` ms, ~15 fps), cutting main-thread WebGL software-render + physics cost so WebCodecs decode recovers. A no-op under HW accel — healthy decode never calls `report()` and `isUnderPressure()` stays false indefinitely. + +6. **Hardware-acceleration detection + `ApplyCapabilityDefault`.** `DeepDrftPublic/Interop/visualizer/hwAccel.ts` probes `WEBGL_debug_renderer_info.UNMASKED_RENDERER_WEBGL` for known software-renderer signatures (SwiftShader, llvmpipe, softpipe, Microsoft Basic Render Driver, Mesa Offscreen, "software"). Absence of the debug extension is treated as accelerated — the probe disables lava only on positive evidence of software rendering, not on uncertainty. `WaveformVisualizerControlState.ApplyCapabilityDefault(bool hardwareAccelerated)` is called once per session by the visualizer bridge on first interactive render: when `hardwareAccelerated` is false it sets `LavaEnabled = false` (the expensive main-thread subsystem) while leaving `WaveformEnabled` on, coerces Theater Mode, and raises `Changed` once so all observers see the default in a single cycle. Guarded by `_capabilityDefaultApplied` so it runs at most once per session and never overrides an explicit in-session user toggle. With HW accel present the call is a no-op. + --- ## Phase 18 — Opus Low-Data Streaming (landed 2026-06-23) diff --git a/DeepDrftPublic.Client/CLAUDE.md b/DeepDrftPublic.Client/CLAUDE.md index 19ab52a..bd24712 100644 --- a/DeepDrftPublic.Client/CLAUDE.md +++ b/DeepDrftPublic.Client/CLAUDE.md @@ -50,7 +50,7 @@ All interactive UI for the site. Blazor WebAssembly. Pages, controls, the stream - `StreamingAudioPlayerService`: Production implementation. Chunked stream from `TrackMediaClient`, adaptive 16–64 KB buffer, early-playback, **seek-beyond-buffer** via offset request to the content API. - `AudioInteropService`: JS interop wrapper over `window.DeepDrftAudio`. Manages `DotNetObjectReference` lifetimes for progress, end-of-playback, spectrum callbacks. - Dark-mode services: `DarkModeServiceBase` (cookie name constant), `DarkModeCookieService` (JS cookie read/write). - - `WaveformVisualizerControlState`: Scoped session-persistent holder for the visualizer's **eight** continuous control positions, **two subsystem on/off toggles** (Phase 15), and one **Theater-Mode flag** (Phase 20): `ScrollSpeed`, `GradientRotationSpeed`, `LavaGravity`, `LavaHeat`, `FluidAmount` (wax count/volume), `FluidViscosity` (cohesion — the second half of the Phase 10 "bubbles" split; `BlobDensity` is gone), `CollisionStrength`, `WaveformWidth`, `LavaEnabled` (bool, default `true`), `WaveformEnabled` (bool, default `true`), `TheaterMode` (bool, default `false` — `DefaultTheaterMode`). Each has a matching `Default*` const. `Changed` event is the decoupling seam — controls mutate state + raise `Changed`; the bridge (`WaveformVisualizer`) subscribes and pushes the affected uniform or subsystem-enable; the Theater observers (the three detail pages and `AudioPlayerBar`) subscribe to react to `TheaterMode`. **`CoerceTheaterMode()`**: enforces the invariant that Theater Mode cannot remain on when both subsystems are off — called from `WaveformVisualizerControls.ToggleLava`/`ToggleWaveform` **before** `NotifyChanged()` so all observers see a consistent, coerced state in the same `Changed` cycle. `TheaterMode` is a page-chrome presentation flag; the visualizer bridge ignores it. Scoped DI so state survives SPA nav within a session and resets on fresh page load. **Phase 20 Wave 2 — playing-release predicates** live in `ReleaseDetailBase` / `CutDetailBase` (not in this state holder): `IsThisReleasePlaying` (`PlayerService?.CurrentTrack?.Release?.EntryKey == EntryKey`), `IsContentHidden` (`TheaterMode && IsThisReleasePlaying`), `ShowTheaterToggle` (`(LavaEnabled || WaveformEnabled) && IsThisReleasePlaying`). Both base classes also subscribe to `IStreamingPlayerService.StateChanged` (idempotent, reference-guarded, disposed) so the predicates re-evaluate live when playback moves between releases. + - `WaveformVisualizerControlState`: Scoped session-persistent holder for the visualizer's **eight** continuous control positions, **two subsystem on/off toggles** (Phase 15), and one **Theater-Mode flag** (Phase 20): `ScrollSpeed`, `GradientRotationSpeed`, `LavaGravity`, `LavaHeat`, `FluidAmount` (wax count/volume), `FluidViscosity` (cohesion — the second half of the Phase 10 "bubbles" split; `BlobDensity` is gone), `CollisionStrength`, `WaveformWidth`, `LavaEnabled` (bool, default `true`), `WaveformEnabled` (bool, default `true`), `TheaterMode` (bool, default `false` — `DefaultTheaterMode`). Each has a matching `Default*` const. `Changed` event is the decoupling seam — controls mutate state + raise `Changed`; the bridge (`WaveformVisualizer`) subscribes and pushes the affected uniform or subsystem-enable; the Theater observers (the three detail pages and `AudioPlayerBar`) subscribe to react to `TheaterMode`. **`CoerceTheaterMode()`**: enforces the invariant that Theater Mode cannot remain on when both subsystems are off — called from `WaveformVisualizerControls.ToggleLava`/`ToggleWaveform` **before** `NotifyChanged()` so all observers see a consistent, coerced state in the same `Changed` cycle. **`ApplyCapabilityDefault(bool hardwareAccelerated)`**: one-time scoped capability default (guarded by `_capabilityDefaultApplied`; never re-applies on SPA navigation, never overrides an explicit in-session toggle). When `hardwareAccelerated` is false (positive software-renderer match from `hwAccel.ts`'s `UNMASKED_RENDERER_WEBGL` probe, or total WebGL failure) sets `LavaEnabled = false` while leaving `WaveformEnabled` at its default on, then calls `CoerceTheaterMode()` + `NotifyChanged()` once so all observers see the default in a single cycle. Called by the visualizer bridge on first interactive render once JS interop (the HW-accel probe via `detectHardwareAcceleration()` exported from `WaveformVisualizer.ts`) is available; a no-op when HW accel is present. `TheaterMode` is a page-chrome presentation flag; the visualizer bridge ignores it. Scoped DI so state survives SPA nav within a session and resets on fresh page load. **Phase 20 Wave 2 — playing-release predicates** live in `ReleaseDetailBase` / `CutDetailBase` (not in this state holder): `IsThisReleasePlaying` (`PlayerService?.CurrentTrack?.Release?.EntryKey == EntryKey`), `IsContentHidden` (`TheaterMode && IsThisReleasePlaying`), `ShowTheaterToggle` (`(LavaEnabled || WaveformEnabled) && IsThisReleasePlaying`). Both base classes also subscribe to `IStreamingPlayerService.StateChanged` (idempotent, reference-guarded, disposed) so the predicates re-evaluate live when playback moves between releases. - `PlayTracker`: Per-session play-session tracker (Phase 16 wave 16.1). Opens on playback start, advances a high-water position on each progress tick (from `StreamingAudioPlayerService` — not the HTTP layer, so seek-beyond-buffer re-fetches are the same play), closes on track-switch / stop / organic-end / page-unload. Engagement floor: ≥3 s OR ≥5% of duration. Three-bucket classification (`partial`/`sampled`/`complete`). Emits at most one event per session via `IPlayEventSink`. No player or JS dependency — testable against a fake sink. - `ShareTracker`: Per-session share tracker (Phase 16 wave 16.1). Called by `SharePopover` after a successful clipboard write; applies a 60-second per-(target, channel) debounce. Sends via `BeaconInterop`. Scoped so debounce memory resets on fresh page load. **Wave 16.3:** injects `IAnonIdProvider`; attaches `_anonId.Current` to `ShareEventDto.AnonId` (omitted when null). - `BeaconInterop`: `navigator.sendBeacon` JS interop wrapper (Phase 16 wave 16.1). Fires JSON payloads to `api/event/{play,share}` fire-and-forget. Also wires a page-unload handler that flushes any pending play event when the page is torn down.