deepdrft/DeepDrftContent/CLAUDE.md

CLAUDE.md - DeepDrftContent.Services

Guidance for working in the DeepDrftContent.Services project (the binary-content domain logic).

See the root CLAUDE.md for full architecture overview. This file covers what is specific to this project.

One-line purpose

Binary-content domain logic. The FileDatabase implementation in full (Models, Services, Utils, Abstractions, Constants), audio processing, and the content-side track service. Consumed by DeepDrftContent (the host) and DeepDrftCli (the admin CLI).

Layout

DeepDrftContent.Services/
├── FileDatabase/                       # The subsystem (port of TypeScript system)
│   ├── Abstractions/                   # Interfaces
│   ├── Models/                         # Data models, DTOs, enums
│   ├── Services/                       # FileDatabase, MediaVault, IndexSystem, IndexWatcher
│   └── Utils/                          # StructuralMap, StructuralSet, FileUtils
├── Processors/
│   ├── AudioProcessor.cs               # WAV file parsing, metadata extraction, streaming PCM header parse
│   ├── AudioStoreStream.cs             # Bounded streaming copy/prefix helpers used by the processors
│   └── ProcessedAudio.cs               # Store-path plan: metadata + streamed WriteToAsync callback
├── Constants/
│   └── VaultConstants.cs               # Vault name definitions
├── TrackService.cs                     # Content-side orchestrator
└── DeepDrftContent.Services.csproj

FileDatabase model (high-level)

See FileDatabase/README.md for the long-form design discussion — it's a port of a TypeScript system and has deep rationale. This section covers the essentials for an agent walking in cold.

Core structure

• FileDatabase: Root object. Created via FileDatabase.FromAsync(rootPath). Holds a collection of MediaVault instances and an IndexWatcher. Implements IDisposable. Singleton in the host.
• MediaVault: A subdirectory under the FileDatabase root. Each vault has its own JSON index file listing entries and per-entry metadata. Typed via MediaVaultType enum (Media | Image | Audio).
  • Concrete implementations: ImageVault (for images), AudioVault (for audio). Do not use ImageDirectoryVault (that's stale docs) — the type is ImageVault.
• Entry filenames: {sanitized-key}{extension}, where sanitisation is Regex.Replace(entryKey, @"[^a-zA-Z0-9]", "-"). So entry id "my-song" with extension .wav → filename my-song.wav.

Binary hierarchy

FileBinary (base: byte buffer)
  └── MediaBinary (+ Extension: string, MIME type inferred via MimeTypeExtensions)
      ├── AudioBinary (+ Duration: double, Bitrate: int)
      └── ImageBinary (+ AspectRatio: double)

Each has a matching *Dto variant for base64 JSON transport (e.g., AudioBinaryDto with buffer encoded as base64).

Read/load path: vault reads (LoadResourceAsync<AudioBinary>) still return a full-buffer AudioBinary. Write/store path is streaming: audio processors return a ProcessedAudio plan (metadata + WriteToAsync callback); RegisterResourceStreamingAsync / MediaVault.AddEntryStreamingAsync write bytes to a temp file, then File.Move atomic-rename into place. The full AudioBinary buffer is never materialized on the store path.

Index lifecycle

• DirectoryIndex: Root index file (at {rootPath}/index). Tracks which vaults exist.
• VaultIndex: Per-vault index (at {vaultPath}/index). Records MediaVaultType and lists all entries in that vault.
• Both are JSON files. Created/loaded via IndexFactoryService.
• When a file is written externally (e.g., the CLI calls FileDatabase.RegisterResourceAsync directly), the IndexWatcher detects the write to the vault's index file and triggers MediaVault.ReloadIndexAsync, so a long-running web host stays consistent without restart.

Error-handling philosophy (load-bearing)

Public Load* / Register* operations swallow exceptions and return null / false to match the TypeScript original.

public async Task<T?> LoadResourceAsync<T>(string vaultId, string entryId) where T : FileBinary
{
    try { /* load and deserialize */ }
    catch { return null; }  // Swallow, return null
}

public async Task<bool> RegisterResourceAsync(string vaultId, string entryId, FileBinary media)
{
    try { /* store and update index */ }
    catch { return false; }  // Swallow, return false
}

// Streaming counterpart (Wave 1 OOM fix) — same bool contract:
public async Task<bool> RegisterResourceStreamingAsync(
    string vaultId, string entryId, MetaData metaData,
    Func<Stream, CancellationToken, Task> writeContent, CancellationToken ct)
{
    try { /* stream to temp → atomic rename → update index */ }
    catch { return false; }  // Swallow (logs non-cancel exceptions), return false
}

MediaVault.AddEntryStreamingAsync (called internally) writes bytes to a temp file in the vault directory, then File.Move(temp, final, overwrite: true) (atomic POSIX rename on the Linux prod host), then updates the index. A cancel or I/O fault before the rename leaves any prior backing file intact; the temp file is cleaned up best-effort.

Callers must check return values. Do not change this without a deliberate design pass — it's embedded in all FileDatabase tests and client code.

Audio processors

Multi-format support via router pattern. All processors live in DeepDrftContent/Processors/:

• AudioProcessor.ProcessWavFileAsync(filePath): WAV-specific processor. Validates RIFF/WAVE structure and format code. Accepts standard PCM (audioFormat=1) and WAVE_FORMAT_EXTENSIBLE (audioFormat=0xFFFE) when the SubFormat GUID indicates PCM. Normalizes EXTENSIBLE-PCM uploads to standard 44-byte PCM WAV before storing. Parses fmt and data chunks; extracts duration and bitrate. Returns ProcessedAudio (metadata + streamed WriteToAsync callback — no whole-file buffer). Header window grows in 64 KB steps capped at 8 MB; the audio body is never read during processing. Standard PCM WAV is stored verbatim (passthrough via AudioStoreStream.CopyFileAsync); EXTENSIBLE-PCM / IEEE-float / padded-container WAVs stream their normalization to standard 24-bit PCM. On parse failure, falls back to defaults (180s / 1411 kbps). Also exposes TryExtractPcm(ReadOnlySpan<byte>) for the whole-buffer waveform parity oracle and TryReadPcmStreamInfoAsync(stream, totalLength) for the streaming waveform compute path (bounded header parse from a stream; returns WavPcmStreamInfo? with DataStart/DataLength/format fields).
• Mp3AudioProcessor.ProcessMp3FileAsync(filePath): MP3 processor. Reads a bounded ≤8 MB prefix for header parsing; skips ID3v2 tag, finds first valid MPEG frame sync, decodes frame header (bitrate, sample rate, channels). Reads Xing/Info header for VBR total-frame count (accurate duration); VBRI header as fallback; CBR estimate from file size otherwise. Returns ProcessedAudio (passthrough plan — MP3 stored unmodified). On parse failure, falls back to defaults (180s / 320 kbps).
• FlacAudioProcessor.ProcessFlacFileAsync(filePath): FLAC processor. Reads a bounded ≤64 KB prefix. Validates fLaC magic, reads STREAMINFO metadata block (20-bit sample rate, 3-bit channels, 5-bit bits-per-sample, 36-bit total samples — all bit-packed). Computes duration from totalSamples / sampleRate; average bitrate from file size. Returns ProcessedAudio (passthrough plan — FLAC stored unmodified). On parse failure, falls back to defaults (180s / 1411 kbps).
• AudioProcessorRouter.ProcessAudioFileAsync(filePath): Routes by extension — .wav → AudioProcessor, .mp3 → Mp3AudioProcessor, .flac → FlacAudioProcessor. Returns ProcessedAudio?. Throws ArgumentException for unsupported extensions.
• ProcessedAudio: Store-path plan returned by all processors. Carries Extension, Duration, Bitrate, Size, and a WriteToAsync(destination, ct) callback that streams the canonical vault bytes to the destination without materializing the whole file. ProcessedAudio.Passthrough(sourcePath, ...) builds a passthrough plan via AudioStoreStream.CopyFileAsync.
• AudioStoreStream: Internal bounded-buffer streaming helpers. CopyFileAsync(sourcePath, destination, ct) does a bounded 80 KB disk-to-disk copy; ReadPrefixAsync(path, cap, ct) reads at most cap bytes from the start of a file (used by processors for header parsing without loading the body).
• ILoudnessAlgorithm / ILoudnessAccumulator: The loudness strategy interface now exposes both Compute(ReadOnlySpan<byte>, ...) (whole-buffer, retained as the parity oracle for tests — no production callers) and CreateAccumulator(pcmByteLength, ...) → ILoudnessAccumulator (streaming: Add(ReadOnlySpan<byte>) / Finish() → double[]). RmsLoudnessAlgorithm implements both; Compute is defined in terms of the accumulator, so streaming and whole-buffer outputs are byte-identical.
• WaveformProfileService: Streaming loudness compute + vault store. The whole-buffer ComputeAndStoreAsync / ComputeAndStoreHighResAsync methods are retained but have no production callers — they exist as the byte-identity parity oracle for tests; do not delete them. The production paths are: ComputeAndStoreProfileStreamingAsync (512-bucket seeker profile, tri-state bool?), ComputeAndStoreHighResStreamingAsync (duration-derived high-res datum, tri-state bool?), and ComputeAndStoreAllStreamingAsync (both datums in a SINGLE streaming pass, used by the upload/replace hot path, tri-state bool?). Tri-state: null = no backing audio stream; false = audio present but not WAV-decodable or vault write failed; true = stored. Streaming reads the WAV in bounded ≤80 KB chunks through one accumulator per datum; peak memory is O(bucket arrays + read buffer), independent of file size.
• WaveformResolution: Enum / constants controlling bucket density for the high-res compute. Renamed from MixWaveformResolution in Phase 12.

Vault stores original bytes with correct extension and MIME type (inferred from file extension or content-type header at upload time).

The primary entry point is TrackContentService.AddTrackAsync(filePath, mimeType) — format-agnostic. It selects the right processor via AudioProcessorRouter, processes the file, generates an entry GUID, stores in vault, returns unpersisted TrackEntity. Legacy AddTrackFromWavAsync(filePath) is now a shim over AddTrackAsync for backward compatibility.

Image processor

ImageProcessor.ProcessImageAsync(buffer, mimeType):

1. Accepts raw image bytes and MIME type (e.g., image/png, image/jpeg).
2. Parses PNG or JPEG headers to extract image dimensions.
3. Computes aspect ratio (width / height). Defaults to 1.0 if parsing fails or format is unsupported.
4. Returns ImageBinary with MIME type and aspect ratio metadata.
5. No disk I/O: operates on byte[] only — no file reading required.

Content-side TrackService (orchestrator)

AddTrackAsync(audioFilePath, ...)

The primary upload entry point. Format-agnostic — routes by extension via AudioProcessorRouter.

1. Calls AudioProcessorRouter.ProcessAudioFileAsync(filePath) → ProcessedAudio plan (no whole-file buffer — Wave 1 OOM fix).
2. Generates a GUID entry key (via Guid.NewGuid().ToString()).
3. Ensures the tracks vault exists (creates if missing).
4. Builds MetaData from the plan's header-extracted fields and calls FileDatabase.RegisterResourceStreamingAsync("tracks", entryKey, metaData, processed.WriteToAsync) — bytes are streamed from the staging file to the vault via a bounded copy; MediaVault.AddEntryStreamingAsync writes to a temp file then atomic-renames into place.
5. Returns a populated TrackEntity (with Id = 0 since it's not yet in SQL, and DurationSeconds populated from the header parse).

If the vault write succeeds and SQL write fails, audio is orphaned (no compensating rollback).

AddTrackFromWavAsync(filePath, ...)

Backward-compatible shim — delegates to AddTrackAsync. The router accepts WAV alongside MP3 and FLAC, so this carries no WAV-specific logic of its own.

ReplaceTrackAudioAsync(entryKey, audioFilePath)

Swaps the vault bytes for an existing track in place, under the same entryKey. Captures the old extension from the vault index metadata (not by loading the file) so a cross-format swap can clean up the stale backing file post-success. Streams the new audio via ProcessedAudio + RegisterResourceStreamingAsync (no whole-file buffer — Wave 1 OOM fix). Returns the new audio's duration on success, null on failure (original audio left intact).

GetAudioBinaryAsync(entryKey)

Reads audio from the tracks vault via FileDatabase.LoadResourceAsync<AudioBinary>("tracks", entryKey). Returns a full-buffer AudioBinary or null if not found. This is the read/playback path — unchanged by the streaming store fix.

OpenAudioStreamAsync(entryKey)

Returns a read-only, seekable Stream over a track's vault audio without buffering the whole file. Used by the streaming waveform compute path (WaveformProfileService). Caller owns and must dispose the stream. Returns null if the entry has no backing file.

OpenAudioMediaStreamAsync(entryKey)

Returns a MediaStream? — a read-only, non-buffering stream over a track's vault audio together with its stored extension, or null if the entry has no backing file. Same non-buffering contract as OpenAudioStreamAsync, but exposes .Extension alongside .Stream so the caller can name a staging file by the original format (the Opus transcode stages the source with the correct extension so ffmpeg can detect the format). The caller owns the returned MediaStream and must dispose it. Follows the FileDatabase swallow-and-return-null contract.

GetAudioDurationAsync(entryKey)

Reads the stored audio duration from the vault index metadata only — no audio body load. Used by the streaming waveform compute to derive the duration-based bucket count, and by UnifiedTrackService.BackfillDurationsAsync. Returns null if the entry is unknown or carries no audio metadata.

InitializeTracksVaultAsync()

Safety call to ensure the tracks vault exists (creates if missing). Called on host startup.

Vault constants

VaultConstants.Tracks = "tracks", VaultConstants.Images = "images", and VaultConstants.TrackWaveforms = "track-waveforms" — the vault names in production use. TrackWaveforms holds the per-track high-res waveform datum keyed by TrackEntity.EntryKey (Phase 12; renamed from the former mix-waveforms, which was Mix-only). New vault names go here when adding new vault types.

Service registration

In DeepDrftContent/Startup.ConfigureDomainServices() and DeepDrftCli/Program.cs:

services.AddSingleton<FileDatabase>(/* from FileDatabase.FromAsync */);
services.AddScoped<AudioProcessor>();
services.AddScoped<TrackService>();  // DeepDrftContent.Services.TrackService

Development commands

# Build
dotnet build DeepDrftContent.Services

# Run tests (FileDatabase tests cover vault/index/factory/utilities thoroughly)
dotnet test DeepDrftTests/

# Run CLI (which consumes this service)
dotnet run --project DeepDrftCli -- add myfile.wav "Track Name" "Artist"

Important patterns

• Async/await: All FileDatabase operations are async. No sync methods.
• Type safety: Generic LoadResourceAsync<T> ensures callers know what they're loading.
• Vault lifecycle: Vaults are created on first boot, then reused. The FileDatabase singleton holds them in memory with live IndexWatcheres.
• Entry sanitisation: Keys are sanitised client-side (in the CLI and web host) and by MediaVault (defensive). Always sanitise before registering — it's the only way to ensure safe filenames.
• Metadata hierarchy: Use the appropriate media type (AudioBinary, ImageBinary) so downstream code can rely on the metadata. Don't store an audio file as a generic MediaBinary — use AudioBinary with duration/bitrate.

What does NOT live here

• HTTP controllers or middleware — that's DeepDrftContent.
• SQL database code — that's DeepDrftWeb.Services.
• Blazor components or UI logic — that's DeepDrftWeb.Client.
• Configuration files (appsettings.json, filedatabase.json) — those are in the host project.

When working with this project, focus on the FileDatabase subsystem (the most complex piece of the codebase), audio processing, and the orchestration logic that bridges binary and SQL databases. The tests (in DeepDrftTests/) are the load-bearing documentation of FileDatabase behaviour — consult them when FileDatabase semantics are unclear.