diff --git a/CLAUDE.md b/CLAUDE.md index 2e761d9..9275242 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -126,7 +126,7 @@ All projects load secrets via `CredentialTools.ResolvePathOrThrow()` from gitign - `DeepDrftPublic/appsettings.json`: Logging and URL config. Secrets loaded from `environment/api.json` (DeepDrftAPI base URL via `Api:ContentApiUrl`). - `DeepDrftManager/appsettings.json`: Logging and URL config. Secrets loaded from `environment/api.json` (DeepDrftAPI base URL via `Api:ContentApiUrl` and API key via `Api:ContentApiKey`). Non-secret upload tunables (in `appsettings.json` itself, not `environment/`): `Upload:IdleTimeoutSeconds` (default 90 — aborts a stalled body-streaming phase) and `Upload:ResponseTimeoutSeconds` (default 1200 — budget for server-side persist after the body is fully sent). -- `DeepDrftAPI/appsettings.json`: Logging and hosting config. Secrets loaded from `environment/filedatabase.json` (FileDatabase vault path), `environment/apikey.json` (API key), `environment/connections.json` (SQL and Auth connection strings), `environment/authblocks.json` (AuthBlocks JWT/email/admin creds). +- `DeepDrftAPI/appsettings.json`: Logging and hosting config. Non-secret upload tunable: `Upload:StagingPath` (default empty → a `staging` subdirectory under the FileDatabase vault path) — the data-disk directory where large audio bodies are staged during upload/replace-audio, kept off the system temp mount (`/tmp` is a small tmpfs on the Linux host); `Startup` also points the framework's multipart buffer here via `ASPNETCORE_TEMP`. Secrets loaded from `environment/filedatabase.json` (FileDatabase vault path), `environment/apikey.json` (API key), `environment/connections.json` (SQL and Auth connection strings), `environment/authblocks.json` (AuthBlocks JWT/email/admin creds). ## Folder-Level Guidance diff --git a/DeepDrftAPI/CLAUDE.md b/DeepDrftAPI/CLAUDE.md index bcffa0c..21db5c7 100644 --- a/DeepDrftAPI/CLAUDE.md +++ b/DeepDrftAPI/CLAUDE.md @@ -168,8 +168,8 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele - `medium` (string, optional): enum `ReleaseMedium` (e.g., `Cut`, `Mix`, `Session`). Defaults to `Cut` if null or unrecognized. - `trackNumber` (int?, optional): track position within the release (1-based). Defaults to 1 if ≤ 0 or null. - `releaseId` (long?, optional): the SQL release ID to attach this track to. Omit (null) on the first row of a submit — this is the **CREATE path**, which mints a new release and blocks a pre-existing (title, artist) with 409. Set to the release id returned by row 1 for rows 2..N of a within-batch multi-track Cut — this is the **ATTACH path**, which skips the (title, artist) pre-existing check and attaches directly to the already-created release after validating the id matches the natural key. The upload form is create-only; appending to a pre-existing release must go through the edit tools. -- The upload stream is copied to a temp file under `Path.GetTempPath()` with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The temp file is always deleted in a `finally` block — success or failure. -- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the temp file, not buffered in memory. +- The upload stream is copied to a staging file under the **upload staging directory** (resolved from `Upload:StagingPath`, defaulting to a `staging` subdirectory under the FileDatabase vault path — on the data disk, **never** `Path.GetTempPath()`) with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The staging file is always deleted in a `finally` block — success or failure. The framework's own multipart file-section buffer is relocated off the system temp mount too: `Startup.ConfigureDomainServices` sets the `ASPNETCORE_TEMP` env var to the same staging directory, so neither on-disk copy of a large body lands on `/tmp` (a small RAM-backed tmpfs on the Linux host). +- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the staging file, not buffered in memory. - `UnifiedTrackService.UploadAsync` orchestrates: release resolution (CREATE or ATTACH, see above) → `TrackContentService.AddTrackAsync` (format-agnostic vault write via router) → `TrackManager` (SQL persist with `createdByUserId`). Release resolution runs the cardinality guard on both paths and, on the CREATE path, calls `ITrackService.FindOrCreateRelease` (returns `(ReleaseDto Release, bool WasCreated)`); if `WasCreated` is false, a concurrent upload won the race and the request is rejected as a duplicate rather than silently attaching. - Returns 200 with the **persisted** `TrackDto` JSON (Id populated) on success. Returns 400 for missing/invalid form fields or unsupported audio format. Returns 409 for two distinct domain conditions: a pre-existing (title, artist) duplicate on the CREATE path (`DUPLICATE_RELEASE:` marker → 409 Conflict), or a track-number conflict within the release (`CARDINALITY_VIOLATION:` marker → 409 Conflict). Returns 500 if processing fails. @@ -189,7 +189,7 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele - **Header `ApiKey`**: required. Validated by `ApiKeyAuthenticationMiddleware`. - **Route parameter `id`** (long): the SQL track ID. - **Form field `audioFile`** (`IFormFile`, required): the replacement audio bytes. File name must end in `.wav`, `.mp3`, or `.flac`. -- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a temp file (correct extension preserved for the audio processor), always deleted in a `finally` block. +- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a staging file under the upload staging directory (the same off-`/tmp` data-disk location as the upload path; correct extension preserved for the audio processor), always deleted in a `finally` block. - Calls `UnifiedTrackService.ReplaceAudioAsync`, which: looks up SQL row by id → calls `TrackContentService.ReplaceTrackAudioAsync(entryKey, tempFilePath)` (registers new audio under the existing `EntryKey`; removes the stale backing file only on a cross-format swap, after the new write succeeds) → regenerates both waveform datums (best-effort; a datum failure is logged and swallowed) → writes the new audio's duration to `DurationSeconds` via `ITrackService.SetDuration` (unconditional overwrite; a failure is surfaced, not swallowed, to prevent derived aggregates like `MixRuntimeSeconds` from silently going stale). - Returns 200 on success. Returns 400 if the file is missing or the format is unsupported. Returns 404 if the track id is not found. Returns 500 if vault processing fails. diff --git a/DeepDrftAPI/Controllers/TrackController.cs b/DeepDrftAPI/Controllers/TrackController.cs index 41360ae..015c11f 100644 --- a/DeepDrftAPI/Controllers/TrackController.cs +++ b/DeepDrftAPI/Controllers/TrackController.cs @@ -20,6 +20,7 @@ public class TrackController : ControllerBase private readonly UnifiedTrackService _unifiedService; private readonly ITrackService _sqlTrackService; private readonly WaveformProfileService _waveformProfileService; + private readonly UploadStagingDirectory _stagingDirectory; private readonly ILogger _logger; // FileDatabase is injected directly for PutTrack because that endpoint receives a pre-processed @@ -34,6 +35,7 @@ public class TrackController : ControllerBase UnifiedTrackService unifiedService, ITrackService sqlTrackService, WaveformProfileService waveformProfileService, + UploadStagingDirectory stagingDirectory, ILogger logger) { _trackContentService = trackContentService; @@ -41,9 +43,47 @@ public class TrackController : ControllerBase _unifiedService = unifiedService; _sqlTrackService = sqlTrackService; _waveformProfileService = waveformProfileService; + _stagingDirectory = stagingDirectory; _logger = logger; } + // Streams an uploaded audio body to a freshly-named staging file on the data disk, preserving the + // validated extension (the processor router selects by extension and reads from disk; .tmp would be + // rejected). Staging lives under UploadStagingDirectory, never Path.GetTempPath() — on the Linux + // host /tmp is a small tmpfs that cannot hold a large WAV. Returns the staging path; the caller + // owns deletion in a finally block. + private async Task StageUploadAsync( + IFormFile audioFile, string uploadExtension, CancellationToken cancellationToken) + { + var stagingPath = Path.Combine( + _stagingDirectory.Path, Guid.NewGuid().ToString("N") + uploadExtension); + + await using var stagingStream = new FileStream( + stagingPath, FileMode.CreateNew, FileAccess.Write, FileShare.None, + bufferSize: 81920, useAsync: true); + await using var uploadStream = audioFile.OpenReadStream(); + await uploadStream.CopyToAsync(stagingStream, cancellationToken); + + return stagingPath; + } + + // Best-effort removal of a staging file. Logs and swallows — a stranded staging file is a + // disk-hygiene concern, not a request failure. + private void DeleteStagingFile(string stagingPath) + { + try + { + if (System.IO.File.Exists(stagingPath)) + { + System.IO.File.Delete(stagingPath); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Failed to delete staging file {StagingPath}", stagingPath); + } + } + // --- Literal-segment routes first --- // These are declared before the parameterized "{trackId}" / "{id:long}" actions so route // resolution never treats "page", "upload", or "meta" as a trackId. @@ -319,23 +359,13 @@ public class TrackController : ControllerBase var resolvedTrackNumber = trackNumber is > 0 ? trackNumber.Value : 1; - // The processor router selects by extension and reads from disk, so the temp file must carry - // the upload's real extension. Path.GetTempFileName() yields .tmp, which the router rejects — - // generate our own path preserving the validated .wav/.mp3/.flac extension. - var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension); - + string? stagingPath = null; try { - await using (var tempStream = new FileStream( - tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None, - bufferSize: 81920, useAsync: true)) - await using (var uploadStream = audioFile.OpenReadStream()) - { - await uploadStream.CopyToAsync(tempStream, cancellationToken); - } + stagingPath = await StageUploadAsync(audioFile, uploadExtension, cancellationToken); var result = await _unifiedService.UploadAsync( - tempPath, + stagingPath, trackName, artist, string.IsNullOrWhiteSpace(album) ? null : album, @@ -381,16 +411,9 @@ public class TrackController : ControllerBase } finally { - try + if (stagingPath is not null) { - if (System.IO.File.Exists(tempPath)) - { - System.IO.File.Delete(tempPath); - } - } - catch (Exception ex) - { - _logger.LogWarning(ex, "UploadTrack: failed to delete temp file {TempPath}", tempPath); + DeleteStagingFile(stagingPath); } } } @@ -567,21 +590,12 @@ public class TrackController : ControllerBase return BadRequest("Uploaded file must have a .wav, .mp3, or .flac extension"); } - // The processor router selects by extension and reads from disk, so the temp file must carry - // the upload's real extension. Mirrors UploadTrack — Path.GetTempFileName() yields .tmp. - var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension); - + string? stagingPath = null; try { - await using (var tempStream = new FileStream( - tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None, - bufferSize: 81920, useAsync: true)) - await using (var uploadStream = audioFile.OpenReadStream()) - { - await uploadStream.CopyToAsync(tempStream, cancellationToken); - } + stagingPath = await StageUploadAsync(audioFile, uploadExtension, cancellationToken); - var result = await _unifiedService.ReplaceAudioAsync(id, tempPath, cancellationToken); + var result = await _unifiedService.ReplaceAudioAsync(id, stagingPath, cancellationToken); if (result.Success) { _logger.LogInformation("ReplaceAudio succeeded: id={Id}", id); @@ -604,16 +618,9 @@ public class TrackController : ControllerBase } finally { - try + if (stagingPath is not null) { - if (System.IO.File.Exists(tempPath)) - { - System.IO.File.Delete(tempPath); - } - } - catch (Exception ex) - { - _logger.LogWarning(ex, "ReplaceAudio: failed to delete temp file {TempPath}", tempPath); + DeleteStagingFile(stagingPath); } } } diff --git a/DeepDrftAPI/Models/UploadSettings.cs b/DeepDrftAPI/Models/UploadSettings.cs new file mode 100644 index 0000000..562f537 --- /dev/null +++ b/DeepDrftAPI/Models/UploadSettings.cs @@ -0,0 +1,13 @@ +namespace DeepDrftAPI.Models +{ + /// + /// Non-secret upload tunables. is the directory used to stage the raw + /// audio body during upload/replace-audio. It must live on the data disk, never the system temp + /// mount (on the Linux host /tmp is a small RAM-backed tmpfs that cannot hold a multi-hundred-MB + /// WAV). When null/empty it defaults to a "staging" subdirectory under the FileDatabase vault path. + /// + public class UploadSettings + { + public string? StagingPath { get; set; } + } +} diff --git a/DeepDrftAPI/Models/UploadStagingDirectory.cs b/DeepDrftAPI/Models/UploadStagingDirectory.cs new file mode 100644 index 0000000..bb3a6ff --- /dev/null +++ b/DeepDrftAPI/Models/UploadStagingDirectory.cs @@ -0,0 +1,10 @@ +namespace DeepDrftAPI.Models +{ + /// + /// The resolved, on-disk staging directory for upload/replace-audio bodies. Resolved once at + /// startup from (or the vault path default) and guaranteed to exist. + /// Injected into TrackController so the upload path never stages on the system temp mount. + /// A typed wrapper rather than a bare string so DI resolves it unambiguously. + /// + public sealed record UploadStagingDirectory(string Path); +} diff --git a/DeepDrftAPI/Startup.cs b/DeepDrftAPI/Startup.cs index 8bbd520..f735fd1 100644 --- a/DeepDrftAPI/Startup.cs +++ b/DeepDrftAPI/Startup.cs @@ -47,9 +47,41 @@ namespace DeepDrftAPI return db; }); + // Upload staging directory. Large audio bodies (multi-hundred-MB WAVs) must never stage on + // the system temp mount — on the Linux host /tmp is a small RAM-backed tmpfs. We move BOTH + // on-disk copies of an upload off /tmp onto the data disk: + // Layer 1 — the framework's multipart file-section buffer (FileBufferingReadStream), which + // reads its directory from the ASPNETCORE_TEMP env var (falling back to + // Path.GetTempPath()). Setting the var here, before the host runs, relocates it. + // Layer 2 — the controller's own staging file, via the injected UploadStagingDirectory. + // Default location is a "staging" subdirectory beside the vaults; override with + // Upload:StagingPath in appsettings.json. + var uploadSettings = builder.Configuration.GetSection("Upload").Get(); + var stagingPath = ResolveStagingPath(uploadSettings?.StagingPath, vaultPath); + Directory.CreateDirectory(stagingPath); + + // AspNetCoreTempDirectory caches this value on first read and throws if the directory is + // absent, so set it (and create the dir) before any request is served. + Environment.SetEnvironmentVariable("ASPNETCORE_TEMP", stagingPath); + builder.Services.AddSingleton(new UploadStagingDirectory(stagingPath)); + return Task.CompletedTask; } + /// + /// Resolves the absolute upload-staging directory. An explicit + /// (from Upload:StagingPath) wins; otherwise it defaults to a staging subdirectory + /// under — on the data disk, never the system temp mount. Pure so + /// the "never /tmp" invariant is unit-testable without standing up the host. + /// + public static string ResolveStagingPath(string? configuredPath, string vaultPath) + { + var path = string.IsNullOrWhiteSpace(configuredPath) + ? Path.Combine(vaultPath, "staging") + : configuredPath; + return Path.GetFullPath(path); + } + private static async Task InitializeTrackVault(FileDatabase fileDatabase) { if (!fileDatabase.HasVault(VaultConstants.Tracks)) diff --git a/DeepDrftAPI/appsettings.json b/DeepDrftAPI/appsettings.json index 08240ca..c2c309b 100644 --- a/DeepDrftAPI/appsettings.json +++ b/DeepDrftAPI/appsettings.json @@ -7,6 +7,9 @@ } }, "AllowedHosts": "*", + "Upload": { + "StagingPath": "" + }, "CorsSettings": { "AllowedOrigins": [ "https://localhost:12778", diff --git a/DeepDrftTests/UploadStagingPathTests.cs b/DeepDrftTests/UploadStagingPathTests.cs new file mode 100644 index 0000000..b55135b --- /dev/null +++ b/DeepDrftTests/UploadStagingPathTests.cs @@ -0,0 +1,59 @@ +using DeepDrftAPI; + +namespace DeepDrftTests; + +/// +/// Guards the upload-staging directory resolution (). The +/// load-bearing invariant: large audio bodies must stage on the data disk, never the system temp +/// mount — on the Linux host /tmp is a small RAM-backed tmpfs that cannot hold a multi-hundred-MB WAV. +/// +[TestFixture] +public class UploadStagingPathTests +{ + [Test] + public void ResolveStagingPath_DefaultsToStagingUnderVault_WhenUnconfigured() + { + var vaultPath = Path.Combine(Path.GetTempPath(), "DeepDrftTests", Guid.NewGuid().ToString()); + + foreach (var configured in new[] { null, "", " " }) + { + var resolved = Startup.ResolveStagingPath(configured, vaultPath); + + Assert.Multiple(() => + { + Assert.That(resolved, Is.EqualTo(Path.GetFullPath(Path.Combine(vaultPath, "staging"))), + "An unset/blank StagingPath must default to a 'staging' subdirectory under the vault path"); + Assert.That(Path.IsPathFullyQualified(resolved), Is.True, + "The resolved staging path must be absolute"); + }); + } + } + + [Test] + public void ResolveStagingPath_HonoursExplicitOverride() + { + var vaultPath = Path.Combine("data", "vaults"); + var configured = Path.Combine(Path.GetTempPath(), "DeepDrftTests", "custom-staging", Guid.NewGuid().ToString()); + + var resolved = Startup.ResolveStagingPath(configured, vaultPath); + + Assert.That(resolved, Is.EqualTo(Path.GetFullPath(configured)), + "An explicit Upload:StagingPath must win over the vault-path default"); + } + + [Test] + public void ResolveStagingPath_NeverResolvesIntoSystemTempDirectory_ForDataDiskVault() + { + // A production-shaped vault path on the data disk (the real config is a relative "../Database/Vaults"). + // The resolved staging dir must sit under that vault, not under Path.GetTempPath() (= /tmp on Linux). + var vaultPath = Path.Combine("..", "Database", "Vaults"); + + var resolved = Startup.ResolveStagingPath(configuredPath: null, vaultPath); + + var systemTemp = Path.GetFullPath(Path.GetTempPath()); + Assert.That(resolved.StartsWith(systemTemp, StringComparison.Ordinal), Is.False, + "The default staging directory must never live under the system temp mount"); + Assert.That(resolved, Does.EndWith(Path.Combine("Database", "Vaults", "staging")), + "The default staging directory must hang off the vault path on the data disk"); + } +}