Merge upload-temp-disk-fix into dev (stage large audio uploads on data disk instead of /tmp)

CLAUDE.md

@@ -126,7 +126,7 @@ All projects load secrets via `CredentialTools.ResolvePathOrThrow()` from gitign
 
 - `DeepDrftPublic/appsettings.json`: Logging and URL config. Secrets loaded from `environment/api.json` (DeepDrftAPI base URL via `Api:ContentApiUrl`).
 - `DeepDrftManager/appsettings.json`: Logging and URL config. Secrets loaded from `environment/api.json` (DeepDrftAPI base URL via `Api:ContentApiUrl` and API key via `Api:ContentApiKey`). Non-secret upload tunables (in `appsettings.json` itself, not `environment/`): `Upload:IdleTimeoutSeconds` (default 90 — aborts a stalled body-streaming phase) and `Upload:ResponseTimeoutSeconds` (default 1200 — budget for server-side persist after the body is fully sent).
-- `DeepDrftAPI/appsettings.json`: Logging and hosting config. Secrets loaded from `environment/filedatabase.json` (FileDatabase vault path), `environment/apikey.json` (API key), `environment/connections.json` (SQL and Auth connection strings), `environment/authblocks.json` (AuthBlocks JWT/email/admin creds).
+- `DeepDrftAPI/appsettings.json`: Logging and hosting config. Non-secret upload tunable: `Upload:StagingPath` (default empty → a `staging` subdirectory under the FileDatabase vault path) — the data-disk directory where large audio bodies are staged during upload/replace-audio, kept off the system temp mount (`/tmp` is a small tmpfs on the Linux host); `Startup` also points the framework's multipart buffer here via `ASPNETCORE_TEMP`. Secrets loaded from `environment/filedatabase.json` (FileDatabase vault path), `environment/apikey.json` (API key), `environment/connections.json` (SQL and Auth connection strings), `environment/authblocks.json` (AuthBlocks JWT/email/admin creds).
 
 ## Folder-Level Guidance
 

DeepDrftAPI/CLAUDE.md

@@ -168,8 +168,8 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele
   - `medium` (string, optional): enum `ReleaseMedium` (e.g., `Cut`, `Mix`, `Session`). Defaults to `Cut` if null or unrecognized.
   - `trackNumber` (int?, optional): track position within the release (1-based). Defaults to 1 if ≤ 0 or null.
   - `releaseId` (long?, optional): the SQL release ID to attach this track to. Omit (null) on the first row of a submit — this is the **CREATE path**, which mints a new release and blocks a pre-existing (title, artist) with 409. Set to the release id returned by row 1 for rows 2..N of a within-batch multi-track Cut — this is the **ATTACH path**, which skips the (title, artist) pre-existing check and attaches directly to the already-created release after validating the id matches the natural key. The upload form is create-only; appending to a pre-existing release must go through the edit tools.
-- The upload stream is copied to a temp file under `Path.GetTempPath()` with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The temp file is always deleted in a `finally` block — success or failure.
+- The upload stream is copied to a staging file under the **upload staging directory** (resolved from `Upload:StagingPath`, defaulting to a `staging` subdirectory under the FileDatabase vault path — on the data disk, **never** `Path.GetTempPath()`) with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The staging file is always deleted in a `finally` block — success or failure. The framework's own multipart file-section buffer is relocated off the system temp mount too: `Startup.ConfigureDomainServices` sets the `ASPNETCORE_TEMP` env var to the same staging directory, so neither on-disk copy of a large body lands on `/tmp` (a small RAM-backed tmpfs on the Linux host).
-- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the temp file, not buffered in memory.
+- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the staging file, not buffered in memory.
 - `UnifiedTrackService.UploadAsync` orchestrates: release resolution (CREATE or ATTACH, see above) → `TrackContentService.AddTrackAsync` (format-agnostic vault write via router) → `TrackManager` (SQL persist with `createdByUserId`). Release resolution runs the cardinality guard on both paths and, on the CREATE path, calls `ITrackService.FindOrCreateRelease` (returns `(ReleaseDto Release, bool WasCreated)`); if `WasCreated` is false, a concurrent upload won the race and the request is rejected as a duplicate rather than silently attaching.
 - Returns 200 with the **persisted** `TrackDto` JSON (Id populated) on success. Returns 400 for missing/invalid form fields or unsupported audio format. Returns 409 for two distinct domain conditions: a pre-existing (title, artist) duplicate on the CREATE path (`DUPLICATE_RELEASE:` marker → 409 Conflict), or a track-number conflict within the release (`CARDINALITY_VIOLATION:` marker → 409 Conflict). Returns 500 if processing fails.
 
@@ -189,7 +189,7 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele
 - **Header `ApiKey`**: required. Validated by `ApiKeyAuthenticationMiddleware`.
 - **Route parameter `id`** (long): the SQL track ID.
 - **Form field `audioFile`** (`IFormFile`, required): the replacement audio bytes. File name must end in `.wav`, `.mp3`, or `.flac`.
-- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a temp file (correct extension preserved for the audio processor), always deleted in a `finally` block.
+- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a staging file under the upload staging directory (the same off-`/tmp` data-disk location as the upload path; correct extension preserved for the audio processor), always deleted in a `finally` block.
 - Calls `UnifiedTrackService.ReplaceAudioAsync`, which: looks up SQL row by id → calls `TrackContentService.ReplaceTrackAudioAsync(entryKey, tempFilePath)` (registers new audio under the existing `EntryKey`; removes the stale backing file only on a cross-format swap, after the new write succeeds) → regenerates both waveform datums (best-effort; a datum failure is logged and swallowed) → writes the new audio's duration to `DurationSeconds` via `ITrackService.SetDuration` (unconditional overwrite; a failure is surfaced, not swallowed, to prevent derived aggregates like `MixRuntimeSeconds` from silently going stale).
 - Returns 200 on success. Returns 400 if the file is missing or the format is unsupported. Returns 404 if the track id is not found. Returns 500 if vault processing fails.
 

DeepDrftAPI/Controllers/TrackController.cs

@@ -20,6 +20,7 @@ public class TrackController : ControllerBase
     private readonly UnifiedTrackService _unifiedService;
     private readonly ITrackService _sqlTrackService;
     private readonly WaveformProfileService _waveformProfileService;
+    private readonly UploadStagingDirectory _stagingDirectory;
     private readonly ILogger<TrackController> _logger;
 
     // FileDatabase is injected directly for PutTrack because that endpoint receives a pre-processed
@@ -34,6 +35,7 @@ public class TrackController : ControllerBase
         UnifiedTrackService unifiedService,
         ITrackService sqlTrackService,
         WaveformProfileService waveformProfileService,
+        UploadStagingDirectory stagingDirectory,
         ILogger<TrackController> logger)
     {
         _trackContentService = trackContentService;
@@ -41,9 +43,48 @@ public class TrackController : ControllerBase
         _unifiedService = unifiedService;
         _sqlTrackService = sqlTrackService;
         _waveformProfileService = waveformProfileService;
+        _stagingDirectory = stagingDirectory;
         _logger = logger;
     }
 
+    // Builds a unique staging file path on the data disk with the validated extension. The caller MUST
+    // assign this to the local that its finally block guards BEFORE calling StageUploadAsync — that
+    // way a mid-copy abort (OperationCanceledException, IO error) still triggers deletion of the
+    // partially-written file. Staging lives under UploadStagingDirectory, never Path.GetTempPath() —
+    // on the Linux host /tmp is a small tmpfs that cannot hold a large WAV.
+    private string BuildStagingPath(string uploadExtension) =>
+        Path.Combine(_stagingDirectory.Path, Guid.NewGuid().ToString("N") + uploadExtension);
+
+    // Streams an uploaded audio body to the pre-allocated staging path. The caller owns the path and
+    // must delete it in a finally block; separating path generation from the copy ensures the finally
+    // guard fires even when CopyToAsync throws before returning.
+    private async Task StageUploadAsync(
+        IFormFile audioFile, string stagingPath, CancellationToken cancellationToken)
+    {
+        await using var stagingStream = new FileStream(
+            stagingPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
+            bufferSize: 81920, useAsync: true);
+        await using var uploadStream = audioFile.OpenReadStream();
+        await uploadStream.CopyToAsync(stagingStream, cancellationToken);
+    }
+
+    // Best-effort removal of a staging file. Logs and swallows — a stranded staging file is a
+    // disk-hygiene concern, not a request failure.
+    private void DeleteStagingFile(string stagingPath)
+    {
+        try
+        {
+            if (System.IO.File.Exists(stagingPath))
+            {
+                System.IO.File.Delete(stagingPath);
+            }
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex, "Failed to delete staging file {StagingPath}", stagingPath);
+        }
+    }
+
     // --- Literal-segment routes first ---
     // These are declared before the parameterized "{trackId}" / "{id:long}" actions so route
     // resolution never treats "page", "upload", or "meta" as a trackId.
@@ -319,23 +360,15 @@ public class TrackController : ControllerBase
 
         var resolvedTrackNumber = trackNumber is > 0 ? trackNumber.Value : 1;
 
-        // The processor router selects by extension and reads from disk, so the temp file must carry
+        // Build the staging path before the copy so the finally block can delete the partial file
-        // the upload's real extension. Path.GetTempFileName() yields .tmp, which the router rejects —
+        // even if CopyToAsync throws mid-stream (client cancellation, disk-full, IO error).
-        // generate our own path preserving the validated .wav/.mp3/.flac extension.
+        var stagingPath = BuildStagingPath(uploadExtension);
-        var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension);
-
         try
         {
-            await using (var tempStream = new FileStream(
+            await StageUploadAsync(audioFile, stagingPath, cancellationToken);
-                tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
-                bufferSize: 81920, useAsync: true))
-            await using (var uploadStream = audioFile.OpenReadStream())
-            {
-                await uploadStream.CopyToAsync(tempStream, cancellationToken);
-            }
 
             var result = await _unifiedService.UploadAsync(
-                tempPath,
+                stagingPath,
                 trackName,
                 artist,
                 string.IsNullOrWhiteSpace(album) ? null : album,
@@ -381,17 +414,7 @@ public class TrackController : ControllerBase
         }
         finally
         {
-            try
+            DeleteStagingFile(stagingPath);
-            {
-                if (System.IO.File.Exists(tempPath))
-                {
-                    System.IO.File.Delete(tempPath);
-                }
-            }
-            catch (Exception ex)
-            {
-                _logger.LogWarning(ex, "UploadTrack: failed to delete temp file {TempPath}", tempPath);
-            }
         }
     }
 
@@ -567,21 +590,14 @@ public class TrackController : ControllerBase
             return BadRequest("Uploaded file must have a .wav, .mp3, or .flac extension");
         }
 
-        // The processor router selects by extension and reads from disk, so the temp file must carry
+        // Build the staging path before the copy so the finally block can delete the partial file
-        // the upload's real extension. Mirrors UploadTrack — Path.GetTempFileName() yields .tmp.
+        // even if CopyToAsync throws mid-stream (client cancellation, disk-full, IO error).
-        var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension);
+        var stagingPath = BuildStagingPath(uploadExtension);
-
         try
         {
-            await using (var tempStream = new FileStream(
+            await StageUploadAsync(audioFile, stagingPath, cancellationToken);
-                tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
-                bufferSize: 81920, useAsync: true))
-            await using (var uploadStream = audioFile.OpenReadStream())
-            {
-                await uploadStream.CopyToAsync(tempStream, cancellationToken);
-            }
 
-            var result = await _unifiedService.ReplaceAudioAsync(id, tempPath, cancellationToken);
+            var result = await _unifiedService.ReplaceAudioAsync(id, stagingPath, cancellationToken);
             if (result.Success)
             {
                 _logger.LogInformation("ReplaceAudio succeeded: id={Id}", id);
@@ -604,17 +620,7 @@ public class TrackController : ControllerBase
         }
         finally
         {
-            try
+            DeleteStagingFile(stagingPath);
-            {
-                if (System.IO.File.Exists(tempPath))
-                {
-                    System.IO.File.Delete(tempPath);
-                }
-            }
-            catch (Exception ex)
-            {
-                _logger.LogWarning(ex, "ReplaceAudio: failed to delete temp file {TempPath}", tempPath);
-            }
         }
     }
 

DeepDrftAPI/Models/UploadSettings.cs

@@ -0,0 +1,13 @@
+namespace DeepDrftAPI.Models
+{
+    /// <summary>
+    /// Non-secret upload tunables. <see cref="StagingPath"/> is the directory used to stage the raw
+    /// audio body during upload/replace-audio. It must live on the data disk, never the system temp
+    /// mount (on the Linux host <c>/tmp</c> is a small RAM-backed tmpfs that cannot hold a multi-hundred-MB
+    /// WAV). When null/empty it defaults to a "staging" subdirectory under the FileDatabase vault path.
+    /// </summary>
+    public class UploadSettings
+    {
+        public string? StagingPath { get; set; }
+    }
+}

DeepDrftAPI/Models/UploadStagingDirectory.cs

@@ -0,0 +1,10 @@
+namespace DeepDrftAPI.Models
+{
+    /// <summary>
+    /// The resolved, on-disk staging directory for upload/replace-audio bodies. Resolved once at
+    /// startup from <see cref="UploadSettings"/> (or the vault path default) and guaranteed to exist.
+    /// Injected into <c>TrackController</c> so the upload path never stages on the system temp mount.
+    /// A typed wrapper rather than a bare string so DI resolves it unambiguously.
+    /// </summary>
+    public sealed record UploadStagingDirectory(string Path);
+}

DeepDrftAPI/Startup.cs

@@ -47,9 +47,41 @@ namespace DeepDrftAPI
                 return db;
             });
 
+            // Upload staging directory. Large audio bodies (multi-hundred-MB WAVs) must never stage on
+            // the system temp mount — on the Linux host /tmp is a small RAM-backed tmpfs. We move BOTH
+            // on-disk copies of an upload off /tmp onto the data disk:
+            //   Layer 1 — the framework's multipart file-section buffer (FileBufferingReadStream), which
+            //             reads its directory from the ASPNETCORE_TEMP env var (falling back to
+            //             Path.GetTempPath()). Setting the var here, before the host runs, relocates it.
+            //   Layer 2 — the controller's own staging file, via the injected UploadStagingDirectory.
+            // Default location is a "staging" subdirectory beside the vaults; override with
+            // Upload:StagingPath in appsettings.json.
+            var uploadSettings = builder.Configuration.GetSection("Upload").Get<UploadSettings>();
+            var stagingPath = ResolveStagingPath(uploadSettings?.StagingPath, vaultPath);
+            Directory.CreateDirectory(stagingPath);
+
+            // AspNetCoreTempDirectory caches this value on first read and throws if the directory is
+            // absent, so set it (and create the dir) before any request is served.
+            Environment.SetEnvironmentVariable("ASPNETCORE_TEMP", stagingPath);
+            builder.Services.AddSingleton(new UploadStagingDirectory(stagingPath));
+
             return Task.CompletedTask;
         }
 
+        /// <summary>
+        /// Resolves the absolute upload-staging directory. An explicit <paramref name="configuredPath"/>
+        /// (from <c>Upload:StagingPath</c>) wins; otherwise it defaults to a <c>staging</c> subdirectory
+        /// under <paramref name="vaultPath"/> — on the data disk, never the system temp mount. Pure so
+        /// the "never <c>/tmp</c>" invariant is unit-testable without standing up the host.
+        /// </summary>
+        public static string ResolveStagingPath(string? configuredPath, string vaultPath)
+        {
+            var path = string.IsNullOrWhiteSpace(configuredPath)
+                ? Path.Combine(vaultPath, "staging")
+                : configuredPath;
+            return Path.GetFullPath(path);
+        }
+
         private static async Task InitializeTrackVault(FileDatabase fileDatabase)
         {
             if (!fileDatabase.HasVault(VaultConstants.Tracks))

DeepDrftAPI/appsettings.json

@@ -7,6 +7,9 @@
     }
   },
   "AllowedHosts": "*",
+  "Upload": {
+    "StagingPath": ""
+  },
   "CorsSettings": {
     "AllowedOrigins": [
       "https://localhost:12778",

DeepDrftTests/UploadStagingPathTests.cs

@@ -0,0 +1,65 @@
+using DeepDrftAPI;
+
+namespace DeepDrftTests;
+
+/// <summary>
+/// Guards the upload-staging directory resolution (<see cref="Startup.ResolveStagingPath"/>). The
+/// load-bearing invariant: large audio bodies must stage on the data disk, never the system temp
+/// mount — on the Linux host /tmp is a small RAM-backed tmpfs that cannot hold a multi-hundred-MB WAV.
+/// </summary>
+[TestFixture]
+public class UploadStagingPathTests
+{
+    [Test]
+    public void ResolveStagingPath_DefaultsToStagingUnderVault_WhenUnconfigured()
+    {
+        var vaultPath = Path.Combine(Path.GetTempPath(), "DeepDrftTests", Guid.NewGuid().ToString());
+
+        foreach (var configured in new[] { null, "", "   " })
+        {
+            var resolved = Startup.ResolveStagingPath(configured, vaultPath);
+
+            Assert.Multiple(() =>
+            {
+                Assert.That(resolved, Is.EqualTo(Path.GetFullPath(Path.Combine(vaultPath, "staging"))),
+                    "An unset/blank StagingPath must default to a 'staging' subdirectory under the vault path");
+                Assert.That(Path.IsPathFullyQualified(resolved), Is.True,
+                    "The resolved staging path must be absolute");
+            });
+        }
+    }
+
+    [Test]
+    public void ResolveStagingPath_HonoursExplicitOverride()
+    {
+        var vaultPath = Path.Combine("data", "vaults");
+        var configured = Path.Combine(Path.GetTempPath(), "DeepDrftTests", "custom-staging", Guid.NewGuid().ToString());
+
+        var resolved = Startup.ResolveStagingPath(configured, vaultPath);
+
+        Assert.That(resolved, Is.EqualTo(Path.GetFullPath(configured)),
+            "An explicit Upload:StagingPath must win over the vault-path default");
+    }
+
+    [Test]
+    public void ResolveStagingPath_NeverResolvesIntoSystemTempDirectory_ForDataDiskVault()
+    {
+        // A production-shaped vault path on the data disk (the real config is a relative "../Database/Vaults").
+        // The resolved staging dir must sit under that vault, not under Path.GetTempPath() (= /tmp on Linux).
+        var vaultPath = Path.Combine("..", "Database", "Vaults");
+
+        var resolved = Startup.ResolveStagingPath(configuredPath: null, vaultPath);
+
+        var systemTemp = Path.GetFullPath(Path.GetTempPath());
+        // Note: because vaultPath is relative, Path.GetFullPath resolves it against the CWD, which is
+        // never the system temp directory. The StartsWith guard therefore catches the case where
+        // ResolveStagingPath mistakenly uses Path.GetTempPath() directly, rather than proving the
+        // absolute production path never overlaps with /tmp on any machine. The EndsWith assertion
+        // is the load-bearing check: it verifies the output is rooted under the vault tree, not
+        // under a hard-coded temp location.
+        Assert.That(resolved.StartsWith(systemTemp, StringComparison.Ordinal), Is.False,
+            "The default staging directory must never live under the system temp mount");
+        Assert.That(resolved, Does.EndWith(Path.Combine("Database", "Vaults", "staging")),
+            "The default staging directory must hang off the vault path on the data disk");
+    }
+}