fix: stage audio uploads on data disk instead of /tmp
Relocate both the framework multipart buffer (via ASPNETCORE_TEMP) and the controller staging file to a configurable data-disk directory, so large WAV/FLAC/MP3 uploads no longer fail on the host's small tmpfs.
This commit is contained in:
@@ -168,8 +168,8 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele
|
||||
- `medium` (string, optional): enum `ReleaseMedium` (e.g., `Cut`, `Mix`, `Session`). Defaults to `Cut` if null or unrecognized.
|
||||
- `trackNumber` (int?, optional): track position within the release (1-based). Defaults to 1 if ≤ 0 or null.
|
||||
- `releaseId` (long?, optional): the SQL release ID to attach this track to. Omit (null) on the first row of a submit — this is the **CREATE path**, which mints a new release and blocks a pre-existing (title, artist) with 409. Set to the release id returned by row 1 for rows 2..N of a within-batch multi-track Cut — this is the **ATTACH path**, which skips the (title, artist) pre-existing check and attaches directly to the already-created release after validating the id matches the natural key. The upload form is create-only; appending to a pre-existing release must go through the edit tools.
|
||||
- The upload stream is copied to a temp file under `Path.GetTempPath()` with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The temp file is always deleted in a `finally` block — success or failure.
|
||||
- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the temp file, not buffered in memory.
|
||||
- The upload stream is copied to a staging file under the **upload staging directory** (resolved from `Upload:StagingPath`, defaulting to a `staging` subdirectory under the FileDatabase vault path — on the data disk, **never** `Path.GetTempPath()`) with the appropriate extension (`.wav`, `.mp3`, or `.flac`). The audio processor reads from disk and requires the correct extension for format detection. The staging file is always deleted in a `finally` block — success or failure. The framework's own multipart file-section buffer is relocated off the system temp mount too: `Startup.ConfigureDomainServices` sets the `ASPNETCORE_TEMP` env var to the same staging directory, so neither on-disk copy of a large body lands on `/tmp` (a small RAM-backed tmpfs on the Linux host).
|
||||
- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` lift the per-request ceiling above the framework default (~28 MB) so production-sized files are accepted. The body is streamed to the staging file, not buffered in memory.
|
||||
- `UnifiedTrackService.UploadAsync` orchestrates: release resolution (CREATE or ATTACH, see above) → `TrackContentService.AddTrackAsync` (format-agnostic vault write via router) → `TrackManager` (SQL persist with `createdByUserId`). Release resolution runs the cardinality guard on both paths and, on the CREATE path, calls `ITrackService.FindOrCreateRelease` (returns `(ReleaseDto Release, bool WasCreated)`); if `WasCreated` is false, a concurrent upload won the race and the request is rejected as a duplicate rather than silently attaching.
|
||||
- Returns 200 with the **persisted** `TrackDto` JSON (Id populated) on success. Returns 400 for missing/invalid form fields or unsupported audio format. Returns 409 for two distinct domain conditions: a pre-existing (title, artist) duplicate on the CREATE path (`DUPLICATE_RELEASE:` marker → 409 Conflict), or a track-number conflict within the release (`CARDINALITY_VIOLATION:` marker → 409 Conflict). Returns 500 if processing fails.
|
||||
|
||||
@@ -189,7 +189,7 @@ Soft-delete a release row. Used by the albums browser to remove an orphaned rele
|
||||
- **Header `ApiKey`**: required. Validated by `ApiKeyAuthenticationMiddleware`.
|
||||
- **Route parameter `id`** (long): the SQL track ID.
|
||||
- **Form field `audioFile`** (`IFormFile`, required): the replacement audio bytes. File name must end in `.wav`, `.mp3`, or `.flac`.
|
||||
- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a temp file (correct extension preserved for the audio processor), always deleted in a `finally` block.
|
||||
- `[RequestSizeLimit(~1.86 GB / 2_000_000_000)]` + `[RequestFormLimits(MultipartBodyLengthLimit = 2_000_000_000)]` mirror the upload ceiling. The body is streamed to a staging file under the upload staging directory (the same off-`/tmp` data-disk location as the upload path; correct extension preserved for the audio processor), always deleted in a `finally` block.
|
||||
- Calls `UnifiedTrackService.ReplaceAudioAsync`, which: looks up SQL row by id → calls `TrackContentService.ReplaceTrackAudioAsync(entryKey, tempFilePath)` (registers new audio under the existing `EntryKey`; removes the stale backing file only on a cross-format swap, after the new write succeeds) → regenerates both waveform datums (best-effort; a datum failure is logged and swallowed) → writes the new audio's duration to `DurationSeconds` via `ITrackService.SetDuration` (unconditional overwrite; a failure is surfaced, not swallowed, to prevent derived aggregates like `MixRuntimeSeconds` from silently going stale).
|
||||
- Returns 200 on success. Returns 400 if the file is missing or the format is unsupported. Returns 404 if the track id is not found. Returns 500 if vault processing fails.
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ public class TrackController : ControllerBase
|
||||
private readonly UnifiedTrackService _unifiedService;
|
||||
private readonly ITrackService _sqlTrackService;
|
||||
private readonly WaveformProfileService _waveformProfileService;
|
||||
private readonly UploadStagingDirectory _stagingDirectory;
|
||||
private readonly ILogger<TrackController> _logger;
|
||||
|
||||
// FileDatabase is injected directly for PutTrack because that endpoint receives a pre-processed
|
||||
@@ -34,6 +35,7 @@ public class TrackController : ControllerBase
|
||||
UnifiedTrackService unifiedService,
|
||||
ITrackService sqlTrackService,
|
||||
WaveformProfileService waveformProfileService,
|
||||
UploadStagingDirectory stagingDirectory,
|
||||
ILogger<TrackController> logger)
|
||||
{
|
||||
_trackContentService = trackContentService;
|
||||
@@ -41,9 +43,47 @@ public class TrackController : ControllerBase
|
||||
_unifiedService = unifiedService;
|
||||
_sqlTrackService = sqlTrackService;
|
||||
_waveformProfileService = waveformProfileService;
|
||||
_stagingDirectory = stagingDirectory;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
// Streams an uploaded audio body to a freshly-named staging file on the data disk, preserving the
|
||||
// validated extension (the processor router selects by extension and reads from disk; .tmp would be
|
||||
// rejected). Staging lives under UploadStagingDirectory, never Path.GetTempPath() — on the Linux
|
||||
// host /tmp is a small tmpfs that cannot hold a large WAV. Returns the staging path; the caller
|
||||
// owns deletion in a finally block.
|
||||
private async Task<string> StageUploadAsync(
|
||||
IFormFile audioFile, string uploadExtension, CancellationToken cancellationToken)
|
||||
{
|
||||
var stagingPath = Path.Combine(
|
||||
_stagingDirectory.Path, Guid.NewGuid().ToString("N") + uploadExtension);
|
||||
|
||||
await using var stagingStream = new FileStream(
|
||||
stagingPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
|
||||
bufferSize: 81920, useAsync: true);
|
||||
await using var uploadStream = audioFile.OpenReadStream();
|
||||
await uploadStream.CopyToAsync(stagingStream, cancellationToken);
|
||||
|
||||
return stagingPath;
|
||||
}
|
||||
|
||||
// Best-effort removal of a staging file. Logs and swallows — a stranded staging file is a
|
||||
// disk-hygiene concern, not a request failure.
|
||||
private void DeleteStagingFile(string stagingPath)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (System.IO.File.Exists(stagingPath))
|
||||
{
|
||||
System.IO.File.Delete(stagingPath);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to delete staging file {StagingPath}", stagingPath);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Literal-segment routes first ---
|
||||
// These are declared before the parameterized "{trackId}" / "{id:long}" actions so route
|
||||
// resolution never treats "page", "upload", or "meta" as a trackId.
|
||||
@@ -319,23 +359,13 @@ public class TrackController : ControllerBase
|
||||
|
||||
var resolvedTrackNumber = trackNumber is > 0 ? trackNumber.Value : 1;
|
||||
|
||||
// The processor router selects by extension and reads from disk, so the temp file must carry
|
||||
// the upload's real extension. Path.GetTempFileName() yields .tmp, which the router rejects —
|
||||
// generate our own path preserving the validated .wav/.mp3/.flac extension.
|
||||
var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension);
|
||||
|
||||
string? stagingPath = null;
|
||||
try
|
||||
{
|
||||
await using (var tempStream = new FileStream(
|
||||
tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
|
||||
bufferSize: 81920, useAsync: true))
|
||||
await using (var uploadStream = audioFile.OpenReadStream())
|
||||
{
|
||||
await uploadStream.CopyToAsync(tempStream, cancellationToken);
|
||||
}
|
||||
stagingPath = await StageUploadAsync(audioFile, uploadExtension, cancellationToken);
|
||||
|
||||
var result = await _unifiedService.UploadAsync(
|
||||
tempPath,
|
||||
stagingPath,
|
||||
trackName,
|
||||
artist,
|
||||
string.IsNullOrWhiteSpace(album) ? null : album,
|
||||
@@ -381,16 +411,9 @@ public class TrackController : ControllerBase
|
||||
}
|
||||
finally
|
||||
{
|
||||
try
|
||||
if (stagingPath is not null)
|
||||
{
|
||||
if (System.IO.File.Exists(tempPath))
|
||||
{
|
||||
System.IO.File.Delete(tempPath);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "UploadTrack: failed to delete temp file {TempPath}", tempPath);
|
||||
DeleteStagingFile(stagingPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -567,21 +590,12 @@ public class TrackController : ControllerBase
|
||||
return BadRequest("Uploaded file must have a .wav, .mp3, or .flac extension");
|
||||
}
|
||||
|
||||
// The processor router selects by extension and reads from disk, so the temp file must carry
|
||||
// the upload's real extension. Mirrors UploadTrack — Path.GetTempFileName() yields .tmp.
|
||||
var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + uploadExtension);
|
||||
|
||||
string? stagingPath = null;
|
||||
try
|
||||
{
|
||||
await using (var tempStream = new FileStream(
|
||||
tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.None,
|
||||
bufferSize: 81920, useAsync: true))
|
||||
await using (var uploadStream = audioFile.OpenReadStream())
|
||||
{
|
||||
await uploadStream.CopyToAsync(tempStream, cancellationToken);
|
||||
}
|
||||
stagingPath = await StageUploadAsync(audioFile, uploadExtension, cancellationToken);
|
||||
|
||||
var result = await _unifiedService.ReplaceAudioAsync(id, tempPath, cancellationToken);
|
||||
var result = await _unifiedService.ReplaceAudioAsync(id, stagingPath, cancellationToken);
|
||||
if (result.Success)
|
||||
{
|
||||
_logger.LogInformation("ReplaceAudio succeeded: id={Id}", id);
|
||||
@@ -604,16 +618,9 @@ public class TrackController : ControllerBase
|
||||
}
|
||||
finally
|
||||
{
|
||||
try
|
||||
if (stagingPath is not null)
|
||||
{
|
||||
if (System.IO.File.Exists(tempPath))
|
||||
{
|
||||
System.IO.File.Delete(tempPath);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "ReplaceAudio: failed to delete temp file {TempPath}", tempPath);
|
||||
DeleteStagingFile(stagingPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
namespace DeepDrftAPI.Models
|
||||
{
|
||||
/// <summary>
|
||||
/// Non-secret upload tunables. <see cref="StagingPath"/> is the directory used to stage the raw
|
||||
/// audio body during upload/replace-audio. It must live on the data disk, never the system temp
|
||||
/// mount (on the Linux host <c>/tmp</c> is a small RAM-backed tmpfs that cannot hold a multi-hundred-MB
|
||||
/// WAV). When null/empty it defaults to a "staging" subdirectory under the FileDatabase vault path.
|
||||
/// </summary>
|
||||
public class UploadSettings
|
||||
{
|
||||
public string? StagingPath { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
namespace DeepDrftAPI.Models
|
||||
{
|
||||
/// <summary>
|
||||
/// The resolved, on-disk staging directory for upload/replace-audio bodies. Resolved once at
|
||||
/// startup from <see cref="UploadSettings"/> (or the vault path default) and guaranteed to exist.
|
||||
/// Injected into <c>TrackController</c> so the upload path never stages on the system temp mount.
|
||||
/// A typed wrapper rather than a bare string so DI resolves it unambiguously.
|
||||
/// </summary>
|
||||
public sealed record UploadStagingDirectory(string Path);
|
||||
}
|
||||
@@ -47,9 +47,41 @@ namespace DeepDrftAPI
|
||||
return db;
|
||||
});
|
||||
|
||||
// Upload staging directory. Large audio bodies (multi-hundred-MB WAVs) must never stage on
|
||||
// the system temp mount — on the Linux host /tmp is a small RAM-backed tmpfs. We move BOTH
|
||||
// on-disk copies of an upload off /tmp onto the data disk:
|
||||
// Layer 1 — the framework's multipart file-section buffer (FileBufferingReadStream), which
|
||||
// reads its directory from the ASPNETCORE_TEMP env var (falling back to
|
||||
// Path.GetTempPath()). Setting the var here, before the host runs, relocates it.
|
||||
// Layer 2 — the controller's own staging file, via the injected UploadStagingDirectory.
|
||||
// Default location is a "staging" subdirectory beside the vaults; override with
|
||||
// Upload:StagingPath in appsettings.json.
|
||||
var uploadSettings = builder.Configuration.GetSection("Upload").Get<UploadSettings>();
|
||||
var stagingPath = ResolveStagingPath(uploadSettings?.StagingPath, vaultPath);
|
||||
Directory.CreateDirectory(stagingPath);
|
||||
|
||||
// AspNetCoreTempDirectory caches this value on first read and throws if the directory is
|
||||
// absent, so set it (and create the dir) before any request is served.
|
||||
Environment.SetEnvironmentVariable("ASPNETCORE_TEMP", stagingPath);
|
||||
builder.Services.AddSingleton(new UploadStagingDirectory(stagingPath));
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the absolute upload-staging directory. An explicit <paramref name="configuredPath"/>
|
||||
/// (from <c>Upload:StagingPath</c>) wins; otherwise it defaults to a <c>staging</c> subdirectory
|
||||
/// under <paramref name="vaultPath"/> — on the data disk, never the system temp mount. Pure so
|
||||
/// the "never <c>/tmp</c>" invariant is unit-testable without standing up the host.
|
||||
/// </summary>
|
||||
public static string ResolveStagingPath(string? configuredPath, string vaultPath)
|
||||
{
|
||||
var path = string.IsNullOrWhiteSpace(configuredPath)
|
||||
? Path.Combine(vaultPath, "staging")
|
||||
: configuredPath;
|
||||
return Path.GetFullPath(path);
|
||||
}
|
||||
|
||||
private static async Task InitializeTrackVault(FileDatabase fileDatabase)
|
||||
{
|
||||
if (!fileDatabase.HasVault(VaultConstants.Tracks))
|
||||
|
||||
@@ -7,6 +7,9 @@
|
||||
}
|
||||
},
|
||||
"AllowedHosts": "*",
|
||||
"Upload": {
|
||||
"StagingPath": ""
|
||||
},
|
||||
"CorsSettings": {
|
||||
"AllowedOrigins": [
|
||||
"https://localhost:12778",
|
||||
|
||||
Reference in New Issue
Block a user