import: a .rsbank lands as a new bank, whole or not at all
Four collisions answered explicitly: ids reminted, names never overwritten, content deduped before the write, bank name auto-suffixed. Degraded ledger refuses before the picker.
This commit is contained in:
@@ -73,6 +73,11 @@ landing after the format.
|
||||
here, computed where payloads are streamed (shell). The bank's `slot_map`
|
||||
rides along. Unknown keys skip at every level; duplicate entry names are
|
||||
rejected both ways.
|
||||
- `import_plan` — the pure import decision, and the reason the whole feature is
|
||||
testable without a DAW: the destination bank's display name after
|
||||
`BankBook`'s own fold, the reminted sample ids and remapped parents, and the
|
||||
per-entry land / collapse / rename disposition. Also `importLedgerRefusal`,
|
||||
the import's ledger gate.
|
||||
- `bank_package` — framing and arithmetic composing the two above:
|
||||
`encodePackage` (prefix bytes + layout + total size, stamping this build's
|
||||
ladder pair and `version::stampVersion()`), `decodePackage` (prefix + observed
|
||||
@@ -91,6 +96,11 @@ landing after the format.
|
||||
deliberately NOT enforced by the codec — they are `import_plan` decisions. The
|
||||
codec rejects only what makes the container itself incoherent (duplicate
|
||||
entry names, invalid names, a non-single-sample nested index).
|
||||
- **`import_plan` consults no other bank's hashes, and that is the ruling, not
|
||||
an omission.** An import always creates a NEW bank, so "already present in the
|
||||
destination bank by content" is exactly "already landed by this same plan".
|
||||
Cross-bank dedup is not enforced anywhere (`core/model/CLAUDE.md`), so a hash
|
||||
the pool already holds still lands its own file here.
|
||||
- `requiredPrefixSize` trusts fields beyond the frozen region only when the
|
||||
version pair classifies `Readable`; for `TooNew` it stops at the semver —
|
||||
don't "fix" it to read the manifest length there, a future structural format
|
||||
@@ -133,15 +143,13 @@ landing after the format.
|
||||
collision class as the ASCII case fold, which `sameEntryName` does catch. A
|
||||
table-free fix does not exist, and restricting names to ASCII would be
|
||||
genuinely over-strict for non-English users. Left open knowingly.
|
||||
- **`duplicateName` is O(n²) over `entries` on the decode path** — pre-existing
|
||||
shape (the double loop is unchanged since `af35fc5`; only the comparator
|
||||
changed). Under the `kMaxManifestBytes` cap (64 MB) a minimal entry is
|
||||
~100 bytes, so a hostile package can declare ~670k entries — ~2×10¹¹ pair
|
||||
comparisons, a multi-minute hang on import. It signals an error rather than
|
||||
UB, so the hostile-input invariant above still holds, but it sits against
|
||||
this module's "a forged header cannot demand gigabytes" posture. Forward
|
||||
obligation for `import_plan`: fold this into a sorted vector or hash set
|
||||
when that track lands; not changed here.
|
||||
- **`duplicateName` folds through a hash set, not a pairwise scan.** Under the
|
||||
`kMaxManifestBytes` cap (64 MB) a minimal entry is ~100 bytes, so a hostile
|
||||
package can declare ~670k entries; the former double loop was ~2×10¹¹ pair
|
||||
comparisons — a multi-minute hang on the decode path an import drives. The
|
||||
set is keyed on `entryNameKey`, which is `sameEntryName`'s ASCII-case fold
|
||||
made explicit, so the equivalence rule still has one home (`lowerAscii`).
|
||||
Do not reintroduce the pairwise scan.
|
||||
- **Cross-module contract with `src/shell/package`:** a genuinely zero-length
|
||||
entry cannot round-trip through the filesystem seam there (`appendPayload`
|
||||
refuses an empty payload — an empty buffer signals an upstream read failure,
|
||||
|
||||
@@ -12,3 +12,10 @@ reasampler_pure_library(bank_package
|
||||
LINK PUBLIC package_format package_manifest PRIVATE app_version)
|
||||
# app_version: the tests pin the stamped writer semver against stampVersion().
|
||||
reasampler_test(bank_package LINK bank_package app_version)
|
||||
|
||||
reasampler_pure_library(import_plan
|
||||
SOURCES import_plan.cpp
|
||||
LINK PUBLIC package_manifest bank_book origin_ledger PRIVATE package_format capture_paths)
|
||||
# tracking_authority: the ledger-gate test proves the import does NOT share prune's
|
||||
# composite blocker, which needs the composite to compare against.
|
||||
reasampler_test(import_plan LINK import_plan tracking_authority)
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
#include "core/package/import_plan.h"
|
||||
|
||||
#include <unordered_map>
|
||||
#include <unordered_set>
|
||||
#include <utility>
|
||||
|
||||
#include "core/capture/capture_paths.h"
|
||||
#include "core/package/package_format.h"
|
||||
|
||||
namespace reasampler::package {
|
||||
|
||||
namespace {
|
||||
|
||||
using capture::bankRelativeForName;
|
||||
using capture::deriveBankPaths;
|
||||
using capture::sanitizeStem;
|
||||
|
||||
bool blankName(const std::string& s) {
|
||||
for (char c : s)
|
||||
if (c != ' ' && c != '\t' && c != '\n' && c != '\r') return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// "kick.wav" -> "kick"; a name with no dot is its own stem. deriveBankPaths re-adds
|
||||
// the extension, so handing it the full name would file "kick.wav" as "kick.wav.wav".
|
||||
std::string stemOf(const std::string& fileName) {
|
||||
const std::size_t dot = fileName.rfind('.');
|
||||
if (dot == std::string::npos || dot == 0) return fileName;
|
||||
return fileName.substr(0, dot);
|
||||
}
|
||||
|
||||
// True when `fileName` is already spelled the way this tool spells a bank file, so a
|
||||
// package landing in a fresh project keeps the names it travelled with. Anything else
|
||||
// is minted through deriveBankPaths, which is also the sanitizer.
|
||||
bool spelledLikeABankFile(const std::string& fileName) {
|
||||
const std::string stem = stemOf(fileName);
|
||||
return stem != fileName && sanitizeStem(stem) == stem && fileName == stem + ".wav";
|
||||
}
|
||||
|
||||
// The bank-folder names an import must not land on: what is there already, plus what
|
||||
// this import has minted so far. Case-folded, because the two filesystems this tool
|
||||
// ships on would treat "Kick.wav" and "kick.wav" as one file.
|
||||
class NameSet {
|
||||
public:
|
||||
explicit NameSet(const std::vector<std::string>& present) {
|
||||
keys_.reserve(present.size());
|
||||
for (const std::string& n : present) keys_.insert(entryNameKey(n));
|
||||
}
|
||||
bool taken(const std::string& name) const { return keys_.count(entryNameKey(name)) != 0; }
|
||||
void claim(const std::string& name) { keys_.insert(entryNameKey(name)); }
|
||||
|
||||
private:
|
||||
std::unordered_set<std::string> keys_;
|
||||
};
|
||||
|
||||
// The name this entry lands under. Terminates: each attempt carries a distinct
|
||||
// counter, and the taken set is finite.
|
||||
std::string mintFileName(const std::string& projectDir, const std::string& packageName,
|
||||
const std::string& uniqueTag, const NameSet& taken) {
|
||||
if (spelledLikeABankFile(packageName) && !taken.taken(packageName)) return packageName;
|
||||
|
||||
const std::string stem = stemOf(packageName);
|
||||
std::string tag = uniqueTag;
|
||||
for (int n = 2;; ++n) {
|
||||
const std::string candidate = deriveBankPaths(projectDir, stem, tag).fileName;
|
||||
if (!taken.taken(candidate)) return candidate;
|
||||
tag = uniqueTag + "-" + std::to_string(n);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
LedgerRefusal importLedgerRefusal(tracking::LedgerStatus status) {
|
||||
switch (status) {
|
||||
case tracking::LedgerStatus::Unreadable: return LedgerRefusal::Malformed;
|
||||
case tracking::LedgerStatus::FutureVersion: return LedgerRefusal::FutureVersion;
|
||||
case tracking::LedgerStatus::Fresh:
|
||||
case tracking::LedgerStatus::Loaded: break;
|
||||
}
|
||||
return LedgerRefusal::None;
|
||||
}
|
||||
|
||||
std::string bankFolderDir(const std::string& projectDir) {
|
||||
// Only the directory half of the result is wanted; the stem is a placeholder.
|
||||
return deriveBankPaths(projectDir, "bank", std::string{}).absoluteDir;
|
||||
}
|
||||
|
||||
ImportPlan planImport(const PackageManifest& manifest,
|
||||
const BankBook& destination,
|
||||
const std::string& projectDir,
|
||||
const std::vector<std::string>& bankFolderFileNames,
|
||||
const std::string& uniqueTag) {
|
||||
ImportPlan plan;
|
||||
|
||||
plan.seedBankName = blankName(manifest.bankDisplayName)
|
||||
? std::string(kDefaultImportBankName)
|
||||
: manifest.bankDisplayName;
|
||||
plan.bankDisplayName = destination.uniqueDisplayName(plan.seedBankName);
|
||||
plan.bankNameAdjusted = plan.bankDisplayName != plan.seedBankName;
|
||||
|
||||
NameSet taken(bankFolderFileNames);
|
||||
// The destination bank is created empty by this same import, so "already in the
|
||||
// destination bank by content" is exactly "already landed by this plan" — the
|
||||
// hash set below IS that bank's findByHash. Cross-bank dedup is deliberately not
|
||||
// enforced (core/model/CLAUDE.md), so other banks' hashes are not consulted.
|
||||
std::unordered_map<std::string, std::string> landedIdForHash;
|
||||
// Every package id, including a collapsed one's, so a parent link that pointed at
|
||||
// a duplicate still resolves to the entry that survived it.
|
||||
std::unordered_map<std::string, std::string> idRemap;
|
||||
|
||||
plan.entries.reserve(manifest.entries.size());
|
||||
for (std::size_t i = 0; i < manifest.entries.size(); ++i) {
|
||||
const PackageEntry& src = manifest.entries[i];
|
||||
|
||||
PlannedEntry e;
|
||||
e.manifestIndex = i;
|
||||
|
||||
const std::string& hash = src.sample.contentHash;
|
||||
if (!hash.empty()) {
|
||||
const auto hit = landedIdForHash.find(hash);
|
||||
if (hit != landedIdForHash.end()) {
|
||||
e.action = EntryAction::Collapse;
|
||||
idRemap[src.sample.id] = hit->second;
|
||||
++plan.collapseCount;
|
||||
plan.entries.push_back(std::move(e));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
e.destFileName = mintFileName(projectDir, src.fileName, uniqueTag, taken);
|
||||
e.renamed = e.destFileName != src.fileName;
|
||||
taken.claim(e.destFileName);
|
||||
|
||||
e.sample = src.sample;
|
||||
e.sample.id = std::string(kImportIdPrefix) + uniqueTag + "-" + e.destFileName;
|
||||
e.sample.relativePath = bankRelativeForName(e.destFileName);
|
||||
|
||||
if (!hash.empty()) landedIdForHash.emplace(hash, e.sample.id);
|
||||
idRemap[src.sample.id] = e.sample.id;
|
||||
|
||||
++plan.landCount;
|
||||
if (e.renamed) ++plan.renameCount;
|
||||
plan.entries.push_back(std::move(e));
|
||||
}
|
||||
|
||||
// Second pass: the remap must be complete before a parent is resolved, since a
|
||||
// sample may precede its own parent in manifest order.
|
||||
for (PlannedEntry& e : plan.entries) {
|
||||
if (e.action != EntryAction::Land || !e.sample.provenance) continue;
|
||||
const auto hit = idRemap.find(e.sample.provenance->parentSampleId);
|
||||
e.sample.provenance->parentSampleId =
|
||||
hit == idRemap.end() ? std::string{} : hit->second;
|
||||
}
|
||||
|
||||
// The package's display order, over the ids that actually landed. SlotMap's own
|
||||
// repair rules settle the rest: two package ids collapsed onto one landed id give
|
||||
// one slot (first wins), and a landed sample the package never positioned is
|
||||
// appended by BankBook::reconcileSlots afterwards.
|
||||
std::vector<std::pair<std::string, int>> slotPairs;
|
||||
for (const std::string& oldId : manifest.slots.orderedIds()) {
|
||||
const auto hit = idRemap.find(oldId);
|
||||
if (hit == idRemap.end()) continue;
|
||||
slotPairs.emplace_back(hit->second, manifest.slots.slotOf(oldId));
|
||||
}
|
||||
plan.slots = model::SlotMap::fromEntries(slotPairs);
|
||||
|
||||
return plan;
|
||||
}
|
||||
|
||||
} // namespace reasampler::package
|
||||
@@ -0,0 +1,76 @@
|
||||
#pragma once
|
||||
// import_plan — the pure import decision: the destination bank's display name after
|
||||
// the book's own uniqueness fold, the reminted sample ids and remapped parents, and
|
||||
// the per-entry land / collapse / rename disposition. Value inputs only; no
|
||||
// filesystem, no session handle, no host types.
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "core/model/bank_book.h"
|
||||
#include "core/model/slot_map.h"
|
||||
#include "core/package/package_manifest.h"
|
||||
#include "core/tracking/origin_ledger.h"
|
||||
|
||||
namespace reasampler::package {
|
||||
|
||||
// The bank name a package that recorded none (or a blank one) imports under.
|
||||
inline constexpr const char* kDefaultImportBankName = "Imported bank";
|
||||
|
||||
// The prefix every imported sample id is reminted under, so a package's own ids —
|
||||
// unique only within the project that made them — never enter this index.
|
||||
inline constexpr const char* kImportIdPrefix = "pkg-";
|
||||
|
||||
// Which of the two refusal messages the import owes the user, if either.
|
||||
//
|
||||
// Keyed on the LEDGER STATUS ALONE, never on prune's composite blockedByTracking:
|
||||
// that flag also fires on undecodable rsusage_* keys, which govern which files a
|
||||
// DELETION may touch. An import deletes nothing and computes no protected set — it
|
||||
// writes birth records — so an unreadable usage key must not refuse one.
|
||||
enum class LedgerRefusal { None, Malformed, FutureVersion };
|
||||
|
||||
LedgerRefusal importLedgerRefusal(tracking::LedgerStatus status);
|
||||
|
||||
// What one manifest entry does when the import runs.
|
||||
// - Land: write the payload under destFileName and add `sample`.
|
||||
// - Collapse: an equal contentHash already lands in this same import, so the payload
|
||||
// is NOT written and no entry is added. Writing it and letting
|
||||
// BankModel::add collapse the entry would leave the file referenced by
|
||||
// nothing — an orphan manufactured by a dedup.
|
||||
enum class EntryAction { Land, Collapse };
|
||||
|
||||
struct PlannedEntry {
|
||||
std::size_t manifestIndex = 0;
|
||||
EntryAction action = EntryAction::Land;
|
||||
std::string destFileName; // Land only — a bare name in the bank folder
|
||||
model::Sample sample; // Land only — id, path and parent already remapped
|
||||
bool renamed = false; // the package's own name was taken, so a fresh one was minted
|
||||
};
|
||||
|
||||
struct ImportPlan {
|
||||
std::string bankDisplayName;
|
||||
bool bankNameAdjusted = false; // the seed was taken, so the name carries a suffix
|
||||
std::string seedBankName; // the seed the probe started from
|
||||
std::vector<PlannedEntry> entries; // one per manifest entry, in manifest order
|
||||
model::SlotMap slots; // the package's slots over the reminted ids
|
||||
int landCount = 0;
|
||||
int collapseCount = 0;
|
||||
int renameCount = 0;
|
||||
};
|
||||
|
||||
// The bank folder an import lands into — the same expression capture uses, so an
|
||||
// imported file is spelled exactly like a captured one.
|
||||
std::string bankFolderDir(const std::string& projectDir);
|
||||
|
||||
// Decides everything about an import except the bytes. `bankFolderFileNames` are the
|
||||
// bare names already present in that folder (never overwritten); `uniqueTag` is the
|
||||
// shell's per-import disambiguator, extended with an ascending counter where one tag
|
||||
// is not enough. Total: every manifest entry yields exactly one PlannedEntry.
|
||||
ImportPlan planImport(const PackageManifest& manifest,
|
||||
const BankBook& destination,
|
||||
const std::string& projectDir,
|
||||
const std::vector<std::string>& bankFolderFileNames,
|
||||
const std::string& uniqueTag);
|
||||
|
||||
} // namespace reasampler::package
|
||||
@@ -103,6 +103,13 @@ bool sameEntryName(const std::string& a, const std::string& b) {
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string entryNameKey(const std::string& name) {
|
||||
std::string key;
|
||||
key.reserve(name.size());
|
||||
for (unsigned char c : name) key += lowerAscii(c);
|
||||
return key;
|
||||
}
|
||||
|
||||
bool isValidNestedSamplePath(const std::string& path) {
|
||||
if (util::isAbsolutePath(path)) return false;
|
||||
// Component-wise, not a substring scan: "take..final/a.wav" is a legal
|
||||
|
||||
@@ -76,6 +76,12 @@ bool isValidEntryName(const std::string& name);
|
||||
// bytes compare exactly (see this directory's CLAUDE.md on NFC/NFD).
|
||||
bool sameEntryName(const std::string& a, const std::string& b);
|
||||
|
||||
// sameEntryName's fold made explicit: the ASCII-lower-cased bytes, so
|
||||
// entryNameKey(a) == entryNameKey(b) exactly when sameEntryName(a, b). For a caller
|
||||
// holding many names at once — folding them into a set is what turns an O(n^2)
|
||||
// pairwise scan into a linear one.
|
||||
std::string entryNameKey(const std::string& name);
|
||||
|
||||
// The one field in the format that CAN express a path: a nested Sample's
|
||||
// relativePath, which is bank-relative by design. Rejects every absolute form
|
||||
// (the shared util::isAbsolutePath test) and any ".." component — BankModel::add
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#include "core/package/package_manifest.h"
|
||||
|
||||
#include <unordered_set>
|
||||
#include <utility>
|
||||
|
||||
#include "core/json/json.h"
|
||||
@@ -14,11 +15,16 @@ using ObjWriter = json::Writer;
|
||||
|
||||
// Shared by serializeManifest and deserializeManifest — see this directory's
|
||||
// CLAUDE.md for why duplicate names are rejected both ways. Equivalence is the
|
||||
// format's, not std::string's: sameEntryName folds ASCII case.
|
||||
// format's, not std::string's: entryNameKey is sameEntryName's ASCII-case fold.
|
||||
//
|
||||
// A set, not the pairwise scan this replaced: under kMaxManifestBytes a hostile
|
||||
// package can declare hundreds of thousands of minimal entries, and O(n^2) over that
|
||||
// is a multi-minute hang on the decode path an import drives.
|
||||
bool duplicateName(const std::vector<PackageEntry>& entries) {
|
||||
for (std::size_t i = 0; i < entries.size(); ++i)
|
||||
for (std::size_t j = i + 1; j < entries.size(); ++j)
|
||||
if (sameEntryName(entries[i].fileName, entries[j].fileName)) return true;
|
||||
std::unordered_set<std::string> seen;
|
||||
seen.reserve(entries.size());
|
||||
for (const auto& e : entries)
|
||||
if (!seen.insert(entryNameKey(e.fileName)).second) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user