Close the RSBK name-collision class: ASCII case folding, UTF-8 well-formedness, nested-path traversal
All three are format-locked and validated on encode and decode. Repeated known keys now reject at the root and inside an entry rather than last-wins.
This commit is contained in:
@@ -25,13 +25,22 @@ landing after the format.
|
||||
header (so the refusal can name the writer's semver and version) and nothing
|
||||
else — no manifest, no layout, no half-success. The fields through the writer
|
||||
semver are FROZEN for all future versions to keep that refusal producible.
|
||||
- **Path expression is structurally impossible.** Entry names are bare file
|
||||
names (`isValidEntryName`: no separators, no `..` component, no
|
||||
drive/UNC/rooted form, no control bytes, no Windows-reserved character, no
|
||||
trailing dot/space, no DOS device name), enforced on encode AND decode
|
||||
because a package can arrive from anywhere. There is no field in the format
|
||||
capable of expressing a path. The rule set is the full authority; see
|
||||
`package_format.h`'s doc comment for the itemized list.
|
||||
- **Every name and path in the format is validated on encode AND decode**,
|
||||
because a package can arrive from anywhere. Three rules, all in
|
||||
`package_format`, whose doc comments are the itemized authority:
|
||||
- `isValidEntryName` — a payload's name is a bare file name (no separators,
|
||||
no `..` component, no drive/UNC/rooted form, no control bytes, no
|
||||
Windows-reserved character, no trailing dot/space, no DOS device name,
|
||||
well-formed UTF-8 only). Path expression is impossible in this field.
|
||||
- `sameEntryName` — two entry names differing only by ASCII case are ONE
|
||||
name. Windows and macOS's default APFS would extract them onto a single
|
||||
file, and a bank authored on a case-sensitive filesystem produces the pair
|
||||
honestly.
|
||||
- `isValidNestedSamplePath` — the nested `Sample::relativePath` IS a path by
|
||||
design, and is the one field here that can express one. It refuses a `..`
|
||||
component and every absolute form; `BankModel::add` checks only the latter,
|
||||
so traversal would otherwise reach a future `import_plan` inside a record
|
||||
the format vouched for.
|
||||
- **Framing only, never a payload.** `bank_package` produces header bytes and
|
||||
an ordered `{name, offset, length}` layout; it never holds, copies, or hashes
|
||||
an entry's audio. `decodePackage` proves prefix + payload lengths equal the
|
||||
@@ -48,8 +57,8 @@ landing after the format.
|
||||
|
||||
- `package_format` — the contract: magic, `kPackageFormatVersion` /
|
||||
`kPackageMinReaderVersion`, the ladder comment, the three-way
|
||||
`classifyPackageVersion` (`Readable` / `TooNew` / `Malformed`), the
|
||||
entry-name rule, and `PackageHeader`.
|
||||
`classifyPackageVersion` (`Readable` / `TooNew` / `Malformed`), the three
|
||||
naming rules above, and `PackageHeader`.
|
||||
- `package_manifest` — the manifest model (`PackageEntry` / `PackageManifest`)
|
||||
and its JSON codec. Per entry: bare name, byte length, and a whole-file
|
||||
`capture::hashBytes` digest (deliberately NOT `hashWavContent`, which skips
|
||||
@@ -87,11 +96,36 @@ landing after the format.
|
||||
region" — both refuse whole and write nothing, so the safety property is
|
||||
unchanged, only the message. `classifyPackageVersion` and the frozen-region
|
||||
`TooNew` path are unaffected; this is the post-manifest-parse branch only.
|
||||
- **The parse branch is the ONLY one that relabels**, deliberately: a newer
|
||||
package that trips the manifest cap, a short manifest read, the layout
|
||||
overflow, or the exact-size proof still reports `Malformed` even with
|
||||
`formatVersion` above ours. The size proof clearly should — "install 1.9.0"
|
||||
does not fix a truncated download — and the other three are indistinguishable
|
||||
from ordinary corruption at the point they fail. Don't "complete" the relabel
|
||||
across them for symmetry; the split is the answer, not an omission.
|
||||
- The format carries no algorithm tag for `byteHash` — it is FNV-1a
|
||||
(`capture::hashBytes`) implicitly. Changing the digest algorithm is a
|
||||
`minReaderVersion` bump, not additive: an old reader would otherwise compare
|
||||
a stored digest against bytes hashed the new way and silently misjudge
|
||||
corruption.
|
||||
- **Obligation on the export track: sanitize, don't relay the refusal.**
|
||||
`serializeManifest` returns one indistinguishable `nullopt` for every rejection
|
||||
— an unrepresentable name, a case-folded collision, a traversing nested path, a
|
||||
zero-length entry, a record `BankModel::add` refuses — and most of the naming
|
||||
rules are Windows'. A bank ingested on macOS/Linux legitimately holds
|
||||
`Hit?.wav`, `snare .wav`, or two names differing only by case, and a nested
|
||||
`relativePath` is only checked for the absolute forms where it is written.
|
||||
Relaying the `nullopt` makes ONE such file an unactionable total failure of the
|
||||
whole export. `export_plan` must map bank entries to package
|
||||
names that satisfy these rules (and disambiguate case-folded collisions) before
|
||||
calling this layer; the codec's refusal is the backstop, not the user-facing
|
||||
behaviour.
|
||||
- **NFC/NFD normalization collisions are accepted, not solved.** macOS compares
|
||||
file names normalization-insensitively, so the NFC and NFD spellings of one
|
||||
accented name are two manifest entries that extract onto one file — the same
|
||||
collision class as the ASCII case fold, which `sameEntryName` does catch. A
|
||||
table-free fix does not exist, and restricting names to ASCII would be
|
||||
genuinely over-strict for non-English users. Left open knowingly.
|
||||
- **Cross-module contract with `src/shell/package`:** a genuinely zero-length
|
||||
entry cannot round-trip through the filesystem seam there (`appendPayload`
|
||||
refuses an empty payload — an empty buffer signals an upstream read failure,
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
#include "core/package/package_format.h"
|
||||
|
||||
#include <cctype>
|
||||
#include <cstddef>
|
||||
|
||||
#include "core/util/relative_path.h"
|
||||
|
||||
namespace reasampler::package {
|
||||
|
||||
@@ -14,21 +16,64 @@ PackageReadability classifyPackageVersion(std::uint32_t formatVersion,
|
||||
|
||||
namespace {
|
||||
|
||||
// Hand-rolled rather than std::tolower: that fold is locale-dependent, so two
|
||||
// machines reading the same package could disagree on which names collide.
|
||||
char lowerAscii(unsigned char c) {
|
||||
return (c >= 'A' && c <= 'Z') ? static_cast<char>(c - 'A' + 'a') : static_cast<char>(c);
|
||||
}
|
||||
|
||||
// Windows device names claim the whole entry regardless of extension
|
||||
// (CON, CON.wav, con.WAV are all the same reserved device) — checked against
|
||||
// the portion before the first dot only.
|
||||
// the portion before the first dot only. The trailing three pairs are the UTF-8
|
||||
// spellings of COM¹/COM²/COM³/LPT¹/LPT²/LPT³: Windows reads those ISO 8859-1
|
||||
// superscripts as digits in a device name. COM0/LPT0 are NOT reserved.
|
||||
bool isDosDeviceName(const std::string& name) {
|
||||
std::string base = name.substr(0, name.find('.'));
|
||||
for (char& c : base) c = static_cast<char>(std::toupper(static_cast<unsigned char>(c)));
|
||||
for (char& c : base) c = lowerAscii(static_cast<unsigned char>(c));
|
||||
static const std::string kReserved[] = {
|
||||
"CON", "PRN", "AUX", "NUL",
|
||||
"COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
|
||||
"LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9",
|
||||
"con", "prn", "aux", "nul",
|
||||
"com1", "com2", "com3", "com4", "com5", "com6", "com7", "com8", "com9",
|
||||
"lpt1", "lpt2", "lpt3", "lpt4", "lpt5", "lpt6", "lpt7", "lpt8", "lpt9",
|
||||
"com\xC2\xB9", "com\xC2\xB2", "com\xC2\xB3",
|
||||
"lpt\xC2\xB9", "lpt\xC2\xB2", "lpt\xC2\xB3",
|
||||
};
|
||||
for (const auto& r : kReserved) if (base == r) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Table-free UTF-8 well-formedness. Overlong encodings, surrogate halves and
|
||||
// code points past U+10FFFF are rejected as hard as a structural length error:
|
||||
// the UTF-8 -> UTF-16 conversion a host must perform maps an ill-formed
|
||||
// sequence to U+FFFD unless it opts into failing, so two names differing only
|
||||
// in invalid bytes would otherwise collapse onto one destination file.
|
||||
bool isWellFormedUtf8(const std::string& s) {
|
||||
const auto* p = reinterpret_cast<const unsigned char*>(s.data());
|
||||
const std::size_t n = s.size();
|
||||
for (std::size_t i = 0; i < n;) {
|
||||
const unsigned char c = p[i];
|
||||
std::size_t extra = 0;
|
||||
std::uint32_t cp = 0;
|
||||
if (c < 0x80) { ++i; continue; }
|
||||
else if ((c & 0xE0) == 0xC0) { extra = 1; cp = c & 0x1Fu; }
|
||||
else if ((c & 0xF0) == 0xE0) { extra = 2; cp = c & 0x0Fu; }
|
||||
else if ((c & 0xF8) == 0xF0) { extra = 3; cp = c & 0x07u; }
|
||||
else return false; // a stray continuation byte, or a 5/6-byte lead
|
||||
if (i + extra >= n) return false;
|
||||
for (std::size_t k = 1; k <= extra; ++k) {
|
||||
const unsigned char cont = p[i + k];
|
||||
if ((cont & 0xC0) != 0x80) return false;
|
||||
cp = (cp << 6) | (cont & 0x3Fu);
|
||||
}
|
||||
if (extra == 1 && cp < 0x80) return false;
|
||||
if (extra == 2 && cp < 0x800) return false;
|
||||
if (extra == 3 && cp < 0x10000) return false;
|
||||
if (cp > 0x10FFFF) return false;
|
||||
if (cp >= 0xD800 && cp <= 0xDFFF) return false;
|
||||
i += extra + 1;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool isValidEntryName(const std::string& name) {
|
||||
@@ -46,7 +91,29 @@ bool isValidEntryName(const std::string& name) {
|
||||
if (c == '*' || c == '?' || c == '|' || c == '<' || c == '>' || c == '"') return false;
|
||||
}
|
||||
if (isDosDeviceName(name)) return false;
|
||||
return isWellFormedUtf8(name);
|
||||
}
|
||||
|
||||
bool sameEntryName(const std::string& a, const std::string& b) {
|
||||
if (a.size() != b.size()) return false;
|
||||
for (std::size_t i = 0; i < a.size(); ++i)
|
||||
if (lowerAscii(static_cast<unsigned char>(a[i])) !=
|
||||
lowerAscii(static_cast<unsigned char>(b[i])))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool isValidNestedSamplePath(const std::string& path) {
|
||||
if (util::isAbsolutePath(path)) return false;
|
||||
// Component-wise, not a substring scan: "take..final/a.wav" is a legal
|
||||
// relative path, "bank/../evil.wav" is not.
|
||||
for (std::size_t start = 0;; ) {
|
||||
const std::size_t sep = path.find_first_of("/\\", start);
|
||||
const std::size_t end = (sep == std::string::npos) ? path.size() : sep;
|
||||
if (path.compare(start, end - start, "..") == 0) return false;
|
||||
if (sep == std::string::npos) return true;
|
||||
start = sep + 1;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace reasampler::package
|
||||
|
||||
@@ -56,25 +56,39 @@ enum class PackageReadability {
|
||||
PackageReadability classifyPackageVersion(std::uint32_t formatVersion,
|
||||
std::uint32_t minReaderVersion);
|
||||
|
||||
// The entry-name rule that makes path expression structurally impossible: a
|
||||
// bare file name only. Rejects empty, ".", the exact ".." component (a name
|
||||
// can only ever be one component, since separators are banned below — a
|
||||
// substring scan would over-reject legal names like "take..final.wav"), any
|
||||
// control byte (NUL included — truncates at the first filesystem call and
|
||||
// collides two distinct manifest entries onto one file) or 0x7F, any '/',
|
||||
// '\\' or ':' (which also bans every absolute form — drive, UNC, rooted), any
|
||||
// Windows-reserved character (`*?|<>"`), a trailing dot or space (silently
|
||||
// stripped at file creation, so "a.wav " and "a.wav" would collide), a DOS
|
||||
// device name (CON/PRN/AUX/NUL/COM1-9/LPT1-9, case-insensitive, with or
|
||||
// without an extension), and names over kMaxEntryNameBytes. Enforced on
|
||||
// encode AND decode by package_manifest.
|
||||
// The entry-name rule: a bare file name only. Rejects empty, ".", the exact
|
||||
// ".." component (a name can only ever be one component, since separators are
|
||||
// banned below — a substring scan would over-reject legal names like
|
||||
// "take..final.wav"), any control byte (NUL included — truncates at the first
|
||||
// filesystem call and collides two distinct manifest entries onto one file) or
|
||||
// 0x7F, any '/', '\\' or ':' (which also bans every absolute form — drive, UNC,
|
||||
// rooted), any Windows-reserved character (`*?|<>"`), a trailing dot or space
|
||||
// (silently stripped at file creation, so "a.wav " and "a.wav" would collide),
|
||||
// a DOS device name (CON/PRN/AUX/NUL/COM1-9/LPT1-9 plus the superscript
|
||||
// COM/LPT 1-3 forms, case-insensitive, with or without an extension), names
|
||||
// over kMaxEntryNameBytes, and any byte sequence that is not well-formed UTF-8.
|
||||
bool isValidEntryName(const std::string& name);
|
||||
|
||||
// The format's name-equivalence rule: two entry names that differ only by ASCII
|
||||
// case are ONE name. Windows and macOS's default APFS are case-insensitive, so
|
||||
// "Kick.wav" and "kick.wav" would extract onto a single file — and a bank
|
||||
// authored on a case-sensitive filesystem produces that pair honestly. Non-ASCII
|
||||
// bytes compare exactly (see this directory's CLAUDE.md on NFC/NFD).
|
||||
bool sameEntryName(const std::string& a, const std::string& b);
|
||||
|
||||
// The one field in the format that CAN express a path: a nested Sample's
|
||||
// relativePath, which is bank-relative by design. Rejects every absolute form
|
||||
// (the shared util::isAbsolutePath test) and any ".." component — BankModel::add
|
||||
// checks only the former, so traversal reaches the format without this.
|
||||
bool isValidNestedSamplePath(const std::string& path);
|
||||
|
||||
// The fixed header, informational semver included. writerVersion is
|
||||
// version::stampVersion() on the write side — it exists so a TooNew refusal can
|
||||
// tell the user which build to install; it never gates. Defaults are 0/0, NOT
|
||||
// the current ladder pair — a caller reading `header` after a Malformed
|
||||
// verdict must see an obviously-unset value, not a plausible-looking 1/1.
|
||||
// the current ladder pair, so a header that never parsed reads as obviously
|
||||
// unset rather than as a plausible 1/1. The fields are meaningful whenever they
|
||||
// are non-zero, not only on success: a decode that got past the header and
|
||||
// failed later (a corrupt manifest) reports the real pair alongside Malformed.
|
||||
struct PackageHeader {
|
||||
std::uint32_t formatVersion = 0;
|
||||
std::uint32_t minReaderVersion = 0;
|
||||
|
||||
@@ -13,11 +13,12 @@ using json::numToStr;
|
||||
using ObjWriter = json::Writer;
|
||||
|
||||
// Shared by serializeManifest and deserializeManifest — see this directory's
|
||||
// CLAUDE.md for why duplicate names are rejected both ways.
|
||||
// CLAUDE.md for why duplicate names are rejected both ways. Equivalence is the
|
||||
// format's, not std::string's: sameEntryName folds ASCII case.
|
||||
bool duplicateName(const std::vector<PackageEntry>& entries) {
|
||||
for (std::size_t i = 0; i < entries.size(); ++i)
|
||||
for (std::size_t j = i + 1; j < entries.size(); ++j)
|
||||
if (entries[i].fileName == entries[j].fileName) return true;
|
||||
if (sameEntryName(entries[i].fileName, entries[j].fileName)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -45,6 +46,7 @@ bool PackageManifest::operator==(const PackageManifest& o) const {
|
||||
std::optional<std::string> serializeManifest(const PackageManifest& m) {
|
||||
for (const auto& e : m.entries) {
|
||||
if (!isValidEntryName(e.fileName)) return std::nullopt;
|
||||
if (!isValidNestedSamplePath(e.sample.relativePath)) return std::nullopt;
|
||||
// Cross-module contract with src/shell/package — see this directory's
|
||||
// CLAUDE.md.
|
||||
if (e.byteLength == 0) return std::nullopt;
|
||||
@@ -122,19 +124,22 @@ bool parseEntry(json::Reader& r, PackageEntry& e) {
|
||||
std::string key;
|
||||
if (!r.parseKey(key)) return false;
|
||||
|
||||
// A repeated key is rejected here exactly as at the root — same format
|
||||
// question, one level down.
|
||||
if (key == "name") {
|
||||
if (!r.parseString(e.fileName)) return false;
|
||||
if (haveName || !r.parseString(e.fileName)) return false;
|
||||
haveName = true;
|
||||
} else if (key == "length") {
|
||||
std::int64_t v = 0;
|
||||
if (!r.parseInt64(v)) return false;
|
||||
if (haveLength || !r.parseInt64(v)) return false;
|
||||
if (v < 0) return false;
|
||||
e.byteLength = static_cast<std::uint64_t>(v);
|
||||
haveLength = true;
|
||||
} else if (key == "hash") {
|
||||
if (!r.parseString(e.byteHash)) return false;
|
||||
if (haveHash || !r.parseString(e.byteHash)) return false;
|
||||
haveHash = true;
|
||||
} else if (key == "index") {
|
||||
if (haveSample) return false;
|
||||
std::string raw;
|
||||
if (!r.captureValue(raw)) return false;
|
||||
auto idx = model::BankModel::deserialize(raw);
|
||||
@@ -150,25 +155,31 @@ bool parseEntry(json::Reader& r, PackageEntry& e) {
|
||||
|
||||
if (!r.consume('}')) return false;
|
||||
if (!haveName || !haveLength || !haveHash || !haveSample) return false;
|
||||
return isValidEntryName(e.fileName);
|
||||
return isValidEntryName(e.fileName) && isValidNestedSamplePath(e.sample.relativePath);
|
||||
}
|
||||
|
||||
bool parseManifest(json::Reader& r, PackageManifest& m) {
|
||||
if (!r.consume('{')) return false;
|
||||
r.skipWs();
|
||||
bool haveEntries = false;
|
||||
// "Which duplicate keys are legal" is a format contract, so it is answered
|
||||
// for every root key rather than only for the one that would accumulate:
|
||||
// a repeated key is rejected, never last-wins. Unknown keys may repeat —
|
||||
// they are skipped, and a future format must stay free to add them.
|
||||
bool haveBankName = false, haveExported = false, haveEntries = false, haveSlots = false;
|
||||
const auto firstTime = [](bool& seen) { const bool ok = !seen; seen = true; return ok; };
|
||||
if (!r.consume('}')) { // not the empty-object shortcut: parse the members
|
||||
do {
|
||||
std::string key;
|
||||
if (!r.parseKey(key)) return false;
|
||||
|
||||
if (key == "bankName") {
|
||||
if (!firstTime(haveBankName)) return false;
|
||||
if (!r.parseString(m.bankDisplayName)) return false;
|
||||
} else if (key == "exported") {
|
||||
if (!firstTime(haveExported)) return false;
|
||||
if (!r.parseInt64(m.exportTimestamp)) return false;
|
||||
} else if (key == "entries") {
|
||||
if (haveEntries) return false; // a repeated key must not accumulate
|
||||
haveEntries = true;
|
||||
if (!firstTime(haveEntries)) return false;
|
||||
if (!r.consume('[')) return false;
|
||||
r.skipWs();
|
||||
if (!r.consume(']')) {
|
||||
@@ -180,6 +191,7 @@ bool parseManifest(json::Reader& r, PackageManifest& m) {
|
||||
if (!r.consume(']')) return false;
|
||||
}
|
||||
} else if (key == "slots") {
|
||||
if (!firstTime(haveSlots)) return false;
|
||||
if (!parseSlots(r, m.slots)) return false;
|
||||
} else {
|
||||
if (!r.skipValue()) return false; // forward-compat unknown keys
|
||||
|
||||
@@ -40,18 +40,21 @@ struct PackageManifest {
|
||||
bool operator==(const PackageManifest& o) const;
|
||||
};
|
||||
|
||||
// Emits the manifest JSON. nullopt when the manifest cannot be represented:
|
||||
// an invalid or duplicate entry name, a zero-length entry (see this
|
||||
// directory's CLAUDE.md — the shell's payload-append seam cannot round-trip
|
||||
// one), or a sample record BankModel itself would reject (empty id, absolute
|
||||
// path) — refusing on encode so an undecodable package is never written.
|
||||
// Emits the manifest JSON. nullopt when the manifest cannot be represented: an
|
||||
// invalid or duplicate entry name (duplicate by sameEntryName, not string
|
||||
// equality), a nested relativePath isValidNestedSamplePath refuses, a
|
||||
// zero-length entry (see this directory's CLAUDE.md — the shell's payload-append
|
||||
// seam cannot round-trip one), or a sample record BankModel itself would reject
|
||||
// (empty id, absolute path) — refusing on encode so an undecodable package is
|
||||
// never written.
|
||||
std::optional<std::string> serializeManifest(const PackageManifest& m);
|
||||
|
||||
// Parses manifest JSON (nullopt on malformed input, never UB). Unknown keys are
|
||||
// skipped at every level, so an additive newer manifest still parses. Rejects
|
||||
// what encode rejects — entry names are validated on BOTH directions because a
|
||||
// package can arrive from anywhere — plus a missing per-entry field or a
|
||||
// negative length.
|
||||
// skipped at every level, so an additive newer manifest still parses; a repeated
|
||||
// KNOWN root key is rejected rather than last-wins. Rejects what encode rejects
|
||||
// except the zero-length entry — names and nested paths are validated on BOTH
|
||||
// directions because a package can arrive from anywhere — plus a missing
|
||||
// per-entry field or a negative length.
|
||||
std::optional<PackageManifest> deserializeManifest(const std::string& json);
|
||||
|
||||
} // namespace reasampler::package
|
||||
|
||||
Reference in New Issue
Block a user