diff --git a/docs/product/midi-playback.md b/docs/product/midi-playback.md new file mode 100644 index 0000000..ed47925 --- /dev/null +++ b/docs/product/midi-playback.md @@ -0,0 +1,312 @@ +# MIDI playback — opportunity & design-space framing + +Framing for a **MIDI-triggered audio sampler** that plays back ReaSampler's captured +banks. This is a **discussion-shaping doc, not a build plan** — no phase, no PLAN.md +points, no settled forks yet. It exists so Daniel and the-boss can react to an honest +map of the option space before anything is scoped. + +Status: framed by product-designer (2026-07-26). Grounded in the vendored REAPER SDK +headers (`vendor/reaper-sdk/sdk/reaper_plugin.h`, `reaper_plugin_functions.h`) — the +plugin-format claims below are checked against those headers, not asserted. The +genuine forks are flagged as **Daniel's to decide**; nothing here pre-decides them. + +--- + +## The need, stated plainly + +ReaSampler today **captures and organizes** audio into per-project banks (on-disk +32-bit float WAVs + a JSON index, multi-bank via `bank_book`). It has **no way to +play those samples back from a MIDI keyboard.** The panel can *audition* a sample on +click — but a one-shot, one-voice audition is not an instrument. There is no +key→sample mapping, no velocity response, no polyphony, no envelope, no round-robin. + +The ask: *what would let me play my captured banks MIDI-triggered, and what does the +REAPER surface actually allow toward that?* + +The rest of this doc is organized as the task framed it: (1) the plugin-format +reality, honestly; (2) the integration seam between capture and playback; (3) product +shape and scope tiers; (4) the forks that are Daniel's to call. + +--- + +## 1. The plugin-format reality (the honest part) + +**The load-bearing fact: a REAPER extension cannot be a MIDI-triggered instrument.** +This is not a limitation we can engineer around by trying harder — it is what the +plugin *format* is. ReaSampler today is a `reaper_*.dll` loaded at startup via the +extension API (`ReaperPluginEntry`). That format gives an extension enormous reach +over the *project* — tracks, items, actions, rendering, project state — but it is +**not a node in any track's audio signal chain.** An instrument track's "read live +MIDI, emit audio per-voice" contract belongs to VST/VST3/CLAP/JSFX plugins, which +REAPER hosts through an entirely different mechanism. + +What the extension SDK *does* expose near audio/MIDI, verified in the headers, and why +none of it is "be an instrument": + +- **`Audio_RegHardwareHook` + `audio_hook_register_t::OnAudioBuffer`** (`reaper_plugin.h` + ~1228). This is a real-time callback into the audio thread, called pre- and + post-REAPER-processing, handing you `GetBuffer(isOutput, idx)` for the **hardware + I/O** channels. It is a *hardware tap*, not a track insert. You could technically + mix sample playback into the hardware output buffer here — but you'd be bypassing + REAPER's entire routing, metering, recording, and rendering path. Output from this + hook **cannot be recorded, rendered, routed, or FX-processed** as track audio. It is + the wrong tool for an instrument by construction, and it runs in the audio thread + with all the real-time-safety hazards that implies. Rejected on sight for this + purpose — noting it only to be complete. + +- **`kbd_OnMidiEvent` / `kbd_OnMidiList`** (`reaper_plugin_functions.h` ~4138). REAPER + *informs* the extension of MIDI events (for action-triggering — this is how MIDI can + fire the capture/insert actions we already plan to bind). It is **notification, not a + routable MIDI input** feeding a synth voice allocator. Great for "MIDI note fires the + capture-to-slot action"; useless as an instrument's note input. + +- **`PlayPreview` / `PlayTrackPreview` + `preview_register_t`** (`reaper_plugin.h` + ~1302). This is REAPER's **preview-playback** mechanism, and **ReaSampler already + uses it** — `bank_panel.cpp` streams a bank WAV through a caller-owned + `preview_register_t` for click-audition (`PCM_Source_CreateFromFile` + + `PlayPreview`). This is the one audio-emitting surface the extension already drives, + and it is load-bearing for the hybrid option below. But note its shape: it plays *a* + source at *a* position with *a* volume — it is a fire-and-forget preview, **not a + polyphonic, per-note, velocity-scaled, envelope-shaped voice engine.** You do not get + MIDI→voice allocation for free; you'd be building a sampler engine on top of a + preview primitive. + +- **`pcmsrc` registration + `PCM_source` subclassing** (`reaper_plugin.h` ~804, the + `PCM_SOURCE_EXT_*` surface). An extension *can* register a custom `PCM_source` type. + This is the surface REAPER itself and extensions like SWS use for custom media. It is + a real seam — but it produces a *media source* (something an item plays), not a + *live-MIDI instrument*. Interesting for "a bank-backed media source" but not the + keyboard-instrument ask. + +**Conclusion, stated for the record:** to get a genuine MIDI-triggered instrument — +play a note, hear the mapped sample, velocity-scaled, polyphonic, with an envelope — +the playback engine must be a **standard instrument plugin** (VST3 / CLAP / JSFX), +*not* the extension. The extension SDK is the wrong format for that job, and this is +the single most important thing for Daniel to internalize before scoping anything. + +### The three honest options + +**Option A — a real VSTi/instrument plugin (JUCE or bare VST3/CLAP SDK) that reads +ReaSampler's banks.** A separate build artifact: a VST3 (and/or CLAP) sampler plugin +that the user instantiates on an instrument track. It reads the bank JSON + WAV layout +ReaSampler writes, maps samples across the keyboard, and plays them MIDI-triggered with +a real voice engine. This is the "sophisticated sampler" answer. +- *Gives you:* everything an instrument is — polyphony, velocity, envelopes, the works, + fully integrated into REAPER's routing/render/record path like any VSTi. +- *Costs:* a **second codebase in a second plugin format**, almost certainly a new + dependency (JUCE is the pragmatic choice; bare VST3 SDK is more code, CLAP is leaner + but younger). It is a real DSP/voice-engine build, not a weekend. Cross-platform DSP, + its own build/release/signing story, its own UI toolkit. This is a **product-sized + commitment**, not a feature. + +**Option B — a JSFX sampler.** JSFX is REAPER's built-in scriptable plugin format +(text `.jsfx` files, JIT-compiled by REAPER, hostable as an instrument). A JSFX +instrument *can* receive live MIDI and emit audio on a track. It can load samples +(`Xen`-style file reads / the JSFX file/serialize API) and play them back. +- *Gives you:* a real in-track instrument with **zero new binary, zero new SDK, zero + JUCE** — ships as a text file alongside the extension, cross-platform for free + (REAPER runs the JIT everywhere it runs). +- *Costs:* JSFX is a constrained DSP scripting language, not C++. A polyphonic + multisample engine with round-robin/velocity-layers/streaming is *doable* but you're + writing DSP in JSFX's idiom, and large-sample streaming / disk I/O is more awkward + than in a native plugin. Reading ReaSampler's JSON index from JSFX is friction (JSFX + is not a general-purpose file parser). Best fit for a **minimal-to-mid** sampler, a + real ceiling for a **sophisticated** one. + +**Option C — the hybrid (recommended framing to explore first).** Keep the extension +as the sole owner of **capture + organization** (which is its whole existing identity +and the load-bearing "capture and placement are separate acts" principle). Add a +**separate instrument** (Option A *or* B) that **consumes the banks** as a +shared-artifact contract. The extension never becomes an instrument; the instrument +never captures. Each does what its format is good at. This is the honest shape of the +whole thing — the two options above are really "which instrument technology" *within* +the hybrid, because the extension is staying regardless. + +The real fork, then, is **not** "extension vs. plugin" (the extension stays either +way) — it is **"which instrument format consumes the banks: JSFX or native VSTi/CLAP,"** +and **"how much sampler do we actually want."** Those are §3 and §4. + +--- + +## 2. The integration seam + +The obvious shared artifact is what ReaSampler already produces: **the bank folder +(project-relative WAVs) + the bank/index JSON.** The question is whether that JSON is +*sufficient* for a playback engine, or whether playback needs mapping data the index +doesn't carry today. + +**What the current index carries** (from `bank_model`'s `Sample`, per CONTEXT.md §Data +model): id, display name, relative path, source range, channel count, sample rate, +length, capture tempo, an **optional key**, peak/RMS/LUFS, content hash, tier, +provenance, timestamp. Notably it *already* has an optional key field and capture +tempo — the seeds of pitch-mapping are there. + +**What a playback engine additionally needs** — none of which the index carries today: + +- **Root note** (the pitch the sample was recorded at, so it can be repitched across + the keyboard). The optional `key` is close but is "musical key," not "root MIDI + note" — different thing. +- **Key range / zone** (low note, high note) — which keys trigger this sample. +- **Velocity layers** (which sample plays at which velocity band; a mapping of samples + to velocity zones). +- **Round-robin groups** (cycle through N samples on repeated same-note hits). +- **Loop points** (sustain loop start/end for held notes; sample-accurate, + zero-crossing-aware — CONTEXT already flags loop/zero-crossing handling as + day-one-relevant for wavetable material). +- **Amplitude envelope** (ADSR) and optionally filter/pitch envelopes. +- **Tuning/gain trim** per sample. + +**The design decision this forces:** does the *mapping* (key ranges, velocity layers, +round-robin, envelopes) live — + +- **(i) in the bank index** (ReaSampler owns the instrument definition; the plugin is a + dumb player of a ReaSampler-authored map), or +- **(ii) in the instrument plugin's own state** (the plugin owns the map; the bank is + just a WAV+metadata pool it maps *over*), or +- **(iii) split** — the bank carries per-sample *intrinsics* (root note, loop points — + facts about the file), and the instrument carries the *arrangement* (zones, layers, + envelopes — a performance choice)? + +**(iii) is the principled answer and worth leading with.** It mirrors ReaSampler's own +deepest instinct — the *capture/placement separation*. Root note and loop points are +**facts about the captured file** (analogous to sample rate, length, peaks) and belong +in the bank, computed or set at/after capture. Key zones, velocity layers, envelopes, +round-robin are a **performance mapping** — a creative arrangement of those facts — and +belong in the instrument. This keeps the bank a clean, portable, tool-agnostic library +(a bank is still just WAVs + facts, readable by anything) while the instrument owns the +opinionated part. It also means **the bank index grows only by file-intrinsic fields** +(root note, loop points), which is a small, safe, additive change — exactly the kind of +seam ReaSampler already knows how to add (mirror of how `provenance` was added as a +field in M1 and populated later). + +**A note on format:** if the instrument is native (Option A), it reads the bank JSON +directly — trivial. If it's JSFX (Option B), reading arbitrary JSON is friction; the +seam might need a **simpler sidecar** (a flat `.txt`/key-value map ReaSampler writes +next to the bank, JSFX-parseable) rather than making JSFX parse the index JSON. That's +a concrete cost of the JSFX path and a reason the seam design and the format choice are +coupled. + +**Portability caveat:** the bank is project-relative and travels with the `.rpp` +(M4 machinery). Any instrument consuming it must resolve paths the same way — so the +instrument needs to know the *current* project bank folder. A native plugin instance +on a track can be told its folder (saved in plugin state); a JSFX likewise. This is a +solvable wiring detail but a real one — flag it, don't hand-wave it. + +--- + +## 3. Product shape & scope tiers + +Sketched as tiers, minimal → sophisticated. The point of tiering is that **Tier 0 +delivers the core promise** ("play my captured banks from a MIDI keyboard") and each +tier above is optional depth, not a prerequisite for the one below. + +**Tier 0 — "the bank plays."** One sample mapped chromatically across the keyboard from +a root note; monophonic-or-basic-polyphony; a simple amp envelope; velocity → volume. +Point one bank sample at a MIDI track and play it repitched. *This is the smallest +thing that delivers the promise* and is the honest MVP. On the JSFX path this is a +genuinely small build; on the native path it's the skeleton of the plugin. + +**Tier 1 — "a keymap."** Multiple samples zoned across the keyboard (key ranges), each +with its own root note. Now a captured *kit* (several one-shots) or a *multisampled +instrument* (same instrument sampled at several pitches) plays correctly. This is where +the **root-note + key-range** seam fields (§2) earn their place. Still one sample per +key-region. + +**Tier 2 — "expressive."** Velocity layers (soft vs. hard hits pick different samples), +round-robin (repeated hits cycle samples — the anti-machine-gun feature), full ADSR, +per-sample tuning/gain trim, sustain loops for held notes. This is a *real* sampler and +where "sophisticated" lives. It's also where the mapping data (§2) gets rich enough that +the "who owns the map" fork (i/ii/iii) really bites. + +**Tier 3 — "instrument polish."** Filter + filter envelope, LFOs, per-voice pan, choke +groups (hi-hat open/closed), maybe a modest built-in FX slot. Standard sampler feature +creep. Explicitly *not* needed to deliver value; a direction to leave room for, not +commit to. + +**The recommended reading:** Tier 0–1 is the "does this even belong in ReaSampler's +world" proof. If the answer is yes, Tier 2 is where it becomes a tool people reach for. +Tier 3 is optional forever. **Scope the decision at Tier 0–1 first** and treat 2/3 as +held futures — don't let a Tier-3 feature list drive the format choice. + +### The one-source-multiple-views angle + +Worth flagging because it recurs in Daniel's work: the bank is **one source** (WAVs + +facts). The panel is one view (organize/audition). A MIDI instrument is **another view +of the same source** (play). That's a clean framing — the instrument doesn't fork the +bank, it consumes it — and it argues for keeping the bank as the single authoritative +artifact and the instrument as a pure consumer (seam option ii/iii, never "the +instrument keeps its own copy of the samples"). + +--- + +## 4. Risks & open decisions — Daniel's to call + +None of these are pre-decided here. Each is a genuine fork. + +**D1 — Instrument format: JSFX vs. native VSTi (VST3/CLAP, likely via JUCE).** The +central fork. JSFX = no new binary, no JUCE, cross-platform free, real ceiling on +sophistication and awkward bank-JSON reading. Native = full sophistication and clean +JSON integration, at the cost of a whole second codebase in a second format with its +own build/release/dependency/signing story. *My lean, for discussion only:* if the goal +is Tier 0–1, **prototype in JSFX first** — it proves the seam and the value with near-zero +format commitment, and if it hits a ceiling the seam you designed still serves a later +native build. Reach for native when Tier 2+ is a firm goal, not before. But this is +squarely Daniel's call and depends on how sophisticated he actually wants this. + +**D2 — Whether to take a JUCE (or any external plugin-SDK) dependency at all.** The +project today is a clean C++ extension with two vendored submodules and a proud +pure-core/shell discipline. A native instrument means a *third* major dependency and a +second build target of a fundamentally different kind. That's a real architectural +weight. JSFX sidesteps it entirely. Flagging it as its own decision because "should we +depend on JUCE" is a bigger standing commitment than "should we build a sampler." + +**D3 — Does the pure-core discipline survive the format boundary?** ReaSampler's +identity is *pure REAPER-free testable core + thin shells*. A sampler's **voice engine, +envelope math, key/velocity mapping, and repitch logic are exactly the kind of pure, +testable core** this project excels at — they could live in a REAPER-free, DAW-free, +unit-tested module (mirror of `bank_model`/`peaks`/`view_mode_model`) with the plugin +format (JSFX or VST3 wrapper) as the thin shell around it. **This is the most +ReaSampler-native way to build it** and I'd argue strongly for it regardless of D1: the +sampler DSP core is pure and tested; the format is a shell. The open question is whether +that discipline can hold across a *different plugin format* — with native it's clean C++ +so yes; with JSFX the "pure core" would be JSFX code, harder to unit-test in the CTest +harness. That tension is real and feeds back into D1. + +**D4 — Where the mapping lives (seam fork i/ii/iii from §2).** Bank-owned map, +instrument-owned map, or split (file-intrinsics in the bank, performance-map in the +instrument). *My lean:* **(iii) split** — it's the one that honors ReaSampler's +capture/placement instinct and keeps the bank tool-agnostic. But it's Daniel's call +whether ReaSampler should author instrument definitions at all, or stay purely a +sample library that a *separate* mapping tool arranges. + +**D5 — Cross-platform.** The extension is already cross-platform (SWELL). JSFX inherits +that for free. A native plugin re-opens the full cross-platform DSP + UI + build matrix +(Win/mac/Linux, code-signing on mac, etc.) as a *separate* artifact. A cost that lands +entirely on the native path. + +**D6 — Is this even one product?** The reframe worth surfacing: ReaSampler's thesis is +"a precision *capture/organize* tool; capture and placement are separate acts." A +MIDI-playback instrument is a *different act* — playback. There's a legitimate reading +where the instrument is a **companion product** that shares the bank format, not a +feature *of* ReaSampler — the way a sample library and a sampler that reads it are +related-but-distinct products. That framing might keep ReaSampler sharp (it stays the +capture tool it is) while letting the instrument evolve on its own clock and format. +The alternative reading — it's all one integrated sampler-workstation — is also +coherent. **Which of those two ReaSampler *is* is the highest-order question here, and +it's Daniel's to answer before format/tier decisions mean much.** + +--- + +## What this doc is asking for + +A direction on the two highest-order forks, in order: + +1. **D6 — one product or two?** Is the instrument a feature of ReaSampler, or a + companion product sharing the bank format? Everything else sits under this. +2. **D1 — JSFX-first prototype, or straight to native?** Given a target of Tier 0–1 to + start, and the pure-core-as-shell discipline (D3) held either way. + +Once those two are called, the seam fields (§2, D4) and the tier scope (§3) become +concrete enough to write an actual phase spec. Until then this stays a framing doc with +no PLAN.md footprint — deliberately, so we don't scope an instrument before deciding +whether we're building one.