reasampler/docs/product/midi-playback.md

MIDI playback — opportunity & design-space framing

Framing for a MIDI-triggered audio sampler that plays back ReaSampler's captured banks. This is a discussion-shaping doc, not a build plan — no phase, no PLAN.md points, no settled forks yet. It exists so Daniel and the-boss can react to an honest map of the option space before anything is scoped.

Status: framed by product-designer (2026-07-26). Grounded in the vendored REAPER SDK headers (vendor/reaper-sdk/sdk/reaper_plugin.h, reaper_plugin_functions.h) — the plugin-format claims below are checked against those headers, not asserted. The genuine forks are flagged as Daniel's to decide; nothing here pre-decides them.

---

The need, stated plainly

ReaSampler today captures and organizes audio into per-project banks (on-disk 32-bit float WAVs + a JSON index, multi-bank via bank_book). It has no way to play those samples back from a MIDI keyboard. The panel can audition a sample on click — but a one-shot, one-voice audition is not an instrument. There is no key→sample mapping, no velocity response, no polyphony, no envelope, no round-robin.

The ask: what would let me play my captured banks MIDI-triggered, and what does the REAPER surface actually allow toward that?

The rest of this doc is organized as the task framed it: (1) the plugin-format reality, honestly; (2) the integration seam between capture and playback; (3) product shape and scope tiers; (4) the forks that are Daniel's to call.

---

1. The plugin-format reality (the honest part)

The load-bearing fact: a REAPER extension cannot be a MIDI-triggered instrument. This is not a limitation we can engineer around by trying harder — it is what the plugin format is. ReaSampler today is a reaper_*.dll loaded at startup via the extension API (ReaperPluginEntry). That format gives an extension enormous reach over the project — tracks, items, actions, rendering, project state — but it is not a node in any track's audio signal chain. An instrument track's "read live MIDI, emit audio per-voice" contract belongs to VST/VST3/CLAP/JSFX plugins, which REAPER hosts through an entirely different mechanism.

What the extension SDK does expose near audio/MIDI, verified in the headers, and why none of it is "be an instrument":

• Audio_RegHardwareHook + audio_hook_register_t::OnAudioBuffer (reaper_plugin.h ~1228). This is a real-time callback into the audio thread, called pre- and post-REAPER-processing, handing you GetBuffer(isOutput, idx) for the hardware I/O channels. It is a hardware tap, not a track insert. You could technically mix sample playback into the hardware output buffer here — but you'd be bypassing REAPER's entire routing, metering, recording, and rendering path. Output from this hook cannot be recorded, rendered, routed, or FX-processed as track audio. It is the wrong tool for an instrument by construction, and it runs in the audio thread with all the real-time-safety hazards that implies. Rejected on sight for this purpose — noting it only to be complete.

• kbd_OnMidiEvent / kbd_OnMidiList (reaper_plugin_functions.h ~4138). REAPER informs the extension of MIDI events (for action-triggering — this is how MIDI can fire the capture/insert actions we already plan to bind). It is notification, not a routable MIDI input feeding a synth voice allocator. Great for "MIDI note fires the capture-to-slot action"; useless as an instrument's note input.

• PlayPreview / PlayTrackPreview + preview_register_t (reaper_plugin.h ~1302). This is REAPER's preview-playback mechanism, and ReaSampler already uses it — bank_panel.cpp streams a bank WAV through a caller-owned preview_register_t for click-audition (PCM_Source_CreateFromFile + PlayPreview). This is the one audio-emitting surface the extension already drives, and it is load-bearing for the hybrid option below. But note its shape: it plays a source at a position with a volume — it is a fire-and-forget preview, not a polyphonic, per-note, velocity-scaled, envelope-shaped voice engine. You do not get MIDI→voice allocation for free; you'd be building a sampler engine on top of a preview primitive.

• pcmsrc registration + PCM_source subclassing (reaper_plugin.h ~804, the PCM_SOURCE_EXT_* surface). An extension can register a custom PCM_source type. This is the surface REAPER itself and extensions like SWS use for custom media. It is a real seam — but it produces a media source (something an item plays), not a live-MIDI instrument. Interesting for "a bank-backed media source" but not the keyboard-instrument ask.

Conclusion, stated for the record: to get a genuine MIDI-triggered instrument — play a note, hear the mapped sample, velocity-scaled, polyphonic, with an envelope — the playback engine must be a standard instrument plugin (VST3 / CLAP / JSFX), not the extension. The extension SDK is the wrong format for that job, and this is the single most important thing for Daniel to internalize before scoping anything.

The three honest options

Option A — a real VSTi/instrument plugin (JUCE or bare VST3/CLAP SDK) that reads ReaSampler's banks. A separate build artifact: a VST3 (and/or CLAP) sampler plugin that the user instantiates on an instrument track. It reads the bank JSON + WAV layout ReaSampler writes, maps samples across the keyboard, and plays them MIDI-triggered with a real voice engine. This is the "sophisticated sampler" answer.

• Gives you: everything an instrument is — polyphony, velocity, envelopes, the works, fully integrated into REAPER's routing/render/record path like any VSTi.
• Costs: a second codebase in a second plugin format, almost certainly a new dependency (JUCE is the pragmatic choice; bare VST3 SDK is more code, CLAP is leaner but younger). It is a real DSP/voice-engine build, not a weekend. Cross-platform DSP, its own build/release/signing story, its own UI toolkit. This is a product-sized commitment, not a feature.

Option B — a JSFX sampler. JSFX is REAPER's built-in scriptable plugin format (text .jsfx files, JIT-compiled by REAPER, hostable as an instrument). A JSFX instrument can receive live MIDI and emit audio on a track. It can load samples (Xen-style file reads / the JSFX file/serialize API) and play them back.

• Gives you: a real in-track instrument with zero new binary, zero new SDK, zero JUCE — ships as a text file alongside the extension, cross-platform for free (REAPER runs the JIT everywhere it runs).
• Costs: JSFX is a constrained DSP scripting language, not C++. A polyphonic multisample engine with round-robin/velocity-layers/streaming is doable but you're writing DSP in JSFX's idiom, and large-sample streaming / disk I/O is more awkward than in a native plugin. Reading ReaSampler's JSON index from JSFX is friction (JSFX is not a general-purpose file parser). Best fit for a minimal-to-mid sampler, a real ceiling for a sophisticated one.

Option C — the hybrid (recommended framing to explore first). Keep the extension as the sole owner of capture + organization (which is its whole existing identity and the load-bearing "capture and placement are separate acts" principle). Add a separate instrument (Option A or B) that consumes the banks as a shared-artifact contract. The extension never becomes an instrument; the instrument never captures. Each does what its format is good at. This is the honest shape of the whole thing — the two options above are really "which instrument technology" within the hybrid, because the extension is staying regardless.

The real fork, then, is not "extension vs. plugin" (the extension stays either way) — it is "which instrument format consumes the banks: JSFX or native VSTi/CLAP," and "how much sampler do we actually want." Those are §3 and §4.

---

2. The integration seam

The obvious shared artifact is what ReaSampler already produces: the bank folder (project-relative WAVs) + the bank/index JSON. The question is whether that JSON is sufficient for a playback engine, or whether playback needs mapping data the index doesn't carry today.

What the current index carries (from bank_model's Sample, per CONTEXT.md §Data model): id, display name, relative path, source range, channel count, sample rate, length, capture tempo, an optional key, peak/RMS/LUFS, content hash, tier, provenance, timestamp. Notably it already has an optional key field and capture tempo — the seeds of pitch-mapping are there.

What a playback engine additionally needs — none of which the index carries today:

• Root note (the pitch the sample was recorded at, so it can be repitched across the keyboard). The optional key is close but is "musical key," not "root MIDI note" — different thing.
• Key range / zone (low note, high note) — which keys trigger this sample.
• Velocity layers (which sample plays at which velocity band; a mapping of samples to velocity zones).
• Round-robin groups (cycle through N samples on repeated same-note hits).
• Loop points (sustain loop start/end for held notes; sample-accurate, zero-crossing-aware — CONTEXT already flags loop/zero-crossing handling as day-one-relevant for wavetable material).
• Amplitude envelope (ADSR) and optionally filter/pitch envelopes.
• Tuning/gain trim per sample.

The design decision this forces: does the mapping (key ranges, velocity layers, round-robin, envelopes) live —

• (i) in the bank index (ReaSampler owns the instrument definition; the plugin is a dumb player of a ReaSampler-authored map), or
• (ii) in the instrument plugin's own state (the plugin owns the map; the bank is just a WAV+metadata pool it maps over), or
• (iii) split — the bank carries per-sample intrinsics (root note, loop points — facts about the file), and the instrument carries the arrangement (zones, layers, envelopes — a performance choice)?

(iii) is the principled answer and worth leading with. It mirrors ReaSampler's own deepest instinct — the capture/placement separation. Root note and loop points are facts about the captured file (analogous to sample rate, length, peaks) and belong in the bank, computed or set at/after capture. Key zones, velocity layers, envelopes, round-robin are a performance mapping — a creative arrangement of those facts — and belong in the instrument. This keeps the bank a clean, portable, tool-agnostic library (a bank is still just WAVs + facts, readable by anything) while the instrument owns the opinionated part. It also means the bank index grows only by file-intrinsic fields (root note, loop points), which is a small, safe, additive change — exactly the kind of seam ReaSampler already knows how to add (mirror of how provenance was added as a field in M1 and populated later).

A note on format: if the instrument is native (Option A), it reads the bank JSON directly — trivial. If it's JSFX (Option B), reading arbitrary JSON is friction; the seam might need a simpler sidecar (a flat .txt/key-value map ReaSampler writes next to the bank, JSFX-parseable) rather than making JSFX parse the index JSON. That's a concrete cost of the JSFX path and a reason the seam design and the format choice are coupled.

Portability caveat: the bank is project-relative and travels with the .rpp (M4 machinery). Any instrument consuming it must resolve paths the same way — so the instrument needs to know the current project bank folder. A native plugin instance on a track can be told its folder (saved in plugin state); a JSFX likewise. This is a solvable wiring detail but a real one — flag it, don't hand-wave it.

---

3. Product shape & scope tiers

Sketched as tiers, minimal → sophisticated. The point of tiering is that Tier 0 delivers the core promise ("play my captured banks from a MIDI keyboard") and each tier above is optional depth, not a prerequisite for the one below.

Tier 0 — "the bank plays." One sample mapped chromatically across the keyboard from a root note; monophonic-or-basic-polyphony; a simple amp envelope; velocity → volume. Point one bank sample at a MIDI track and play it repitched. This is the smallest thing that delivers the promise and is the honest MVP. On the JSFX path this is a genuinely small build; on the native path it's the skeleton of the plugin.

Tier 1 — "a keymap." Multiple samples zoned across the keyboard (key ranges), each with its own root note. Now a captured kit (several one-shots) or a multisampled instrument (same instrument sampled at several pitches) plays correctly. This is where the root-note + key-range seam fields (§2) earn their place. Still one sample per key-region.

Tier 2 — "expressive." Velocity layers (soft vs. hard hits pick different samples), round-robin (repeated hits cycle samples — the anti-machine-gun feature), full ADSR, per-sample tuning/gain trim, sustain loops for held notes. This is a real sampler and where "sophisticated" lives. It's also where the mapping data (§2) gets rich enough that the "who owns the map" fork (i/ii/iii) really bites.

Tier 3 — "instrument polish." Filter + filter envelope, LFOs, per-voice pan, choke groups (hi-hat open/closed), maybe a modest built-in FX slot. Standard sampler feature creep. Explicitly not needed to deliver value; a direction to leave room for, not commit to.

The recommended reading: Tier 0–1 is the "does this even belong in ReaSampler's world" proof. If the answer is yes, Tier 2 is where it becomes a tool people reach for. Tier 3 is optional forever. Scope the decision at Tier 0–1 first and treat 2/3 as held futures — don't let a Tier-3 feature list drive the format choice.

The one-source-multiple-views angle

Worth flagging because it recurs in Daniel's work: the bank is one source (WAVs + facts). The panel is one view (organize/audition). A MIDI instrument is another view of the same source (play). That's a clean framing — the instrument doesn't fork the bank, it consumes it — and it argues for keeping the bank as the single authoritative artifact and the instrument as a pure consumer (seam option ii/iii, never "the instrument keeps its own copy of the samples").

---

4. Risks & open decisions — Daniel's to call

None of these are pre-decided here. Each is a genuine fork.

D1 — Instrument format: JSFX vs. native VSTi (VST3/CLAP, likely via JUCE). The central fork. JSFX = no new binary, no JUCE, cross-platform free, real ceiling on sophistication and awkward bank-JSON reading. Native = full sophistication and clean JSON integration, at the cost of a whole second codebase in a second format with its own build/release/dependency/signing story. My lean, for discussion only: if the goal is Tier 0–1, prototype in JSFX first — it proves the seam and the value with near-zero format commitment, and if it hits a ceiling the seam you designed still serves a later native build. Reach for native when Tier 2+ is a firm goal, not before. But this is squarely Daniel's call and depends on how sophisticated he actually wants this.

D2 — Whether to take a JUCE (or any external plugin-SDK) dependency at all. The project today is a clean C++ extension with two vendored submodules and a proud pure-core/shell discipline. A native instrument means a third major dependency and a second build target of a fundamentally different kind. That's a real architectural weight. JSFX sidesteps it entirely. Flagging it as its own decision because "should we depend on JUCE" is a bigger standing commitment than "should we build a sampler."

D3 — Does the pure-core discipline survive the format boundary? ReaSampler's identity is pure REAPER-free testable core + thin shells. A sampler's voice engine, envelope math, key/velocity mapping, and repitch logic are exactly the kind of pure, testable core this project excels at — they could live in a REAPER-free, DAW-free, unit-tested module (mirror of bank_model/peaks/view_mode_model) with the plugin format (JSFX or VST3 wrapper) as the thin shell around it. This is the most ReaSampler-native way to build it and I'd argue strongly for it regardless of D1: the sampler DSP core is pure and tested; the format is a shell. The open question is whether that discipline can hold across a different plugin format — with native it's clean C++ so yes; with JSFX the "pure core" would be JSFX code, harder to unit-test in the CTest harness. That tension is real and feeds back into D1.

D4 — Where the mapping lives (seam fork i/ii/iii from §2). Bank-owned map, instrument-owned map, or split (file-intrinsics in the bank, performance-map in the instrument). My lean: (iii) split — it's the one that honors ReaSampler's capture/placement instinct and keeps the bank tool-agnostic. But it's Daniel's call whether ReaSampler should author instrument definitions at all, or stay purely a sample library that a separate mapping tool arranges.

D5 — Cross-platform. The extension is already cross-platform (SWELL). JSFX inherits that for free. A native plugin re-opens the full cross-platform DSP + UI + build matrix (Win/mac/Linux, code-signing on mac, etc.) as a separate artifact. A cost that lands entirely on the native path.

D6 — Is this even one product? The reframe worth surfacing: ReaSampler's thesis is "a precision capture/organize tool; capture and placement are separate acts." A MIDI-playback instrument is a different act — playback. There's a legitimate reading where the instrument is a companion product that shares the bank format, not a feature of ReaSampler — the way a sample library and a sampler that reads it are related-but-distinct products. That framing might keep ReaSampler sharp (it stays the capture tool it is) while letting the instrument evolve on its own clock and format. The alternative reading — it's all one integrated sampler-workstation — is also coherent. Which of those two ReaSampler is is the highest-order question here, and it's Daniel's to answer before format/tier decisions mean much.

---

What this doc is asking for

A direction on the two highest-order forks, in order:

1. D6 — one product or two? Is the instrument a feature of ReaSampler, or a companion product sharing the bank format? Everything else sits under this.
2. D1 — JSFX-first prototype, or straight to native? Given a target of Tier 0–1 to start, and the pure-core-as-shell discipline (D3) held either way.

Once those two are called, the seam fields (§2, D4) and the tier scope (§3) become concrete enough to write an actual phase spec. Until then this stays a framing doc with no PLAN.md footprint — deliberately, so we don't scope an instrument before deciding whether we're building one.