reasampler/docs/product/midi-playback.md

# MIDI playback — opportunity & design-space framing

Framing for a **MIDI-triggered audio sampler** that plays back ReaSampler's captured
banks. This is a **discussion-shaping doc, not a build plan** — no phase, no PLAN.md
points, no settled forks yet. It exists so Daniel and the-boss can react to an honest
map of the option space before anything is scoped.

Status: framed by product-designer (2026-07-26). Grounded in the vendored REAPER SDK
headers (`vendor/reaper-sdk/sdk/reaper_plugin.h`, `reaper_plugin_functions.h`) — the
plugin-format claims below are checked against those headers, not asserted. The
genuine forks are flagged as **Daniel's to decide**; nothing here pre-decides them.

---

## The need, stated plainly

ReaSampler today **captures and organizes** audio into per-project banks (on-disk
32-bit float WAVs + a JSON index, multi-bank via `bank_book`). It has **no way to
play those samples back from a MIDI keyboard.** The panel can *audition* a sample on
click — but a one-shot, one-voice audition is not an instrument. There is no
key→sample mapping, no velocity response, no polyphony, no envelope, no round-robin.

The ask: *what would let me play my captured banks MIDI-triggered, and what does the
REAPER surface actually allow toward that?*

The rest of this doc is organized as the task framed it: (1) the plugin-format
reality, honestly; (2) the integration seam between capture and playback; (3) product
shape and scope tiers; (4) the forks that are Daniel's to call.

---

## 1. The plugin-format reality (the honest part)

**The load-bearing fact: a REAPER extension cannot be a MIDI-triggered instrument.**
This is not a limitation we can engineer around by trying harder — it is what the
plugin *format* is. ReaSampler today is a `reaper_*.dll` loaded at startup via the
extension API (`ReaperPluginEntry`). That format gives an extension enormous reach
over the *project* — tracks, items, actions, rendering, project state — but it is
**not a node in any track's audio signal chain.** An instrument track's "read live
MIDI, emit audio per-voice" contract belongs to VST/VST3/CLAP/JSFX plugins, which
REAPER hosts through an entirely different mechanism.

What the extension SDK *does* expose near audio/MIDI, verified in the headers, and why
none of it is "be an instrument":

- **`Audio_RegHardwareHook` + `audio_hook_register_t::OnAudioBuffer`** (`reaper_plugin.h`
  ~1228). This is a real-time callback into the audio thread, called pre- and
  post-REAPER-processing, handing you `GetBuffer(isOutput, idx)` for the **hardware
  I/O** channels. It is a *hardware tap*, not a track insert. You could technically
  mix sample playback into the hardware output buffer here — but you'd be bypassing
  REAPER's entire routing, metering, recording, and rendering path. Output from this
  hook **cannot be recorded, rendered, routed, or FX-processed** as track audio. It is
  the wrong tool for an instrument by construction, and it runs in the audio thread
  with all the real-time-safety hazards that implies. Rejected on sight for this
  purpose — noting it only to be complete.

- **`kbd_OnMidiEvent` / `kbd_OnMidiList`** (`reaper_plugin_functions.h` ~4138). REAPER
  *informs* the extension of MIDI events (for action-triggering — this is how MIDI can
  fire the capture/insert actions we already plan to bind). It is **notification, not a
  routable MIDI input** feeding a synth voice allocator. Great for "MIDI note fires the
  capture-to-slot action"; useless as an instrument's note input.

- **`PlayPreview` / `PlayTrackPreview` + `preview_register_t`** (`reaper_plugin.h`
  ~1302). This is REAPER's **preview-playback** mechanism, and **ReaSampler already
  uses it** — `bank_panel.cpp` streams a bank WAV through a caller-owned
  `preview_register_t` for click-audition (`PCM_Source_CreateFromFile` +
  `PlayPreview`). This is the one audio-emitting surface the extension already drives,
  and it is load-bearing for the hybrid option below. But note its shape: it plays *a*
  source at *a* position with *a* volume — it is a fire-and-forget preview, **not a
  polyphonic, per-note, velocity-scaled, envelope-shaped voice engine.** You do not get
  MIDI→voice allocation for free; you'd be building a sampler engine on top of a
  preview primitive.

- **`pcmsrc` registration + `PCM_source` subclassing** (`reaper_plugin.h` ~804, the
  `PCM_SOURCE_EXT_*` surface). An extension *can* register a custom `PCM_source` type.
  This is the surface REAPER itself and extensions like SWS use for custom media. It is
  a real seam — but it produces a *media source* (something an item plays), not a
  *live-MIDI instrument*. Interesting for "a bank-backed media source" but not the
  keyboard-instrument ask.

**Conclusion, stated for the record:** to get a genuine MIDI-triggered instrument —
play a note, hear the mapped sample, velocity-scaled, polyphonic, with an envelope —
the playback engine must be a **standard instrument plugin** (VST3 / CLAP / JSFX),
*not* the extension. The extension SDK is the wrong format for that job, and this is
the single most important thing for Daniel to internalize before scoping anything.

### The three honest options

**Option A — a real VSTi/instrument plugin (JUCE or bare VST3/CLAP SDK) that reads
ReaSampler's banks.** A separate build artifact: a VST3 (and/or CLAP) sampler plugin
that the user instantiates on an instrument track. It reads the bank JSON + WAV layout
ReaSampler writes, maps samples across the keyboard, and plays them MIDI-triggered with
a real voice engine. This is the "sophisticated sampler" answer.
- *Gives you:* everything an instrument is — polyphony, velocity, envelopes, the works,
  fully integrated into REAPER's routing/render/record path like any VSTi.
- *Costs:* a **second codebase in a second plugin format**, almost certainly a new
  dependency (JUCE is the pragmatic choice; bare VST3 SDK is more code, CLAP is leaner
  but younger). It is a real DSP/voice-engine build, not a weekend. Cross-platform DSP,
  its own build/release/signing story, its own UI toolkit. This is a **product-sized
  commitment**, not a feature.

**Option B — a JSFX sampler.** JSFX is REAPER's built-in scriptable plugin format
(text `.jsfx` files, JIT-compiled by REAPER, hostable as an instrument). A JSFX
instrument *can* receive live MIDI and emit audio on a track. It can load samples
(`Xen`-style file reads / the JSFX file/serialize API) and play them back.
- *Gives you:* a real in-track instrument with **zero new binary, zero new SDK, zero
  JUCE** — ships as a text file alongside the extension, cross-platform for free
  (REAPER runs the JIT everywhere it runs).
- *Costs:* JSFX is a constrained DSP scripting language, not C++. A polyphonic
  multisample engine with round-robin/velocity-layers/streaming is *doable* but you're
  writing DSP in JSFX's idiom, and large-sample streaming / disk I/O is more awkward
  than in a native plugin. Reading ReaSampler's JSON index from JSFX is friction (JSFX
  is not a general-purpose file parser). Best fit for a **minimal-to-mid** sampler, a
  real ceiling for a **sophisticated** one.

**Option C — the hybrid (recommended framing to explore first).** Keep the extension
as the sole owner of **capture + organization** (which is its whole existing identity
and the load-bearing "capture and placement are separate acts" principle). Add a
**separate instrument** (Option A *or* B) that **consumes the banks** as a
shared-artifact contract. The extension never becomes an instrument; the instrument
never captures. Each does what its format is good at. This is the honest shape of the
whole thing — the two options above are really "which instrument technology" *within*
the hybrid, because the extension is staying regardless.

The real fork, then, is **not** "extension vs. plugin" (the extension stays either
way) — it is **"which instrument format consumes the banks: JSFX or native VSTi/CLAP,"**
and **"how much sampler do we actually want."** Those are §3 and §4.

---

## 2. The integration seam

The obvious shared artifact is what ReaSampler already produces: **the bank folder
(project-relative WAVs) + the bank/index JSON.** The question is whether that JSON is
*sufficient* for a playback engine, or whether playback needs mapping data the index
doesn't carry today.

**What the current index carries** (from `bank_model`'s `Sample`, per CONTEXT.md §Data
model): id, display name, relative path, source range, channel count, sample rate,
length, capture tempo, an **optional key**, peak/RMS/LUFS, content hash, tier,
provenance, timestamp. Notably it *already* has an optional key field and capture
tempo — the seeds of pitch-mapping are there.

**What a playback engine additionally needs** — none of which the index carries today:

- **Root note** (the pitch the sample was recorded at, so it can be repitched across
  the keyboard). The optional `key` is close but is "musical key," not "root MIDI
  note" — different thing.
- **Key range / zone** (low note, high note) — which keys trigger this sample.
- **Velocity layers** (which sample plays at which velocity band; a mapping of samples
  to velocity zones).
- **Round-robin groups** (cycle through N samples on repeated same-note hits).
- **Loop points** (sustain loop start/end for held notes; sample-accurate,
  zero-crossing-aware — CONTEXT already flags loop/zero-crossing handling as
  day-one-relevant for wavetable material).
- **Amplitude envelope** (ADSR) and optionally filter/pitch envelopes.
- **Tuning/gain trim** per sample.

**The design decision this forces:** does the *mapping* (key ranges, velocity layers,
round-robin, envelopes) live —

- **(i) in the bank index** (ReaSampler owns the instrument definition; the plugin is a
  dumb player of a ReaSampler-authored map), or
- **(ii) in the instrument plugin's own state** (the plugin owns the map; the bank is
  just a WAV+metadata pool it maps *over*), or
- **(iii) split** — the bank carries per-sample *intrinsics* (root note, loop points —
  facts about the file), and the instrument carries the *arrangement* (zones, layers,
  envelopes — a performance choice)?

**(iii) is the principled answer and worth leading with.** It mirrors ReaSampler's own
deepest instinct — the *capture/placement separation*. Root note and loop points are
**facts about the captured file** (analogous to sample rate, length, peaks) and belong
in the bank, computed or set at/after capture. Key zones, velocity layers, envelopes,
round-robin are a **performance mapping** — a creative arrangement of those facts — and
belong in the instrument. This keeps the bank a clean, portable, tool-agnostic library
(a bank is still just WAVs + facts, readable by anything) while the instrument owns the
opinionated part. It also means **the bank index grows only by file-intrinsic fields**
(root note, loop points), which is a small, safe, additive change — exactly the kind of
seam ReaSampler already knows how to add (mirror of how `provenance` was added as a
field in M1 and populated later).

**A note on format:** if the instrument is native (Option A), it reads the bank JSON
directly — trivial. If it's JSFX (Option B), reading arbitrary JSON is friction; the
seam might need a **simpler sidecar** (a flat `.txt`/key-value map ReaSampler writes
next to the bank, JSFX-parseable) rather than making JSFX parse the index JSON. That's
a concrete cost of the JSFX path and a reason the seam design and the format choice are
coupled.

**Portability caveat:** the bank is project-relative and travels with the `.rpp`
(M4 machinery). Any instrument consuming it must resolve paths the same way — so the
instrument needs to know the *current* project bank folder. A native plugin instance
on a track can be told its folder (saved in plugin state); a JSFX likewise. This is a
solvable wiring detail but a real one — flag it, don't hand-wave it.

---

## 3. Product shape & scope tiers

Sketched as tiers, minimal → sophisticated. The point of tiering is that **Tier 0
delivers the core promise** ("play my captured banks from a MIDI keyboard") and each
tier above is optional depth, not a prerequisite for the one below.

**Tier 0 — "the bank plays."** One sample mapped chromatically across the keyboard from
a root note; monophonic-or-basic-polyphony; a simple amp envelope; velocity → volume.
Point one bank sample at a MIDI track and play it repitched. *This is the smallest
thing that delivers the promise* and is the honest MVP. On the JSFX path this is a
genuinely small build; on the native path it's the skeleton of the plugin.

**Tier 1 — "a keymap."** Multiple samples zoned across the keyboard (key ranges), each
with its own root note. Now a captured *kit* (several one-shots) or a *multisampled
instrument* (same instrument sampled at several pitches) plays correctly. This is where
the **root-note + key-range** seam fields (§2) earn their place. Still one sample per
key-region.

**Tier 2 — "expressive."** Velocity layers (soft vs. hard hits pick different samples),
round-robin (repeated hits cycle samples — the anti-machine-gun feature), full ADSR,
per-sample tuning/gain trim, sustain loops for held notes. This is a *real* sampler and
where "sophisticated" lives. It's also where the mapping data (§2) gets rich enough that
the "who owns the map" fork (i/ii/iii) really bites.

**Tier 3 — "instrument polish."** Filter + filter envelope, LFOs, per-voice pan, choke
groups (hi-hat open/closed), maybe a modest built-in FX slot. Standard sampler feature
creep. Explicitly *not* needed to deliver value; a direction to leave room for, not
commit to.

**The recommended reading:** Tier 0–1 is the "does this even belong in ReaSampler's
world" proof. If the answer is yes, Tier 2 is where it becomes a tool people reach for.
Tier 3 is optional forever. **Scope the decision at Tier 0–1 first** and treat 2/3 as
held futures — don't let a Tier-3 feature list drive the format choice.

### The one-source-multiple-views angle

Worth flagging because it recurs in Daniel's work: the bank is **one source** (WAVs +
facts). The panel is one view (organize/audition). A MIDI instrument is **another view
of the same source** (play). That's a clean framing — the instrument doesn't fork the
bank, it consumes it — and it argues for keeping the bank as the single authoritative
artifact and the instrument as a pure consumer (seam option ii/iii, never "the
instrument keeps its own copy of the samples").

---

## 4. Risks & open decisions — Daniel's to call

None of these are pre-decided here. Each is a genuine fork.

**D1 — Instrument format: JSFX vs. native VSTi (VST3/CLAP, likely via JUCE).** The
central fork. JSFX = no new binary, no JUCE, cross-platform free, real ceiling on
sophistication and awkward bank-JSON reading. Native = full sophistication and clean
JSON integration, at the cost of a whole second codebase in a second format with its
own build/release/dependency/signing story. *My lean, for discussion only:* if the goal
is Tier 0–1, **prototype in JSFX first** — it proves the seam and the value with near-zero
format commitment, and if it hits a ceiling the seam you designed still serves a later
native build. Reach for native when Tier 2+ is a firm goal, not before. But this is
squarely Daniel's call and depends on how sophisticated he actually wants this.

**D2 — Whether to take a JUCE (or any external plugin-SDK) dependency at all.** The
project today is a clean C++ extension with two vendored submodules and a proud
pure-core/shell discipline. A native instrument means a *third* major dependency and a
second build target of a fundamentally different kind. That's a real architectural
weight. JSFX sidesteps it entirely. Flagging it as its own decision because "should we
depend on JUCE" is a bigger standing commitment than "should we build a sampler."

**D3 — Does the pure-core discipline survive the format boundary?** ReaSampler's
identity is *pure REAPER-free testable core + thin shells*. A sampler's **voice engine,
envelope math, key/velocity mapping, and repitch logic are exactly the kind of pure,
testable core** this project excels at — they could live in a REAPER-free, DAW-free,
unit-tested module (mirror of `bank_model`/`peaks`/`view_mode_model`) with the plugin
format (JSFX or VST3 wrapper) as the thin shell around it. **This is the most
ReaSampler-native way to build it** and I'd argue strongly for it regardless of D1: the
sampler DSP core is pure and tested; the format is a shell. The open question is whether
that discipline can hold across a *different plugin format* — with native it's clean C++
so yes; with JSFX the "pure core" would be JSFX code, harder to unit-test in the CTest
harness. That tension is real and feeds back into D1.

**D4 — Where the mapping lives (seam fork i/ii/iii from §2).** Bank-owned map,
instrument-owned map, or split (file-intrinsics in the bank, performance-map in the
instrument). *My lean:* **(iii) split** — it's the one that honors ReaSampler's
capture/placement instinct and keeps the bank tool-agnostic. But it's Daniel's call
whether ReaSampler should author instrument definitions at all, or stay purely a
sample library that a *separate* mapping tool arranges.

**D5 — Cross-platform.** The extension is already cross-platform (SWELL). JSFX inherits
that for free. A native plugin re-opens the full cross-platform DSP + UI + build matrix
(Win/mac/Linux, code-signing on mac, etc.) as a *separate* artifact. A cost that lands
entirely on the native path.

**D6 — Is this even one product?** The reframe worth surfacing: ReaSampler's thesis is
"a precision *capture/organize* tool; capture and placement are separate acts." A
MIDI-playback instrument is a *different act* — playback. There's a legitimate reading
where the instrument is a **companion product** that shares the bank format, not a
feature *of* ReaSampler — the way a sample library and a sampler that reads it are
related-but-distinct products. That framing might keep ReaSampler sharp (it stays the
capture tool it is) while letting the instrument evolve on its own clock and format.
The alternative reading — it's all one integrated sampler-workstation — is also
coherent. **Which of those two ReaSampler *is* is the highest-order question here, and
it's Daniel's to answer before format/tier decisions mean much.**

---

## What this doc is asking for

A direction on the two highest-order forks, in order:

1. **D6 — one product or two?** Is the instrument a feature of ReaSampler, or a
   companion product sharing the bank format? Everything else sits under this.
2. **D1 — JSFX-first prototype, or straight to native?** Given a target of Tier 0–1 to
   start, and the pure-core-as-shell discipline (D3) held either way.

Once those two are called, the seam fields (§2, D4) and the tier scope (§3) become
concrete enough to write an actual phase spec. Until then this stays a framing doc with
no PLAN.md footprint — deliberately, so we don't scope an instrument before deciding
whether we're building one.