Add MIDI-playback opportunity & design-space framing note
Frame the option space for a MIDI-triggered sampler consuming ReaSampler banks: the extension-format ceiling (verified vs. vendored SDK headers), the bank-as-shared-artifact seam, scope tiers, and the forks for Daniel.
This commit is contained in:
@@ -0,0 +1,312 @@
|
||||
# MIDI playback — opportunity & design-space framing
|
||||
|
||||
Framing for a **MIDI-triggered audio sampler** that plays back ReaSampler's captured
|
||||
banks. This is a **discussion-shaping doc, not a build plan** — no phase, no PLAN.md
|
||||
points, no settled forks yet. It exists so Daniel and the-boss can react to an honest
|
||||
map of the option space before anything is scoped.
|
||||
|
||||
Status: framed by product-designer (2026-07-26). Grounded in the vendored REAPER SDK
|
||||
headers (`vendor/reaper-sdk/sdk/reaper_plugin.h`, `reaper_plugin_functions.h`) — the
|
||||
plugin-format claims below are checked against those headers, not asserted. The
|
||||
genuine forks are flagged as **Daniel's to decide**; nothing here pre-decides them.
|
||||
|
||||
---
|
||||
|
||||
## The need, stated plainly
|
||||
|
||||
ReaSampler today **captures and organizes** audio into per-project banks (on-disk
|
||||
32-bit float WAVs + a JSON index, multi-bank via `bank_book`). It has **no way to
|
||||
play those samples back from a MIDI keyboard.** The panel can *audition* a sample on
|
||||
click — but a one-shot, one-voice audition is not an instrument. There is no
|
||||
key→sample mapping, no velocity response, no polyphony, no envelope, no round-robin.
|
||||
|
||||
The ask: *what would let me play my captured banks MIDI-triggered, and what does the
|
||||
REAPER surface actually allow toward that?*
|
||||
|
||||
The rest of this doc is organized as the task framed it: (1) the plugin-format
|
||||
reality, honestly; (2) the integration seam between capture and playback; (3) product
|
||||
shape and scope tiers; (4) the forks that are Daniel's to call.
|
||||
|
||||
---
|
||||
|
||||
## 1. The plugin-format reality (the honest part)
|
||||
|
||||
**The load-bearing fact: a REAPER extension cannot be a MIDI-triggered instrument.**
|
||||
This is not a limitation we can engineer around by trying harder — it is what the
|
||||
plugin *format* is. ReaSampler today is a `reaper_*.dll` loaded at startup via the
|
||||
extension API (`ReaperPluginEntry`). That format gives an extension enormous reach
|
||||
over the *project* — tracks, items, actions, rendering, project state — but it is
|
||||
**not a node in any track's audio signal chain.** An instrument track's "read live
|
||||
MIDI, emit audio per-voice" contract belongs to VST/VST3/CLAP/JSFX plugins, which
|
||||
REAPER hosts through an entirely different mechanism.
|
||||
|
||||
What the extension SDK *does* expose near audio/MIDI, verified in the headers, and why
|
||||
none of it is "be an instrument":
|
||||
|
||||
- **`Audio_RegHardwareHook` + `audio_hook_register_t::OnAudioBuffer`** (`reaper_plugin.h`
|
||||
~1228). This is a real-time callback into the audio thread, called pre- and
|
||||
post-REAPER-processing, handing you `GetBuffer(isOutput, idx)` for the **hardware
|
||||
I/O** channels. It is a *hardware tap*, not a track insert. You could technically
|
||||
mix sample playback into the hardware output buffer here — but you'd be bypassing
|
||||
REAPER's entire routing, metering, recording, and rendering path. Output from this
|
||||
hook **cannot be recorded, rendered, routed, or FX-processed** as track audio. It is
|
||||
the wrong tool for an instrument by construction, and it runs in the audio thread
|
||||
with all the real-time-safety hazards that implies. Rejected on sight for this
|
||||
purpose — noting it only to be complete.
|
||||
|
||||
- **`kbd_OnMidiEvent` / `kbd_OnMidiList`** (`reaper_plugin_functions.h` ~4138). REAPER
|
||||
*informs* the extension of MIDI events (for action-triggering — this is how MIDI can
|
||||
fire the capture/insert actions we already plan to bind). It is **notification, not a
|
||||
routable MIDI input** feeding a synth voice allocator. Great for "MIDI note fires the
|
||||
capture-to-slot action"; useless as an instrument's note input.
|
||||
|
||||
- **`PlayPreview` / `PlayTrackPreview` + `preview_register_t`** (`reaper_plugin.h`
|
||||
~1302). This is REAPER's **preview-playback** mechanism, and **ReaSampler already
|
||||
uses it** — `bank_panel.cpp` streams a bank WAV through a caller-owned
|
||||
`preview_register_t` for click-audition (`PCM_Source_CreateFromFile` +
|
||||
`PlayPreview`). This is the one audio-emitting surface the extension already drives,
|
||||
and it is load-bearing for the hybrid option below. But note its shape: it plays *a*
|
||||
source at *a* position with *a* volume — it is a fire-and-forget preview, **not a
|
||||
polyphonic, per-note, velocity-scaled, envelope-shaped voice engine.** You do not get
|
||||
MIDI→voice allocation for free; you'd be building a sampler engine on top of a
|
||||
preview primitive.
|
||||
|
||||
- **`pcmsrc` registration + `PCM_source` subclassing** (`reaper_plugin.h` ~804, the
|
||||
`PCM_SOURCE_EXT_*` surface). An extension *can* register a custom `PCM_source` type.
|
||||
This is the surface REAPER itself and extensions like SWS use for custom media. It is
|
||||
a real seam — but it produces a *media source* (something an item plays), not a
|
||||
*live-MIDI instrument*. Interesting for "a bank-backed media source" but not the
|
||||
keyboard-instrument ask.
|
||||
|
||||
**Conclusion, stated for the record:** to get a genuine MIDI-triggered instrument —
|
||||
play a note, hear the mapped sample, velocity-scaled, polyphonic, with an envelope —
|
||||
the playback engine must be a **standard instrument plugin** (VST3 / CLAP / JSFX),
|
||||
*not* the extension. The extension SDK is the wrong format for that job, and this is
|
||||
the single most important thing for Daniel to internalize before scoping anything.
|
||||
|
||||
### The three honest options
|
||||
|
||||
**Option A — a real VSTi/instrument plugin (JUCE or bare VST3/CLAP SDK) that reads
|
||||
ReaSampler's banks.** A separate build artifact: a VST3 (and/or CLAP) sampler plugin
|
||||
that the user instantiates on an instrument track. It reads the bank JSON + WAV layout
|
||||
ReaSampler writes, maps samples across the keyboard, and plays them MIDI-triggered with
|
||||
a real voice engine. This is the "sophisticated sampler" answer.
|
||||
- *Gives you:* everything an instrument is — polyphony, velocity, envelopes, the works,
|
||||
fully integrated into REAPER's routing/render/record path like any VSTi.
|
||||
- *Costs:* a **second codebase in a second plugin format**, almost certainly a new
|
||||
dependency (JUCE is the pragmatic choice; bare VST3 SDK is more code, CLAP is leaner
|
||||
but younger). It is a real DSP/voice-engine build, not a weekend. Cross-platform DSP,
|
||||
its own build/release/signing story, its own UI toolkit. This is a **product-sized
|
||||
commitment**, not a feature.
|
||||
|
||||
**Option B — a JSFX sampler.** JSFX is REAPER's built-in scriptable plugin format
|
||||
(text `.jsfx` files, JIT-compiled by REAPER, hostable as an instrument). A JSFX
|
||||
instrument *can* receive live MIDI and emit audio on a track. It can load samples
|
||||
(`Xen`-style file reads / the JSFX file/serialize API) and play them back.
|
||||
- *Gives you:* a real in-track instrument with **zero new binary, zero new SDK, zero
|
||||
JUCE** — ships as a text file alongside the extension, cross-platform for free
|
||||
(REAPER runs the JIT everywhere it runs).
|
||||
- *Costs:* JSFX is a constrained DSP scripting language, not C++. A polyphonic
|
||||
multisample engine with round-robin/velocity-layers/streaming is *doable* but you're
|
||||
writing DSP in JSFX's idiom, and large-sample streaming / disk I/O is more awkward
|
||||
than in a native plugin. Reading ReaSampler's JSON index from JSFX is friction (JSFX
|
||||
is not a general-purpose file parser). Best fit for a **minimal-to-mid** sampler, a
|
||||
real ceiling for a **sophisticated** one.
|
||||
|
||||
**Option C — the hybrid (recommended framing to explore first).** Keep the extension
|
||||
as the sole owner of **capture + organization** (which is its whole existing identity
|
||||
and the load-bearing "capture and placement are separate acts" principle). Add a
|
||||
**separate instrument** (Option A *or* B) that **consumes the banks** as a
|
||||
shared-artifact contract. The extension never becomes an instrument; the instrument
|
||||
never captures. Each does what its format is good at. This is the honest shape of the
|
||||
whole thing — the two options above are really "which instrument technology" *within*
|
||||
the hybrid, because the extension is staying regardless.
|
||||
|
||||
The real fork, then, is **not** "extension vs. plugin" (the extension stays either
|
||||
way) — it is **"which instrument format consumes the banks: JSFX or native VSTi/CLAP,"**
|
||||
and **"how much sampler do we actually want."** Those are §3 and §4.
|
||||
|
||||
---
|
||||
|
||||
## 2. The integration seam
|
||||
|
||||
The obvious shared artifact is what ReaSampler already produces: **the bank folder
|
||||
(project-relative WAVs) + the bank/index JSON.** The question is whether that JSON is
|
||||
*sufficient* for a playback engine, or whether playback needs mapping data the index
|
||||
doesn't carry today.
|
||||
|
||||
**What the current index carries** (from `bank_model`'s `Sample`, per CONTEXT.md §Data
|
||||
model): id, display name, relative path, source range, channel count, sample rate,
|
||||
length, capture tempo, an **optional key**, peak/RMS/LUFS, content hash, tier,
|
||||
provenance, timestamp. Notably it *already* has an optional key field and capture
|
||||
tempo — the seeds of pitch-mapping are there.
|
||||
|
||||
**What a playback engine additionally needs** — none of which the index carries today:
|
||||
|
||||
- **Root note** (the pitch the sample was recorded at, so it can be repitched across
|
||||
the keyboard). The optional `key` is close but is "musical key," not "root MIDI
|
||||
note" — different thing.
|
||||
- **Key range / zone** (low note, high note) — which keys trigger this sample.
|
||||
- **Velocity layers** (which sample plays at which velocity band; a mapping of samples
|
||||
to velocity zones).
|
||||
- **Round-robin groups** (cycle through N samples on repeated same-note hits).
|
||||
- **Loop points** (sustain loop start/end for held notes; sample-accurate,
|
||||
zero-crossing-aware — CONTEXT already flags loop/zero-crossing handling as
|
||||
day-one-relevant for wavetable material).
|
||||
- **Amplitude envelope** (ADSR) and optionally filter/pitch envelopes.
|
||||
- **Tuning/gain trim** per sample.
|
||||
|
||||
**The design decision this forces:** does the *mapping* (key ranges, velocity layers,
|
||||
round-robin, envelopes) live —
|
||||
|
||||
- **(i) in the bank index** (ReaSampler owns the instrument definition; the plugin is a
|
||||
dumb player of a ReaSampler-authored map), or
|
||||
- **(ii) in the instrument plugin's own state** (the plugin owns the map; the bank is
|
||||
just a WAV+metadata pool it maps *over*), or
|
||||
- **(iii) split** — the bank carries per-sample *intrinsics* (root note, loop points —
|
||||
facts about the file), and the instrument carries the *arrangement* (zones, layers,
|
||||
envelopes — a performance choice)?
|
||||
|
||||
**(iii) is the principled answer and worth leading with.** It mirrors ReaSampler's own
|
||||
deepest instinct — the *capture/placement separation*. Root note and loop points are
|
||||
**facts about the captured file** (analogous to sample rate, length, peaks) and belong
|
||||
in the bank, computed or set at/after capture. Key zones, velocity layers, envelopes,
|
||||
round-robin are a **performance mapping** — a creative arrangement of those facts — and
|
||||
belong in the instrument. This keeps the bank a clean, portable, tool-agnostic library
|
||||
(a bank is still just WAVs + facts, readable by anything) while the instrument owns the
|
||||
opinionated part. It also means **the bank index grows only by file-intrinsic fields**
|
||||
(root note, loop points), which is a small, safe, additive change — exactly the kind of
|
||||
seam ReaSampler already knows how to add (mirror of how `provenance` was added as a
|
||||
field in M1 and populated later).
|
||||
|
||||
**A note on format:** if the instrument is native (Option A), it reads the bank JSON
|
||||
directly — trivial. If it's JSFX (Option B), reading arbitrary JSON is friction; the
|
||||
seam might need a **simpler sidecar** (a flat `.txt`/key-value map ReaSampler writes
|
||||
next to the bank, JSFX-parseable) rather than making JSFX parse the index JSON. That's
|
||||
a concrete cost of the JSFX path and a reason the seam design and the format choice are
|
||||
coupled.
|
||||
|
||||
**Portability caveat:** the bank is project-relative and travels with the `.rpp`
|
||||
(M4 machinery). Any instrument consuming it must resolve paths the same way — so the
|
||||
instrument needs to know the *current* project bank folder. A native plugin instance
|
||||
on a track can be told its folder (saved in plugin state); a JSFX likewise. This is a
|
||||
solvable wiring detail but a real one — flag it, don't hand-wave it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Product shape & scope tiers
|
||||
|
||||
Sketched as tiers, minimal → sophisticated. The point of tiering is that **Tier 0
|
||||
delivers the core promise** ("play my captured banks from a MIDI keyboard") and each
|
||||
tier above is optional depth, not a prerequisite for the one below.
|
||||
|
||||
**Tier 0 — "the bank plays."** One sample mapped chromatically across the keyboard from
|
||||
a root note; monophonic-or-basic-polyphony; a simple amp envelope; velocity → volume.
|
||||
Point one bank sample at a MIDI track and play it repitched. *This is the smallest
|
||||
thing that delivers the promise* and is the honest MVP. On the JSFX path this is a
|
||||
genuinely small build; on the native path it's the skeleton of the plugin.
|
||||
|
||||
**Tier 1 — "a keymap."** Multiple samples zoned across the keyboard (key ranges), each
|
||||
with its own root note. Now a captured *kit* (several one-shots) or a *multisampled
|
||||
instrument* (same instrument sampled at several pitches) plays correctly. This is where
|
||||
the **root-note + key-range** seam fields (§2) earn their place. Still one sample per
|
||||
key-region.
|
||||
|
||||
**Tier 2 — "expressive."** Velocity layers (soft vs. hard hits pick different samples),
|
||||
round-robin (repeated hits cycle samples — the anti-machine-gun feature), full ADSR,
|
||||
per-sample tuning/gain trim, sustain loops for held notes. This is a *real* sampler and
|
||||
where "sophisticated" lives. It's also where the mapping data (§2) gets rich enough that
|
||||
the "who owns the map" fork (i/ii/iii) really bites.
|
||||
|
||||
**Tier 3 — "instrument polish."** Filter + filter envelope, LFOs, per-voice pan, choke
|
||||
groups (hi-hat open/closed), maybe a modest built-in FX slot. Standard sampler feature
|
||||
creep. Explicitly *not* needed to deliver value; a direction to leave room for, not
|
||||
commit to.
|
||||
|
||||
**The recommended reading:** Tier 0–1 is the "does this even belong in ReaSampler's
|
||||
world" proof. If the answer is yes, Tier 2 is where it becomes a tool people reach for.
|
||||
Tier 3 is optional forever. **Scope the decision at Tier 0–1 first** and treat 2/3 as
|
||||
held futures — don't let a Tier-3 feature list drive the format choice.
|
||||
|
||||
### The one-source-multiple-views angle
|
||||
|
||||
Worth flagging because it recurs in Daniel's work: the bank is **one source** (WAVs +
|
||||
facts). The panel is one view (organize/audition). A MIDI instrument is **another view
|
||||
of the same source** (play). That's a clean framing — the instrument doesn't fork the
|
||||
bank, it consumes it — and it argues for keeping the bank as the single authoritative
|
||||
artifact and the instrument as a pure consumer (seam option ii/iii, never "the
|
||||
instrument keeps its own copy of the samples").
|
||||
|
||||
---
|
||||
|
||||
## 4. Risks & open decisions — Daniel's to call
|
||||
|
||||
None of these are pre-decided here. Each is a genuine fork.
|
||||
|
||||
**D1 — Instrument format: JSFX vs. native VSTi (VST3/CLAP, likely via JUCE).** The
|
||||
central fork. JSFX = no new binary, no JUCE, cross-platform free, real ceiling on
|
||||
sophistication and awkward bank-JSON reading. Native = full sophistication and clean
|
||||
JSON integration, at the cost of a whole second codebase in a second format with its
|
||||
own build/release/dependency/signing story. *My lean, for discussion only:* if the goal
|
||||
is Tier 0–1, **prototype in JSFX first** — it proves the seam and the value with near-zero
|
||||
format commitment, and if it hits a ceiling the seam you designed still serves a later
|
||||
native build. Reach for native when Tier 2+ is a firm goal, not before. But this is
|
||||
squarely Daniel's call and depends on how sophisticated he actually wants this.
|
||||
|
||||
**D2 — Whether to take a JUCE (or any external plugin-SDK) dependency at all.** The
|
||||
project today is a clean C++ extension with two vendored submodules and a proud
|
||||
pure-core/shell discipline. A native instrument means a *third* major dependency and a
|
||||
second build target of a fundamentally different kind. That's a real architectural
|
||||
weight. JSFX sidesteps it entirely. Flagging it as its own decision because "should we
|
||||
depend on JUCE" is a bigger standing commitment than "should we build a sampler."
|
||||
|
||||
**D3 — Does the pure-core discipline survive the format boundary?** ReaSampler's
|
||||
identity is *pure REAPER-free testable core + thin shells*. A sampler's **voice engine,
|
||||
envelope math, key/velocity mapping, and repitch logic are exactly the kind of pure,
|
||||
testable core** this project excels at — they could live in a REAPER-free, DAW-free,
|
||||
unit-tested module (mirror of `bank_model`/`peaks`/`view_mode_model`) with the plugin
|
||||
format (JSFX or VST3 wrapper) as the thin shell around it. **This is the most
|
||||
ReaSampler-native way to build it** and I'd argue strongly for it regardless of D1: the
|
||||
sampler DSP core is pure and tested; the format is a shell. The open question is whether
|
||||
that discipline can hold across a *different plugin format* — with native it's clean C++
|
||||
so yes; with JSFX the "pure core" would be JSFX code, harder to unit-test in the CTest
|
||||
harness. That tension is real and feeds back into D1.
|
||||
|
||||
**D4 — Where the mapping lives (seam fork i/ii/iii from §2).** Bank-owned map,
|
||||
instrument-owned map, or split (file-intrinsics in the bank, performance-map in the
|
||||
instrument). *My lean:* **(iii) split** — it's the one that honors ReaSampler's
|
||||
capture/placement instinct and keeps the bank tool-agnostic. But it's Daniel's call
|
||||
whether ReaSampler should author instrument definitions at all, or stay purely a
|
||||
sample library that a *separate* mapping tool arranges.
|
||||
|
||||
**D5 — Cross-platform.** The extension is already cross-platform (SWELL). JSFX inherits
|
||||
that for free. A native plugin re-opens the full cross-platform DSP + UI + build matrix
|
||||
(Win/mac/Linux, code-signing on mac, etc.) as a *separate* artifact. A cost that lands
|
||||
entirely on the native path.
|
||||
|
||||
**D6 — Is this even one product?** The reframe worth surfacing: ReaSampler's thesis is
|
||||
"a precision *capture/organize* tool; capture and placement are separate acts." A
|
||||
MIDI-playback instrument is a *different act* — playback. There's a legitimate reading
|
||||
where the instrument is a **companion product** that shares the bank format, not a
|
||||
feature *of* ReaSampler — the way a sample library and a sampler that reads it are
|
||||
related-but-distinct products. That framing might keep ReaSampler sharp (it stays the
|
||||
capture tool it is) while letting the instrument evolve on its own clock and format.
|
||||
The alternative reading — it's all one integrated sampler-workstation — is also
|
||||
coherent. **Which of those two ReaSampler *is* is the highest-order question here, and
|
||||
it's Daniel's to answer before format/tier decisions mean much.**
|
||||
|
||||
---
|
||||
|
||||
## What this doc is asking for
|
||||
|
||||
A direction on the two highest-order forks, in order:
|
||||
|
||||
1. **D6 — one product or two?** Is the instrument a feature of ReaSampler, or a
|
||||
companion product sharing the bank format? Everything else sits under this.
|
||||
2. **D1 — JSFX-first prototype, or straight to native?** Given a target of Tier 0–1 to
|
||||
start, and the pure-core-as-shell discipline (D3) held either way.
|
||||
|
||||
Once those two are called, the seam fields (§2, D4) and the tier scope (§3) become
|
||||
concrete enough to write an actual phase spec. Until then this stays a framing doc with
|
||||
no PLAN.md footprint — deliberately, so we don't scope an instrument before deciding
|
||||
whether we're building one.
|
||||
Reference in New Issue
Block a user