diff --git a/docs/product/midi-playback.md b/docs/product/midi-playback.md index ed47925..cef7bd7 100644 --- a/docs/product/midi-playback.md +++ b/docs/product/midi-playback.md @@ -5,10 +5,35 @@ banks. This is a **discussion-shaping doc, not a build plan** — no phase, no P points, no settled forks yet. It exists so Daniel and the-boss can react to an honest map of the option space before anything is scoped. -Status: framed by product-designer (2026-07-26). Grounded in the vendored REAPER SDK -headers (`vendor/reaper-sdk/sdk/reaper_plugin.h`, `reaper_plugin_functions.h`) — the -plugin-format claims below are checked against those headers, not asserted. The -genuine forks are flagged as **Daniel's to decide**; nothing here pre-decides them. +Status: framed by product-designer (2026-07-26), **revised 2026-07-26** to correct a +material omission — the earlier draft missed REAPER's **VST-host bridge**: a VST/VST3 +plugin *hosted inside REAPER* can call back into REAPER's own API from within the +plugin, resolving API function pointers by name over the host `audioMaster` callback. +That single fact changes the integration seam (§2), the "one product or two" question +(D6), and the JSFX-vs-native calculus (D1/D2). The corrections are made plainly +in-place, not appended as a footnote; where the prior draft was too pessimistic about +what a native instrument can reach, it is fixed and the correction is flagged. Grounded +in the vendored REAPER SDK headers (`vendor/reaper-sdk/sdk/reaper_plugin.h`, +`reaper_plugin_functions.h`, `video_processor.h`, `reaper_plugin_fx_embed.h`) and +REAPER's published VST-extensions SDK page (`reaper.fm/sdk/vst/vst_ext.php`) — the +plugin-format claims below are checked against those, not asserted. The genuine forks +are flagged as **Daniel's to decide**; nothing here pre-decides them. + +> **The bridge, stated once, up front (the correction).** A hosted VST is not limited to +> scraping bank WAVs + JSON off disk. REAPER hands the plugin its host callback; calling +> `hostcb(&effect, 0xdeadbeef, 0xdeadf00d, 0, "FunctionName", 0.0)` resolves *any* REAPER +> API function by name — the same string-keyed API surface the extension uses (verified: +> `video_processor.h` line 44 imports `video_CreateVideoProcessor` exactly this way; +> `GetProjExtState`/`SetProjExtState`/`EnumProjExtState` are ordinary entries in that same +> string-keyed table, `reaper_plugin_functions.h` lines 8376/8796/9969). The plugin can +> also fetch its **host context** — the track/take/project it's instantiated in (sibling +> opcode `0xdeadf00e`, `video_processor.h` line 40; CLAP has `clap_get_reaper_context`, +> `reaper_plugin.h` line 142). **Consequence:** a native ReaSampler instrument can read +> the *same* `"reasampler"` project ext-state that `persist` writes — live, project-aware, +> following the active project — not a file it re-parses off disk. This is a +> REAPER-VST-specific capability (it exists because the plugin is hosted *in REAPER*), and +> it is the thumb on the scale the prior draft failed to weigh. The cost of leaning on it +> is stated honestly in D1/D2: it makes the native plugin **REAPER-coupled.** --- @@ -88,16 +113,25 @@ the single most important thing for Daniel to internalize before scoping anythin **Option A — a real VSTi/instrument plugin (JUCE or bare VST3/CLAP SDK) that reads ReaSampler's banks.** A separate build artifact: a VST3 (and/or CLAP) sampler plugin -that the user instantiates on an instrument track. It reads the bank JSON + WAV layout -ReaSampler writes, maps samples across the keyboard, and plays them MIDI-triggered with -a real voice engine. This is the "sophisticated sampler" answer. +that the user instantiates on an instrument track. It maps samples across the keyboard +and plays them MIDI-triggered with a real voice engine. This is the "sophisticated +sampler" answer. - *Gives you:* everything an instrument is — polyphony, velocity, envelopes, the works, - fully integrated into REAPER's routing/render/record path like any VSTi. + fully integrated into REAPER's routing/render/record path like any VSTi. **And, + because it's hosted in REAPER, it reaches ReaSampler's project state directly via the + VST-host bridge** — it doesn't merely parse bank JSON off disk, it can read the live + `"reasampler"` ext-state `persist` writes, know its own host project, and follow the + active project. The prior draft treated Option A as a divorced file-reader; that was + the omission. A native ReaSampler instrument can be **natively, tightly integrated with + the extension's project state**, not loosely coupled through a file format alone. - *Costs:* a **second codebase in a second plugin format**, almost certainly a new dependency (JUCE is the pragmatic choice; bare VST3 SDK is more code, CLAP is leaner but younger). It is a real DSP/voice-engine build, not a weekend. Cross-platform DSP, its own build/release/signing story, its own UI toolkit. This is a **product-sized - commitment**, not a feature. + commitment**, not a feature. And leaning on the bridge for state means the instrument + is **REAPER-coupled** — it stops being a portable VST that runs in any host and becomes + a REAPER companion. That may be exactly right (see D6), but it is a real narrowing and + must be a conscious choice, not a side effect. **Option B — a JSFX sampler.** JSFX is REAPER's built-in scriptable plugin format (text `.jsfx` files, JIT-compiled by REAPER, hostable as an instrument). A JSFX @@ -112,6 +146,17 @@ instrument *can* receive live MIDI and emit audio on a track. It can load sample than in a native plugin. Reading ReaSampler's JSON index from JSFX is friction (JSFX is not a general-purpose file parser). Best fit for a **minimal-to-mid** sampler, a real ceiling for a **sophisticated** one. +- *What JSFX can't match, now that the bridge is on the table:* the VST-host bridge is a + **VST/VST3/CLAP capability** — it is how a *hosted plugin* resolves REAPER API pointers. + JSFX is not a hosted VST; it has REAPER integration of its own kind (`ext_noinit`, + `file_*` ops, the shared `gmem[]`, `slider`/parameter plumbing), but it does **not** get + the same string-keyed REAPER-API surface a native VST does, and in particular has no + clean, native path to read `persist`'s `"reasampler"` project ext-state the way a native + VST can. On the JSFX path, the seam is genuinely "read a file ReaSampler wrote next to + the bank" (hence the sidecar note in §2). This asymmetry is new information: the bridge + makes **native meaningfully more integratable** than the prior draft assumed — it's not + just "native is more powerful DSP," it's "native can be *part of ReaSampler's state + model*, JSFX stays a file-coupled consumer." **Option C — the hybrid (recommended framing to explore first).** Keep the extension as the sole owner of **capture + organization** (which is its whole existing identity @@ -130,10 +175,28 @@ and **"how much sampler do we actually want."** Those are §3 and §4. ## 2. The integration seam -The obvious shared artifact is what ReaSampler already produces: **the bank folder -(project-relative WAVs) + the bank/index JSON.** The question is whether that JSON is -*sufficient* for a playback engine, or whether playback needs mapping data the index -doesn't carry today. +The prior draft assumed one seam: **the bank folder (project-relative WAVs) + the +bank/index JSON on disk**, with the playback engine as a file-reader. That's still the +seam for *sample audio* (WAVs are files; there's no getting a live PCM stream across the +bridge, nor would you want to). But for **everything that isn't the raw audio** — the +index, the mapping data, which project's bank is active — the bridge opens a **second, +richer seam** the prior draft missed: + +- **File seam (audio + a portable snapshot).** WAVs on disk; index/mapping as JSON or a + sidecar. Tool-agnostic, host-agnostic, portable. Any instrument (JSFX or native, in + REAPER or any other host) can consume it. This is the *only* seam JSFX gets, and the + *fallback/export* seam for native. +- **Live-state seam (native-in-REAPER only).** A native VST instance reads the + `"reasampler"` project ext-state directly via the bridge (`GetProjExtState` / + `EnumProjExtState`, resolved by name over `hostcb`), and knows its own host project via + the context callback. It sees what `persist` last wrote, follows the active project, and + needs no "point me at the right bank folder" wiring — it *asks REAPER* which project it's + in. This is strictly more than the file seam and it's REAPER-coupled by construction. + +The design question below — "is the JSON sufficient, or does playback need mapping the +index doesn't carry" — is unchanged. What the bridge changes is *where that mapping can +live and how the instrument gets it*: for native, the mapping can be **live shared state** +between extension and instrument, not just a file handed across. **What the current index carries** (from `bank_model`'s `Sample`, per CONTEXT.md §Data model): id, display name, relative path, source range, channel count, sample rate, @@ -179,18 +242,29 @@ opinionated part. It also means **the bank index grows only by file-intrinsic fi seam ReaSampler already knows how to add (mirror of how `provenance` was added as a field in M1 and populated later). -**A note on format:** if the instrument is native (Option A), it reads the bank JSON -directly — trivial. If it's JSFX (Option B), reading arbitrary JSON is friction; the -seam might need a **simpler sidecar** (a flat `.txt`/key-value map ReaSampler writes -next to the bank, JSFX-parseable) rather than making JSFX parse the index JSON. That's -a concrete cost of the JSFX path and a reason the seam design and the format choice are -coupled. +**A note on format (revised for the bridge):** the friction is asymmetric, and more so +than the prior draft said. +- *Native (Option A):* doesn't just "read the bank JSON off disk" — it can read the map + out of **live project ext-state** via the bridge (whichever of i/ii/iii you choose, the + extension and instrument can share the *same* `"reasampler"` state rather than one + writing a file the other re-parses). The `.wav` audio is still files; the mapping need + not be. Cleanest possible integration. +- *JSFX (Option B):* no bridge — reading arbitrary JSON is friction, so the seam likely + needs a **simpler sidecar** (a flat `.txt`/key-value map ReaSampler writes next to the + bank, JSFX-parseable). On the JSFX path the seam is unavoidably file-based. That's a + concrete cost of JSFX and a reason the seam design and the format choice stay coupled. -**Portability caveat:** the bank is project-relative and travels with the `.rpp` -(M4 machinery). Any instrument consuming it must resolve paths the same way — so the -instrument needs to know the *current* project bank folder. A native plugin instance -on a track can be told its folder (saved in plugin state); a JSFX likewise. This is a -solvable wiring detail but a real one — flag it, don't hand-wave it. +**Portability caveat (partly dissolved on the native path).** The bank is project-relative +and travels with the `.rpp` (M4 machinery). Any *file-seam* consumer must resolve paths +the same way — so a JSFX instrument still needs to be told the current project bank folder +(saved in its own state), a real wiring detail, don't hand-wave it. But a **native** +instance sidesteps this: via the context callback it asks REAPER which project it's in, +then reads that project's bank location straight from ext-state — no "point me at the +folder" step. What was a shared wiring cost is now a **cost only on the JSFX path.** Note +the flip side: a native instrument that resolves its bank *only* through the bridge won't +work outside REAPER at all — so if a portable/exportable instrument is ever a goal, keep +the file seam as the source of truth and treat the live-state seam as an accelerator, not +the sole path. --- @@ -245,13 +319,29 @@ None of these are pre-decided here. Each is a genuine fork. **D1 — Instrument format: JSFX vs. native VSTi (VST3/CLAP, likely via JUCE).** The central fork. JSFX = no new binary, no JUCE, cross-platform free, real ceiling on -sophistication and awkward bank-JSON reading. Native = full sophistication and clean -JSON integration, at the cost of a whole second codebase in a second format with its -own build/release/dependency/signing story. *My lean, for discussion only:* if the goal -is Tier 0–1, **prototype in JSFX first** — it proves the seam and the value with near-zero -format commitment, and if it hits a ceiling the seam you designed still serves a later -native build. Reach for native when Tier 2+ is a firm goal, not before. But this is -squarely Daniel's call and depends on how sophisticated he actually wants this. +sophistication, awkward bank-JSON reading, **and no VST-host bridge** (file-seam only). +Native = full sophistication, clean integration, **plus the bridge**: live `"reasampler"` +ext-state, project-awareness, embedded TCP/MCP UI (see below) — at the cost of a whole +second codebase in a second format with its own build/release/dependency/signing story, +**and REAPER-coupling if it leans on the bridge.** + +*What the bridge does to the native column (the honest update).* The prior draft's native +pitch was "more DSP power + parses JSON cleanly." The bridge adds a category the prior +draft didn't weigh: native can be **part of ReaSampler's live state model**, not a +file-coupled consumer. That is a real pull toward native beyond raw DSP — it's the +difference between "an instrument that reads a ReaSampler export" and "an instrument that +*is* ReaSampler's playback surface." It doesn't make JSFX wrong; it makes native's ceiling +higher and its integration tighter than I credited. Weigh it against the coupling cost: a +bridge-dependent VST is a **REAPER-only** artifact, no longer a portable VST. + +*My lean, updated but unchanged in direction, for discussion only:* if the goal is +Tier 0–1, **still prototype in JSFX first** — it proves the *sampler value and the +key/velocity seam* with near-zero format commitment, and the file seam you design there is +exactly the portable/export seam a native build would keep anyway. The bridge is an +*integration* advantage, not a *does-the-sampler-work* advantage, so it doesn't change what +the cheapest proof is. Reach for native when either (a) Tier 2+ sophistication is a firm +goal, or (b) **tight live integration with ReaSampler's project state is itself the point** +— that second trigger is new, and it's the bridge's real contribution to this fork. **D2 — Whether to take a JUCE (or any external plugin-SDK) dependency at all.** The project today is a clean C++ extension with two vendored submodules and a proud @@ -260,6 +350,17 @@ second build target of a fundamentally different kind. That's a real architectur weight. JSFX sidesteps it entirely. Flagging it as its own decision because "should we depend on JUCE" is a bigger standing commitment than "should we build a sampler." +*The bridge's bearing on D2.* Weigh two coupling costs, not one. A native instrument that +uses the bridge takes on **both** a plugin-SDK dependency (JUCE/VST3/CLAP) *and* a +REAPER-API coupling (the same `reaper_plugin_functions.h` surface the extension binds — so +the instrument would vendor and bind against it too, likely resolving pointers via `hostcb` +rather than the extension's startup path). The upside is that the pure sampler core stays +REAPER-free and testable regardless (D3); it's only the *shell* that touches the bridge, so +the coupling is contained where the project already puts REAPER coupling. The downside is +plain: choosing the bridge means choosing a REAPER-only instrument. If Daniel wants a +sampler that also runs in other hosts, the bridge is off the table and D1's native pitch +loses its integration edge — collapsing back toward "native = more DSP power only." + **D3 — Does the pure-core discipline survive the format boundary?** ReaSampler's identity is *pure REAPER-free testable core + thin shells*. A sampler's **voice engine, envelope math, key/velocity mapping, and repitch logic are exactly the kind of pure, @@ -292,8 +393,50 @@ feature *of* ReaSampler — the way a sample library and a sampler that reads it related-but-distinct products. That framing might keep ReaSampler sharp (it stays the capture tool it is) while letting the instrument evolve on its own clock and format. The alternative reading — it's all one integrated sampler-workstation — is also -coherent. **Which of those two ReaSampler *is* is the highest-order question here, and -it's Daniel's to answer before format/tier decisions mean much.** +coherent. + +*Re-argued with the bridge on the scale (the prior draft weighed this fork without it).* +The prior draft implicitly assumed the two products could only ever share a **file +format** — which makes "companion product" feel clean and natural, because a shared file +is exactly what independent products share. The bridge changes the terms. A native +instrument can be **tightly, live-integrated with ReaSampler's project state**: same +`"reasampler"` ext-state, same project identity, following the active project — a degree +of coupling that is *more than* "two products that happen to read the same files" and +starts to look like "two faces of one tool sharing one state model." That is a genuine +thumb toward the **one-integrated-product** reading. It's not decisive, and I won't +pretend it is: + - It only applies if the instrument is **native** (JSFX can't reach the bridge, so a + JSFX instrument *is* naturally a file-coupled companion — the format choice and this + fork are entangled: pick JSFX and you've half-answered D6 toward "companion"). + - It's an *available* integration, not a *required* one — a native instrument could + still be built to the file seam only, deliberately staying a loose companion for + portability. The bridge widens the option, it doesn't force it. + - The tight-integration reading buys **REAPER-coupling**; the companion reading buys + **portability and independent evolution.** Those are real, opposed goods, and the + bridge doesn't resolve which Daniel wants — it just makes the integrated option + *materially more capable* than the prior draft credited, so the companion reading no + longer wins by default on "well, they can only share a file anyway." + +**Net:** the bridge strengthens "one integrated product" from a coherent-but-equal option +into the one with the stronger technical affordance — *if* native-in-REAPER is the path. +It does not pre-decide the fork. This is still the highest-order question and it's Daniel's +to answer, but he should answer it knowing the integrated option is richer than the +file-sharing picture implied — and knowing that choosing JSFX quietly tilts it the other +way. + +**D7 — Embedded TCP/MCP UI (new option the bridge surfaces; native/CLAP only).** While +verifying the bridge I found a second REAPER-VST-specific affordance the prior draft never +mentioned: a hosted plugin can **draw its own embedded UI directly in the track/mixer +control panel** (`reaper_plugin_fx_embed.h`: VST2 answers `canDo("hasCockosEmbeddedUI")` +with `0xbeef0000` and draws via `effVendorSpecific/effEditDraw`; VST3 implements +`IReaperUIEmbedInterface`; CLAP exposes `cockos.reaper_embedui`). Concretely: a native +ReaSampler instrument could render a compact keymap/level strip *inline in the TCP/MCP*, +not only in its own plugin window — the same surface REAPER's own JS/embedded FX use. This +is **native/CLAP-only** (JSFX can't do it) and is pure polish, not a Tier-0 need — but it's +a real integration affordance that only exists on the native-in-REAPER path, and it +compounds the D1/D6 tilt: if "ReaSampler's playback surface, deeply woven into REAPER's UI" +is the vision, this is a lever only native reaches. Flagging it for completeness, not +recommending it — it's a Tier-3-ish nicety, deferred by default. ---