ReaLearn: Add modulation system

[helgoboss]

This forum discussion brought up the topic of modulators. I think it could be a natural fit for ReaLearn.

In a nutshell: First-class modulator objects that can modulate multiple targets, are parameterizable and controllable from outside, can modulate other modulators, visualize current value and upcoming values, come with an overview GUI and can be intuitively connected to target parameters.

Here are the requirements.

Requirements

1. There should be the concept of a modulator

At its core, a modulator just produces different values over time. Mathematically, it's f(t) = y, where t is the current time or beat and y is the produced value, usually a percentage.

Example: Sinus LFO

2. A modulator should be able to modulate parameters

A modulator is only useful if it modulates something: Some kind of parameter.

Examples of parameters: FX parameters, track volume, send volume

3. One modulator should be able to modulate multiple parameters, not just one

It wouldn't be enough if one modulator can modulate just ONE parameter. It should potentially be many.

4. Modulators should be parameterizable

It should be possible to change the behavior of a modulator from outside by modifying one of its parameters.

Mathematically spoken, this just means that the function gets more inputs: f(t, p1, p2, p3, ...) = y

Example: A modulator "Sinus LFO" could have a parameter "Amplitude"

5. Modulator parameters should be changeable via MIDI, OSC and so on

Just providing a GUI for controlling a modulator's parameters is not enough. It should also be possible to change them from controllers or MIDI items.

6. Modulators should be able to modulate parameters of other modulators

As the title says.

7. Modulators should be startable/stoppable/pausable via MIDI, OSC and so on

One could see this as a specialization of requirement 5 (at the end, starting/stopping/pausing could be just parameters).

However, it might make sense to build dedicated support for this, by influencing the input time/beat.

Mathematically spoken:

• restart = set t to 0,
• stopping = set t to 0 and don't increase it anymore
• pausing = don't increase t anymore

8. Modulators should control parameters in real-time

I'm talking about worst-case latency here, i.e., "How many milliseconds may it take at a maximum to apply a new modulator value?""

When I say "real-time", I mean that the modulator should be able to apply the new parameter value at least on the start of each audio block, not later!

For example, with an audio buffer size of 256 samples and a sample rate of 48000 Hz, the worst case latency of applying the new modulator value would then be about 5,33ms.

9. There should be pre-defined modulators

The system should provide multiple useful kinds of modulators out of the box.

10. GUI: Creating and connecting modulators and parameters should be intuitive

• It should be easy to create modulators and connect them to parameters. It's important that this step doesn't interrupt the creative flow.
• It should be possible to see all modulators in one view (at least per track).

11. GUI: Modulators should visualize their current value

Modulators should be visible, visualize their current value (e.g. as an animation) and ideally look pretty. Because humans like pretty things.

12. GUI: Modulators should be visually integrated into existing REAPER views?

I personally don't see this as an important requirement but I feel that some people here do. So it's up to them to describe what exactly they mean by that.

13. Modulations must be rendered (online and offline)

[helgoboss]

Proposal

Backend (logic)

First-class modulators

• Make it possible to add multiple modulators to a ReaLearn instance
• Each modulator is a small EEL2 program that will be executed repeatedly from a real-time thread
• The input of each execution is the current time and beat
• In addition, a modulator can expose parameters that can be modified at any time
• The description of the parameters that it takes could be encoded as YAML frontmatter (similar to ReaLearn Luau presets)
• The actual parameters will be made available as variables to the EEL program
• The YAML frontmatter could also contain visualization hints

Fulfilled requirements: 1, 4

Modulator presets

• There should be a few preset for basic modulators such as Sinus LFO

Fulfilled requirements: 9

Modulator source

• Add a new mapping source type "Modulator"
• Lets you pick one of the previously added modulators within the same instance
• Emits the values generated by that modulator

Fulfilled requirements: 2, 3 (and to some degree 10)

Target "Modulator: Set parameter value"

• Add a new mapping target type with above name
• Lets you pick one of the previously added modulators within the same instance
• Lets you pick one of the exposed parameters of that modulator
• Incoming control values change that parameter by adjusting the variable in the EEL2 program
• The target should also support feedback
• Nice-to-have: The feedback should also occur when changes are made to the parameter values from inside the EEL2 program

Fulfilled requirements: 5, 6

Target "Modulator: Invoke transport action"

• Add a new mapping target type with above name
• Lets you pick among different transport actions:
  • (Re)start: Reset time/beat
  • Stop: Reset time/beat and disable modulator
  • Pause: Disable modulator
• Incoming "on" control values trigger that transport action.

Fulfilled requirements: 7

Real-time FX parameter adjustments

• Make it possible to invoke target "FX parameter: Set value" in real-time context
• Only if the ReaLearn instance is on the same track as the modulated FX
• Nice-to-have: Show a warning if current constellation doesn't support real-time control (and therefore also not rendering)

Fulfilled requirements: 8 and 13 (with the mentioned limitations)

Frontend (GUI)

• Add a new top-level "Modulators" page to the Helgobox App
• "Add" button for adding a new modulator. Prompts you for a pre-defined type or custom (empty) modulator.
• Present all modulators currently added to the instance, always showing their current and upcoming value as a diagram
• Have a sort of plus button next to each modulator which allows learning a new target (in the background, this would just add a ReaLearn mapping to the main unit) or remove it (removes mapping)
• Have an edit button next to each modulator that opens a basic editor for changing the EEL2 code.

Alternatives

Using REAPER's built-in modulation feature

Okay, but currently doesn't trivially support requirements 3, 4, 5, 6, 7, 10. At least not that I know of. And some others maybe not to the same extent as ReaLearn would support them.

Combining FX that generates MIDI with ReaLearn

One could use JSFX that generates MIDI events or something like Melda MCCGenerator and then feed it into ReaLearn. That works already now!

In detail

Let's look at how this works in detail and how it supports our requirements.

MIDI-generating FX

• You add a MIDI-generating FX to the FX chain
• This FX plays the role of a modulator

Fulfilled requirements:

• 1 (partially)
• 4 (JSFX and VSTs can have parameters)
• 7 (partially)
• 9 (the JSFX itself is already a pre-defined modulator, and something like Melda MCCGenerator already has presets, I guess)

ReaLearn mappings for modulating arbitrary FX parameters

• Add a ReaLearn instance below that FX with input set to "FX input"
• Add one or more ReaLearn mappings with the corresponding MIDI source and the desired modulation target

Fulfilled requirements: 2, 3

ReaLearn mappings for changing modulator parameters

• Since the modulator is also just an FX, one can add ReaLearn mappings that changes the modulator FX's parameters

Fulfilled requirements: 5, 6

Assessment / missing things

Requirements that are not fulfilled (at least not in status quo):

• 8 and 13
  • I could add this by making FX parameters controllable in real-time, as in the main proposal. A good first improvement with a deep impact!
• 10
  • This is definitely not given with this solution
  • It's a problem as soon as multiple modulators are involved: MIDI routing horror. One would need to deal with different MIDI channels or busses to route the modulator outputs to the correct ReaLearn source. Totally takes the drive out.
  • A dedicated "Modulator overview GUI" could maybe bring order into this chaos. But that would require that the modulator JSFXs all adhere to a certain standards that allows for equal treatment of each modulator JSFX. Not easy to do.
  • Another alternative would be to have one ReaLearn instance for each LFO, but that's way too hacky.
• 11
  • Only if the JSFX itself supports that.
  • One issue I see here is that JSFX graphics support is very low-level. The GUIs of many existing JSFX look quite mediocre compared to modern polished GUIs.
  • Another issue is the lack of homogeneity: Because of the low-level nature of JSFX graphics, different modulators from different parties would feel inconsistently designed.
  • Potentially, a dedicated "Modulator overview GUI" could at least present the current value in a homogeneous manner. But it couldn't easily foresee upcoming values, which means a "look-ahead" value indication is not possible.

Requirements that are fulfilled only partially or not in the highest-possible quality:

• 1:
  • One limitation is the resolution. There's no type of MIDI message that transmits a floating-point value. 7-bit integers are common. And when being ready to send 2 messages, 14-bit is also doable.
  • 14-bit should be enough for most use cases, so this is not a deal-breaker.
• 7:
  • to some degree at least, by enabling/disabling the modulator FX or using dedicated FX parameters that it maybe provides

[mschnell1]

Re: "First-class modulators":

By "Realime Thread" do you mean the (track-) thread that drives the FX chain the plugin is loaded in, or is the plan to spawn a dedicated thread for that ?

==================================================================================================
Re: EEL programming of modulators:

I suggest to take a look at RePack->ReaRack.

This modular synth has a lot of sophisticated modulators (modules) with nice functionality and GUI. These communicate by midi high resolution CC. And hence the output of a "modulation machine" built from them also is high resolution CC (which in reaper can be used for all kinds of modulations, e.g with the help of ReaLearn if (e.g.) Reaper internal parameters are the target. )

The code might be reused, but IMHO it would be most versatile, best learning curve as well for users as for programmers (and maybe easiest to implement) if a new "Modulation Slot" would be able to host "midi filter" JSFXes and feed the Midi output into the ReaLeran infrastructure.

==================================================================================================
Modulator parameters should be changeable via MIDI, OSC and so on:

... envelopes ...

==================================================================================================
Modulators should control parameters in real-time:

Beware what is meant by "real time".

The most critical issue is the consistent latency between the different triggers of all event involved and the sequence of the rendered samples.

Here PDC and anticipative FX needs to be considered.

And the fact that the samples that come together when rendering are handled by different software agents "somewhere" in a range of sample stream positions (i.e. "time") that is as big as the latency defined by block count, block size (and sample rate when rendering "realtime"). This can be as much as a second when configuring the project for heavy CPU load.

==================================================================================================
There should be pre-defined modulators:

Like selecting plugins from the list of plugins in Reaper. Maybe a "tag:Modulator" could be introduced (similar to tag:instrument) in the EEL code.

==================================================================================================
GUI: Creating and connecting modulators and parameters should be intuitive:

like the cute FX-Chain GUI extension (ReaScript) by Sexan ?

==================================================================================================
I could add this by making FX parameters controllable in real-time, as in the main proposal.

I don't see how this timing-wise could be different from (a) directly using Reaper to convert Midi CC messages to Plugin Parameters and (b) using ReaLearn to convert Midi CC messages to Reaper internal settings/actions.

==================================================================================================
One limitation is the resolution. There's no type of MIDI message that transmits a floating-point value

True !
AFAIR, Midi v2 does provide 32 Bit resolution, which always will be sufficient. (But no support for Midi v2 in Reaper yet.)
OTOH a special kind of "ReaLearn Modulation JSFX" could be defined that outputs (and digests) samples. (That is easy to do in JSFX. I some time ago did a JSFX that created text encoded in samples and same could be read from the rendered file. A much more nasty hack :) :) )

[helgoboss]

@mschnell1 Could you please move your many comments into just one? I'd like to keep this clean, if possible.

[KalleGnOhm]

I am very interested in this :) if i can be at any help please let me know :)