New Controller Engine

proposals/2024-05-23_controller_engine_v2.md

@@ -0,0 +1,365 @@
+# Controller Engine V2
+
+* **Owners:**
+  * @Swiftb0y, @acolombier
+
+* **Implementation Status:** `Not implemented`
+
+* **Related Issues and PRs:**
+  * [Create a reactive programming API for
+    controllers](https://github.com/mixxxdj/mixxx/issues/13440)
+  * [QML-based components API for
+    controllers](https://github.com/mixxxdj/mixxx/pull/13459)
+  * [Respect the Midi timestamp when
+    scratching](https://github.com/mixxxdj/mixxx/issues/6951)
+  * [make brake, soft start, and spinback part of the effects
+    system](https://github.com/mixxxdj/mixxx/issues/8867)
+  * [Move the controller screen rendering feature away from
+    `ControllerScriptEngineLegacy`](https://github.com/mixxxdj/mixxx/issues/13203)
+  * [Confusing Midi Sysex
+    handling](https://github.com/mixxxdj/mixxx/issues/12824)
+  * [Ability for controller to share data at
+    runtime](https://github.com/mixxxdj/mixxx/pull/12199)
+  * [Allow Controller Mappings to be located in their own
+    directory](https://github.com/mixxxdj/mixxx/issues/9906)
+  * [support external displays on
+    controllers](https://github.com/mixxxdj/mixxx/issues/8695)
+  * [Handle hot-plugging and graceful recovery of
+    controllers](https://github.com/mixxxdj/mixxx/issues/5614)
+  * [Controller scripts need to be able to
+    cross-communicate](https://github.com/mixxxdj/mixxx/issues/5165)
+
+This document is supposed to serve as an overview for a new controller engine
+for Mixxx. This does not only refer to the runtime "execution engine" of some
+mapping specific code, but almost all aspects from protocol IO, to the metadata
+schema, point-and-click mapping editor, and the scripting API.
+
+## Definitions
+
+* Mapping: a file or set of files which defines everything that’s needed to a
+  controller to communicate with mixxx
+* Controller: Physical self-contained piece of hardware, possibly consisting of
+  multiple sinks and sources
+* Source: a source of data from the controller (midi/HID messages from a
+  particular port/endpoint)
+* Sink: a port/endpoint of the controller that receives data from mixxx (e.g
+  screens, HID output report, write bulk endpoint are example of different sink
+  types)
+* Manifest: Metadata about the controller definition. Execution Engine and
+  Source/sink topology. (equivalent to what we do with the XML in the current
+  legacy engine)
+* Execution Engine: Scripting engine that evaluates the scripting source files
+  of the engine (so essentially what makes the scripting interactive/smart)
+* Module: ES6/QML Module that's part of a mapping, evaluated in an execution
+  engine (its what you expect ;) )
+* Capability: abstract definition of a concept that is not native to mixxx (eg.
+  shift, controlling different decks with the same physical hardware (1/3, 2/4
+  deck switching)). This is needed for “open-ended” cross-mapping communication
+  (avoiding “controller lock-in”).
+* Point-and-click editor: A tool to easily customize a controller mapping by
+  clicking a button on screen and on the hardware surface and those two
+  corresponding to each other.
+
+## Why
+
+The current controller engine suffers from a number of issues primarily
+summarized as a lack of flexibility.
+
+### Pitfalls of the current solution
+
+To summarize the issues linked in the Related Issues and
+PRs section:
+
+ 1. The current controller engine is not able to handle multiple controllers at
+    once. This is important when a single piece of hardware is advertising
+    multiple different control endpoints (eg HID + Bulk for control + screens
+    such as Native Instruments Traktor devices) as well as when multiple pieces
+    of hardware are supposed to present a single unified control surface (such
+    as the modular Behringer CMD-MM1/-PL1/-DC1/-LC1 or modular NI Traktor
+    Kontrol F1/X1/Z1). Being able to bundle multiple different endpoints into
+    the same mapping is important for a good user experience and to allow for
+    more complex mappings.
+ 2. The current controller engine (`ControllerScriptEngineLegacy`) is not able
+    to support modules of any kind. This is important for code organization and
+    reusability. In order to increase mapping code quality and maintainability,
+    it is important to be able to split the mapping code into multiple files and
+    to be able to reuse code between different mappings. This is especially
+    important now that mapping authors are starting to reuse functionality
+    across different hardware devices (see Numark NS6II and Mixtrack variants or
+    similarity between different pioneer controllers)
+ 3. Current mappings modify global state and assume that they are the only
+    mapping running. This needs to be fixed in order to support multiple
+    mappings running at the same time in the same "execution engine".
+ 4. Device hotplug is currently not possible / hard to implement. This is
+    primarily because of the underlying protocol IO layer not willing to support
+    it. This document proposes an alternative approach that lets us more easily
+    switch the underlying library.
+ 5. Allow changing the manifest format. Many people have expressed distaste in
+    the XML manifest format. Careful implementation of the new format should
+    allow experimentation with different formats.
+ 6. Changing out infrastructure like this is virtually impossible. The proposed
+    architecture is designed to be modular and allows for all the required
+    components to be implemented gradually.
+
+## Goals
+
+Goals and use cases for the solution as proposed in [How](#how):
+
+* Allow multiple controllers to be used at the same time.
+* Mappings should be able to be partitioned / modularized within their subfolder.
+* Code reuse across mappings should be easier and encouraged (complex hardware of the same vendor often shares functionality).
+* IO protocols should easily be changeable and not not be tied to the mapping.
+* Allow easy, gradual implementation of the system.
+
+### Audience
+
+This is primarily targeted at developers and power users who are interested in
+the controller engine. End-users only interested in the point-and-click mapping
+editor should not experience any significant changes.
+
+## Non-Goals
+
+* Replace `ControllerScriptEngine` in the short term until we are confident in
+  the new design.
+* make changes to the scratching code as it is orthogonal to this proposal.
+* Make changes to the effects system as it is orthogonal to this proposal.
+
+## How
+
+Explain the full overview of the proposed solution. Some guidelines:
+
+### File Structure
+
+Built-in mappings reside in `res/controllers/` where each mapping is a directory
+containing at least a manifest file named `manifest.xxx` (where `.xxx` is the
+language-specific file extension (eg. `.xml`, `.yml`, `.json`, etc)).
+Additionally, `res/controllers/lib` contains shared modules that can be used by
+multiple mappings (such as componentsJS). Mapping folders may be zipped during
+export (see considerations regarding shared modules) for the users convenience.
+
+### Manifest
+
+The manifest is a declarative file that describes the mapping. It contains the
+following information:
+
+* Sources and Sinks
+* Execution Engines
+  * Used Mixxx APIs
+* Capabilities
+
+In order to reduce boilerplate markup in the manifest, syntactic shorthands may
+be introduced to cover the most common usecases. The most common usecase may be that
+only one source and sink of a particular protocol coupled to a single execution
+engine and/or protocol dispatcher.
+
+### Sources and Sinks
+
+Sources and Sinks are the endpoints of the controller. They are defined in the
+manifest and are used to define the IO protocol of the controller. They are
+characterized by their protocol (eg. MIDI, HID, Bulk, etc) and their direction (eg.
+input, output, bidirectional). They will also
+contain a heuristic description to map the source/sink to the physical
+controller (protocol specific, in the case of HID, usb vid&pid could be used for
+example).
+
+### Execution Engines
+
+Execution engines is yet another section of the manifest. It defines the
+scripting engine that will be used to evaluate the mapping. It is defined by its
+type (JS, QML, something else?) along with a specific entry point. In the case
+of JS, this would be a ES6 module that exports a controller class to be
+instantiated. In the case of QML, this would be a QML file that defines a
+controller object. Each Execution engine should be run in its own independent
+thread. Data exchange between threads should be non-blocking where possible,
+preferably using Qt Signal/Slots or Non-blocking pipes (since those two are used
+in mixxx already). Scheduling requirements are handled by each engine individually
+(eg refreshrate for a GUI engine or latency requirements of an engine handling IO).
+
+### Capabilities
+
+Capabilities are the concept that allows for cross-mapping communication while
+avoiding controller-lock-in. They are defined by mixxx and are used to define
+concepts that can't be easily mapped to ControlObjects. For example, a
+capability could be `shift` or `deck-switching`. Capabilities are defined in the
+manifest and can be used by the execution engine to communicate with other
+mappings. Capabilities are essentially "opt-in" APIs.
+
+### Point-and-click editor
+
+In order to still support the point and click style use-case, we add yet another
+section that directly connects patterns of midi messages to a control object.
+This section is again attached to a source/sink. There is no support for
+interacting with an execution engine via this section. This essentially works
+like the current mapping editor, but removes the
+`<key>path.to.some.JS.input.handler</key>` feature as it is not compatible with
+the new architecture. This method is favoured over code generation as that is
+not always possible, nor would it likely produce a good result. Moreover, this
+would let us recycle parts of the current codebase. _No need to reinvent the
+wheel._ Execution engines can still be wired up explicitly to access the
+dispatcher here as well. This is to avoid the need to have a separate dispatcher
+defined in the mapping and for easier integration with "hybrid" mappings. the
+API is essentially already implemented as [Registering MIDI Input Handlers From
+Javascript](https://github.com/mixxxdj/mixxx/pull/12781), though QML would need
+a separate declarative API.
+
+### Wiring it all together
+
+Since this flexible layout results in a many-to-many relationship sources/sinks
+and execution engines, we specify a separate section in the manifest that
+defines how sources and sinks are connected to execution engines.
+
+### Sharing mappings with modules
+
+In order to share mappings that access modules outside the mappings root folder,
+the execution engine must be able to create a list of all files accessed and
+export them along the controller files. This would be done by creating a copy of
+all the required files in the root of the mapping during exporting. That tree is
+then overlaid over the built-in libraries. This is currently only possible
+within a `QQmlEngine` using `QQmlEngine::addUrlInterceptor` and
+`QQmlEngine::addImportPath`.
+
+### Mixxx APIs
+
+In order to obtain optimal developer experience of mixxx APIs in the execution
+engine, each execution engine should have its own implementation that maps
+idiomatically to the strengths of the execution engine. This is especially
+important for QML, as it is a reaktive declarative language that does have many
+more features than JS. This is essentially already the case in our current
+codebase, see `*JSProxy` and `*QMLProxy` classes as examples.
+
+### Migration from `ControllerScriptEngineLegacy`
+
+We can gradually transition to the new architecture by modelling the semantics
+of `ControllerScriptEngineLegacy` as a separate execution engine. This would
+allow us to gradually migrate old mappings to the new architecture.
+
+## Current unknowns
+
+How feasible is the heuristic detection of sources and sinks? How do we handle
+multiple sources/sinks of the same type originating from different controllers?
+Eg how do we avoid that the screen from one piece of hardware is connected to
+the wrong controller?
+
+## Alternatives
+
+1. Shoehorn the new architecture into the existing `ControllerScriptEngine`
+   class. This would not eliminate the one-controller-per-mapping limitation and
+   would not allow us to iterate on the design nor on the API.
+
+## Architecture summary
+
+```mermaid
+flowchart LR
+    %% Define front-end web components
+    subgraph io["IO"]
+        co["Control Object"]
+        midiIO["Midi IO"]
+        hidIO["HID IO"]
+        mixxxLib["Mixxx Library"]
+        mixxxUi["Mixxx UI"]
+    end
+    style io fill:transparent,stroke:green,color:#fff
+
+    subgraph layers["Intermediary Layers"]
+        co---midiDispatcher["MIDI Dispatcher"]
+        midiIO---midiDispatcher
+        mixxxUi---uiShift["UI Shift Capability"]
+    end
+    style layers fill:transparent,stroke:yellow,color:#fff
+
+    subgraph proxies["API Proxies"]
+        co---coJSProxy["COJSProxy"]
+        co---coQMLProxy["COQmlProxy"]
+        midiIO---midiJSProxy["MIDI IO JSProxy"]
+        midiIO---midiQMLProxy["MIDI IO QmlProxy"]
+        hidIO---hidJSProxy["HID IO JSProxy"]
+        hidIO---hidQMLProxy["HID IO QMLProxy"]
+        midiDispatcher---midiDispatcherJSProxy["MIDI Dispatcher JSProxy"]
+        mixxxLib---libQmlProxy["Library QmlProxy"]
+        uiShift---CapJSProxy["Capability JSProxy"]
+    end
+    style proxies fill:transparent,stroke:cyan,color:#fff
+
+    subgraph engine["Execution Engine"]
+        coJSProxy---jsEngine["JSExecution Engine"]
+        midiJSProxy---jsEngine
+        hidJSProxy---jsEngine
+        hidQMLProxy---qmlEngine
+        midiDispatcherJSProxy---jsEngine
+        CapJSProxy---jsEngine
+        coJSProxy---jsLegacyEngine["LegacyJS Execution Engine"]
+        hidJSProxy---jsLegacyEngine
+        hidQMLProxy---jsLegacyEngine
+        midiJSProxy---jsLegacyEngine
+        CapJSProxy---jsLegacyEngine
+        coQMLProxy---qmlEngine["QmlExecutionEngine"]
+        midiQMLProxy---qmlEngine
+        libQmlProxy---qmlEngine
+    end
+    style engine fill:transparent,stroke:red,color:#fff
+```
+
+### Manifest Mockup
+
+While the manifest is supposed to be implemented language-independent, the easiest
+implementation would likely be based on XML, as Qt already contains the necessary
+parsing infrastructure. So it makes sense to use XML for mockups. Inline XML comments
+serve as explanation to the reader of this proposal and are not intended to be part
+of real manifests.
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- lets start with the same header are our current mapping, though we distinguish from the old format
+using the schemaVersion -->
+<MixxxControllerPreset mixxxVersion="2.6.0+" schemaVersion="2">
+  <!-- the info section matches the old schema for now -->
+  <info>
+    <name></name>
+    <author></author>
+    <description></description>
+    <manual></manual>
+  </info>
+  <sources>
+    <!-- the id is used to refer to this entry by name in the remaining document-->
+    <midi id="controlInput"/>
+  </sources>
+  <sinks>
+    <midi id="controlOutput"/>
+    <bulk id="screenOutput"/>
+  </sinks>
+  <engines>
+    <Javascript id="js" entry="path-to-main-module.mjs">
+      <!-- the handler is a property on the exported module object that receives all data from the sink -->
+      <sinkRef id="controlInput" handler="incomingData"/>
+      <sourceRef id="controlOutput"/>
+      <!-- if the sources and sinks are not referenced from multiple locations, the sinkref
+      could be replaced directly by the corresponding entry -->
+    </Javascript>
+    <QMLUI id="js">
+      <sinkRef id="screenOutput">
+    </QMLUI>
+  </engines>
+  <dispatchers>
+    <!-- should this midi tag be disambiguated from the ones in other sections? -->
+    <midi source="controlInput" sink="controlOutput">
+      <!-- this would contain the <control> entries from the legacy format -->
+    </midi>
+  </dispatchers>
+  <settings>
+    <!-- same as legacy format -->
+  </settings>
+```
+
+## Action Plan
+
+The following action plan is very abstract because the architecture is designed
+to be implemented gradually. Once we have decided on priorities, we can decide
+on a more concrete plan.
+
+1. Define the abstract Manifest outline.
+2. Implement a concrete manifest parser of the basic outline.
+3. Choose any component from the architecture summary diagram and implement it
+   using tests and making instanstiable by via the manifest.
+4. Once 2 connected components are implemented, implement their connection via
+   the manifest.
+5. Repeat until all components are implemented.