diff --git a/nodes/src/nodes/accessibility_describe/README.md b/nodes/src/nodes/accessibility_describe/README.md index afc4c41266..ea2d9ae5bf 100644 --- a/nodes/src/nodes/accessibility_describe/README.md +++ b/nodes/src/nodes/accessibility_describe/README.md @@ -1,107 +1,128 @@ # accessibility_describe -A RocketRide image-filter node that turns an image into a structured scene description optimized for blind and visually impaired users. +A RocketRide image node that converts an image into a safety-oriented scene +description for blind and visually impaired users. Choose it when the next step +needs accessible text, including hazards, spatial references, visible text, and +navigation guidance. -## What it does +## About Google Gemini + +Google Gemini is the vision-model provider used by this node through the +`google-genai` Python package. The node sends image bytes and its analysis +prompt to the configured Gemini model in one content-generation request. -Receives an image on its input lane, sends it to **Google Gemini Vision** (via the -`google-genai` SDK), and emits a text description designed for assistive use, for -example real-time narration on smart glasses. The description covers environment type, -hazards with positions, key objects, visible text read verbatim (OCR), people, and -navigation guidance, kept under 150 words by the default prompt. +## What it does -The node buffers the incoming image stream, base64-encodes it as a data URL, and sends -it together with the analysis prompt in a single `generate_content` call. Requests run -with `temperature: 0.3` and `max_output_tokens: 1024`. Transient failures (timeouts, -connection errors, 5xx responses) are retried up to 3 times with exponential backoff. -API errors are translated into user-friendly messages covering authentication, rate -limiting, safety blocks, model unavailability, and timeouts. +The node buffers an incoming image stream until it finishes, then passes the +image and configured analysis prompt to Gemini and writes Gemini's response to +the `text` lane. Its built-in instructions emphasize hazards, spatial +orientation, readable text, concision, and environmental context; the default +analysis prompt requests environment, hazards, objects, text, people, and +navigation in fewer than 150 words. -The default model is `gemini-2.5-flash`. A Google AI API key is required: the node -fails at startup without one, and rejects keys starting with `sk-` (OpenAI keys) with a -clear error. +It is a pipeline filter, not an agent tool. Use it for a complete +accessibility-focused narration instead of an image operation whose downstream +consumer needs another kind of result. An empty image stream produces no text. --- -## Configuration +## Lanes -### Lanes +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `text` | Analyze the completed image stream and emit the generated description. | -| Lane | Direction | Description | -|------|-----------|-------------| -| `image` | input | The image to describe (streamed; any image MIME type) | -| `text` | output | The accessibility-optimized scene description | +## Profiles -### Fields +Default: **Gemini 2.5 Flash - Fast & Efficient (1M tokens)** (`gemini-2.5-flash`). -| Field | Type | Description | -|---|---|---| -| `model` | string | Google Gemini vision model | -| `modelTotalTokens` | number | Maximum context length in tokens | -| `systemPrompt` | string | Default "You are an accessibility-focused scene analyzer designed to help blind and visually impaired users understand their surroundings through image descriptions.". Define the accessibility description behavior and priorities | -| `prompt` | string | Default "Describe this image for a blind person. Include: environment type, hazards with positions, key objects with clock positions, visible text, people, and navigation guidance. Keep under 150 words.". Prompt template for generating accessibility descriptions from images | -| `prioritizeHazards` | string | Default "high". How aggressively to prioritize hazard detection | -| `spatialFormat` | string | Default "clock". How to describe spatial positions | -| `profile` | string | Default "gemini-2.5-flash". Select the Gemini vision model for accessibility descriptions | +| Profile | Model | Context limit | +| --- | --- | --- | +| Gemini 2.5 Flash - Fast & Efficient (1M tokens) **(default)** | `gemini-2.5-flash` | `1048576` tokens | +| Gemini 2.5 Pro - High Quality (1M tokens) | `gemini-2.5-pro` | `1048576` tokens | +| Gemini 2.0 Flash - Balanced (1M tokens) | `gemini-2.0-flash` | `1048576` tokens | -If `accessibility.systemPrompt` or `accessibility.prompt` is left empty, the node falls -back to a generic `systemPrompt` / `prompt` config value, then to its built-in defaults. +## Configuration -### Hazard priority +Start with the default Flash profile, provide its API key, and keep the built-in +instructions and prompt if their safety-first structure fits the task. Adjust +the text fields when the user needs a different description policy, then use +the two formatting controls to make the output easier to act on. The model and +context-limit values are supplied by the selected profile. -| Value | Effect | -|-------|--------| -| `high` (default) | The model must lead with hazards; if none exist it explicitly states the area appears safe | -| `medium` | Hazards are included in their spatial context when present | -| `low` | Standard description order, no extra hazard emphasis | +### System Instructions -### Spatial format +This text is the model's system instruction. By default it directs the model to +lead with hazards, use clock positions and relative distances, read visible text +exactly, favor actionable information, and identify the environment and +landmarks. Change it for a durable policy that should apply to every image—for +example, to add a site-specific safety rule—rather than to ask about one image. -| Value | Effect | -|-------|--------| -| `clock` (default) | Clock positions (12 o'clock = straight ahead) | -| `relative` | Relative directions (left, right, ahead, behind) | -| `both` | Both clock positions and relative directions | +An empty value falls back first to the generic `systemPrompt` runtime setting, +then to the built-in instruction. The selected hazard priority and spatial +format are appended to this instruction, so keep it compatible with those +controls instead of duplicating contradictory spatial or ordering rules. -Both settings are applied as modifiers appended to the system prompt at runtime. +### Analysis Prompt ---- +The analysis prompt is the per-image request. Its default requests six +sections—environment, hazards, key objects, text, people, and navigation—and +asks Gemini to keep the description under 150 words. Change it when the output +has to match a particular consumer or structure; for example, ask for a brief +navigation-only description when a downstream voice interface must respond +quickly. -## Profiles +Leaving it empty falls back to the generic `prompt` runtime setting and then to +the built-in prompt. The implementation sends this text with the image, while +the system instructions remain a separate model setting. -| Profile | Model | Notes | -|---------|-------|-------| -| `gemini-2.5-flash` (default) | `gemini-2.5-flash` | Fast and efficient, suitable for real-time use | -| `gemini-2.5-pro` | `gemini-2.5-pro` | Highest quality | -| `gemini-2.0-flash` | `gemini-2.0-flash` | Balanced | +### Hazard Priority -All profiles use a 1M (1,048,576) token context window. +This control appends a safety-ordering instruction to the system prompt. The +default `high` tells the model to lead with hazards and to state that the area +appears safe when it finds none. Choose `medium` to include hazards in their +spatial context without requiring them first, or `low` for no extra hazard +emphasis. Keep `high` for navigation-oriented descriptions; lower it only when +another description order is more useful. + +### Spatial Format + +This control appends the requested spatial-language style to the system prompt. +`clock`, the default, asks for clock positions with 12 o'clock straight ahead; +`relative` asks for left, right, ahead, and behind; `both` asks for both forms. +Use `relative` for readers unfamiliar with clock positions, or `both` when +redundant orientation is valuable. It works alongside Hazard Priority because +hazards can use whichever spatial convention is selected. --- -## Default output structure +## Authentication -```text -1. ENVIRONMENT - type of place -2. HAZARDS - obstacles, stairs, vehicles (with positions) -3. KEY OBJECTS - notable items with clock positions and distances -4. TEXT - any visible text read verbatim -5. PEOPLE - count, positions, and actions -6. NAVIGATION - clear path forward, turns, or barriers -``` +Set the selected profile's Google AI API key in the node configuration. The +node will not start without a key, and it rejects a value beginning with `sk-` +as an OpenAI key rather than sending it to Gemini. Create the required key in +[Google AI Studio](https://aistudio.google.com/apikey). -Customize the **Analysis Prompt** (`accessibility.prompt`) field to change this structure. +## Notes ---- +### Request and failure behavior -## Authentication +At the end of the image stream, the node base64-encodes the received bytes into +a data URL and sends the decoded bytes to Gemini with temperature `0.3` and a +maximum of `1024` output tokens. It makes one initial request plus up to three +retries for errors whose messages indicate a timeout, connection failure, HTTP +`500`, `502`, `503`, or `504`, or an internal server error; retry waits are 1, +2, and 4 seconds. -Requires a **Google AI API key**: get one at -[aistudio.google.com/apikey](https://aistudio.google.com/apikey) and set it in the -node's API key field. The node validates the key at startup: a missing key raises an -error immediately, and a key with the `sk-` prefix is rejected as an OpenAI key. +If all attempts fail, it raises an accessibility-vision error. Authentication, +rate-limit, invalid-input, unavailable-model, timeout, and safety-block +messages are translated to user-facing error text; other failures retain the +Google AI error message. A missing image or malformed image data URL raises a +value error before the model request. ---- +## Upstream docs + +- [Google Gemini API documentation](https://ai.google.dev/gemini-api/docs) diff --git a/nodes/src/nodes/audio_player/README.md b/nodes/src/nodes/audio_player/README.md index 397bbf4af1..10d68c0ae3 100644 --- a/nodes/src/nodes/audio_player/README.md +++ b/nodes/src/nodes/audio_player/README.md @@ -1,50 +1,46 @@ # audio_player -A RocketRide audio node that plays incoming audio through the system's default output device in real time. +A RocketRide audio sink that plays incoming audio or a video's audio track +through the system output device. ## What it does -Accepts container-format audio bytes (for example WAV or MP3, identified by MIME type) on its `audio` lane and plays them through the machine's default speakers. Video input is also accepted on the `video` lane: the audio track is extracted and played while video frames are discarded. This is a terminal node with no output lanes. +The node passes incoming `audio` and `video` streams to a PCM player and has no +downstream lane. Choose it when a local pipeline needs to play a stream rather +than transform it, transcribe it, or generate speech. It serializes playback +across node instances with one shared lock. -Incoming bytes are piped through **ffmpeg** (the binary bundled with `imageio_ffmpeg`, so no host ffmpeg install is required), which decodes any common container/codec to raw PCM. Playback is handled by **sounddevice** (PortAudio) via a low-latency output stream at 44,100 Hz stereo int16. +## Lanes -Playback is serialized engine-wide: a shared `threading.Lock` created at node startup ensures only one stream plays at a time. Because the node drives a physical audio device, it carries the `nosaas` capability and is not available in SaaS deployments (local only). - ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `audio` | — | Pass the audio stream to the player. | +| `video` | — | Extract and play the video's audio track. | ## Configuration -### Lanes - -| Lane | Direction | Behavior | -|---------|-----------|----------------------------------------------------------------------| -| `audio` | input | Decoded and played through the default output device | -| `video` | input | Audio track extracted and played; video frames are discarded | - -Neither lane produces downstream output. The node is a sink. - -### Fields +This node exposes no configuration fields. It has one empty `default` profile; +playback behavior is fixed by the implementation. -No configuration options: the node exposes no fields and uses a single empty `default` profile. Playback settings are fixed in the implementation: +## Limitations -| Parameter | Value | -|-------------|----------------------------------------------------| -| Sample rate | 44,100 Hz | -| Channels | Stereo (2) | -| Format | PCM int16 (internal playback format after decode) | -| Latency | Low (`latency='low'`, blocksize 1024 frames) | -| Chunk size | 16 KB per queued buffer | -| Queue depth | 32 chunks maximum (~512 KB, roughly 3 s of audio) | +The `nosaas` capability means this node is excluded from SaaS deployments. It +opens an output stream through `sounddevice`, so it is intended for a runtime +that can access an audio output device. ---- +## Notes -## Playback behavior +### Fixed playback format and buffering -Decoded PCM is accumulated into 16 KB chunks and placed into a bounded queue. When the queue holds 32 chunks, upstream writes block, providing natural backpressure that prevents unbounded memory growth. +The player is configured for 44,100 Hz, two-channel, signed 16-bit PCM output +with a low-latency stream and 1,024-frame callback blocks. It accumulates input +into 16 KiB chunks and enqueues at most 32 chunks; a full queue blocks incoming +writes. When stopped, it waits for the queue and callback buffer to drain. -On stop, the node polls until the queue and the internal playback buffer are both fully drained before closing the stream, so buffered audio plays to completion. One known limitation: the final partial frame smaller than the callback blocksize (1024 frames, about 24 ms at 44.1 kHz) is dropped rather than padded with silence. Roughly the last 24 ms of a stream may be cut off. +If playback has ended with less than a complete callback frame buffered, the +callback stops rather than padding silence. This can cut off the final ~24 ms of +audio. ---- diff --git a/nodes/src/nodes/audio_transcribe/README.md b/nodes/src/nodes/audio_transcribe/README.md index 22f69caf78..5ac96a7284 100644 --- a/nodes/src/nodes/audio_transcribe/README.md +++ b/nodes/src/nodes/audio_transcribe/README.md @@ -1,73 +1,81 @@ # audio_transcribe -A RocketRide audio filter node that transcribes spoken audio or video to text using OpenAI Whisper. +A RocketRide audio node that transcribes incoming audio or video streams into +text with the configured Whisper backend. -## What it does - -Receives an audio or video stream, extracts the audio track as 16 kHz mono PCM, buffers it in 60-second chunks (forced flush at 120 seconds), and runs Whisper with built-in voice activity detection (VAD). Segments are merged until they end in terminal punctuation (`.`, `?`, `!`), so output arrives as whole sentences, each carrying the timestamp of its first segment. +## About Whisper -Uses `ai.common.models.Whisper`: transcription routes to the model server when the engine is started with `--modelserver`, otherwise it runs locally via `faster-whisper`. No API key is required either way. Whisper is invoked with `beam_size=5` and `vad_filter=True`, and transcription calls are serialized through a global lock so a single loaded model is shared safely across instances. +Whisper is the transcription backend constructed by this node. The node uses +it to turn buffered PCM audio into timestamped text segments. -Models are downloaded from HuggingFace on first use. GPU is used automatically when available (`compute_type` defaults to `float16`). - ---- +## What it does -## Configuration +The node accepts `audio` or `video`, buffers decoded PCM, transcribes it, and +emits merged sentences on `text`. Pick it when a pipeline needs spoken content +as text, rather than local playback or speech synthesis. When a `documents` +listener is attached, it also writes one document per merged sentence; the +declared lanes remain the `text` outputs below. -### Lanes +## Lanes -| Input lane | Output lane | Behaviour | -|------------|-------------|-----------| -| `audio` | `text` | Transcribed sentences, one per segment | -| `video` | `text` | Audio track is extracted from the video and transcribed | +| Lane in | Lane out | Description | +| --- | --- | --- | +| `audio` | `text` | Transcribe the audio stream into merged text sentences. | +| `video` | `text` | Transcribe audio decoded from the video stream into merged text sentences. | -When a `documents` listener is attached, the node also emits one document per merged sentence with `chunkId` (sequential per stream, reset on each new stream) and `time_stamp` (seconds from stream start) in the document metadata, plus `metadata.source` (the source audio's media detail — `source_mime`, `duration`, `sample_rate`, …) and `metadata.name` = `.segment.txt` when the input carried a stream descriptor. +## Profiles -### Fields +Default: `default`, which loads the `base` model. -| Field | Type | Description | -|---|---|---| -| `model` | string | Default "base". The Whisper model to use for transcription | -| `silence_threshold` | number | Default 0.25. The silence threshold to detect silence in speech (in seconds) | -| `min_seconds` | number | Default 240. The minimum seconds of audio to process in a batch and looking for silence | -| `max_seconds` | number | Default 300. The maximum seconds of audio to buffer to process | -| `vad_level` | number | Default 1. The VAD level to use for silence detection (0-3) | -| `profile` | string | Default "default". | +| Profile | Declared `mode` | Context | +| --- | --- | --- | +| `default` **(default)** | `base` | Default profile settings. | +| `tiny` | `tiny` | Tiny profile settings. | +| `base` | `base` | Base profile settings. | +| `small` | `small` | Small profile settings. | +| `medium` | `medium` | Medium profile settings. | +| `large-v3` | `large-v3` | Large-v3 profile settings. | -### VAD levels +## Configuration -| Level | Behaviour | -|-------|-----------| -| `0` | Most permissive: detects the most audio as speech (risk: includes noise) | -| `1` | Slightly more aggressive: skips minor background noise (default) | -| `2` | Balanced: moderate filtering of non-speech | -| `3` | Most aggressive: filters aggressively, may cut off quiet or short speech | +The implementation buffers audio in 60-second chunks and forces a flush at 120 +seconds; those timings are fixed in its Python source. Select a profile for +the declared configuration preset, then test the active runtime configuration +before relying on any profile-specific behavior. ---- +### Model -## Models +The schema offers `tiny`, `base`, `small`, `medium`, and `large-v3`, with +`base` as its default. The runtime code reads a configuration key named +`model`; the profiles instead declare `mode`. This node source does not show +the mapping between those names, so confirm the loaded model from the startup +debug message before changing a profile for speed or accuracy. -| Model | Notes | -|------------|-------| -| `tiny` | Fastest, least accurate | -| `base` | Fast, low accuracy (default) | -| `small` | Medium speed and accuracy | -| `medium` | Slower, high accuracy | -| `large-v3` | Slowest, highest accuracy | +### Silence Threshold, Minimum Seconds, Maximum Seconds, and VAD Level ---- +These configuration fields describe silence and batching controls, but the +runtime code reads differently named keys: `vad_threshold`, +`vad_min_silence_duration_ms`, and `vad_max_speech_duration_s`. It always +calls Whisper with voice-activity detection enabled, a beam size of 5, and +those runtime values' defaults of 0.5, 500 ms, and 20 seconds respectively. +Because this node source does not show a mapping from the displayed fields to +those runtime keys, validate the observed segmentation before depending on a +panel value to tune pauses or maximum speech length. -## Profiles +## Notes -The node ships one profile per model size (`tiny`, `base`, `small`, `medium`, `large-v3`) plus `default`, which is an alias for `base`. Every profile uses the same defaults: `language: en`, `silence_threshold: 0.25`, `min_seconds: 240`, `max_seconds: 300`, `vad_level: 1`. Only the model differs between profiles. +### Segmentation and documents ---- +The transcriber serializes calls through a shared lock. It combines consecutive +segments until one ends in `.`, `?`, or `!`, then writes that text with the +timestamp of its first segment. For document output, it resets `chunkId` for +each opened stream and attaches source and generated segment-name metadata when +the input begins with a descriptor. -## Language +## Upstream docs -Defaults to English (`en`). Change the `language` config value to transcribe other languages. Any language supported by Whisper is accepted. +- [Whisper repository](https://github.com/openai/whisper) ---- diff --git a/nodes/src/nodes/audio_tts/README.md b/nodes/src/nodes/audio_tts/README.md index a1493f2c7a..e0ca9889b4 100644 --- a/nodes/src/nodes/audio_tts/README.md +++ b/nodes/src/nodes/audio_tts/README.md @@ -1,67 +1,67 @@ # audio_tts -A RocketRide pipe node that converts incoming text into spoken audio using the Kokoro-82M text-to-speech engine. +A RocketRide audio node that turns incoming text, documents, questions, or +answers into a WAV audio stream through Kokoro. -## What it does +## About Kokoro -Takes text arriving on any of its input lanes, synthesizes it with Kokoro-82M (`hexgrad/Kokoro-82M`), and emits the result on the **audio** lane as WAV bytes (MIME `audio/wav`) via the `writeAudio` BEGIN / WRITE / END sequence. Locally generated audio is mono, 16-bit, 24 kHz; synthesis speed is fixed at 1. +Kokoro is the text-to-speech backend used by this node. The node can construct +its pipeline locally or ask a configured model server to load the Kokoro +backend. -The node runs in one of two modes, chosen automatically at startup: +## What it does -- **Local**: when no model server is configured, the node installs its own requirements (`numpy`, `kokoro`, `soundfile`) at runtime and constructs a `kokoro.KPipeline` in-process. The spaCy `en_core_web_sm` model (needed by Kokoro's misaki G2P) is downloaded and installed automatically, matched to the installed spaCy version. -- **Model server (`--modelserver`)**: when a model server address is available, the node connects a `ModelClient` and loads the `kokoro` loader on the server instead. The heavy local dependencies are skipped entirely; audio comes back base64-encoded over the inference command. +The node synthesizes non-empty text and streams a `audio/wav` file on its +`audio` lane. Choose it when a pipeline needs to make text audible rather than +play already-produced media or turn speech into text. It extracts text from +the four supported input lanes, so it can speak data already moving through a +pipeline without an agent-tool interface. -Audio is written to a temporary WAV file during synthesis and deleted as soon as the bytes have been streamed, including on error, so no orphan files are left on disk. Empty or whitespace-only input is silently skipped and produces no output. Startup fails with `Kokoro: choose a voice from the list` if no voice is configured. +## Lanes ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `text` | `audio` | Synthesize the supplied text. | +| `documents` | `audio` | Join eligible document `page_content` values with newlines, then synthesize it. | +| `questions` | `audio` | Join question text with spaces, then synthesize it. | +| `answers` | `audio` | Synthesize the answer text. | ## Configuration -### Lanes - -All four input lanes produce output on the **audio** lane. - -| Input lane | What gets synthesized | -|-------------|-------------------------------------------------------------------------------------------------------------------------| -| `text` | The raw text, as-is. | -| `documents` | The `page_content` of each document, joined with newlines. Documents of type `Image`, `Audio`, or `Video` are skipped. | -| `questions` | The text of every question, joined with spaces. | -| `answers` | The answer text (via `getText()`). | - -### Fields +Choose the single `kokoro` profile and its voice. Most users only need to +choose a voice appropriate for their text; the profile selector does not offer +an alternative backend. -The node has a single profile, **`kokoro`** (the default), selected by the `profile` field. +### Voice -| Field | Type | Description | -|---|---|---| -| `kokoro_voice` | string | Default "af_heart". Kokoro voice. The language is derived automatically from the voice prefix (af_/am_ → American, bf_/bm_ → British, ef_/em_ → Spanish, etc.). | -| `profile` | string | Default "kokoro". | +The voice is required at startup; a blank value raises `Kokoro: choose a voice +from the list`. Its first character is passed to the local or remote Kokoro +pipeline as the language code, so select a voice with the appropriate prefix +for the language you need. The default `af_heart` starts with `a`; change it +when the desired voice or its language-code prefix differs. -### Voices and language +## Notes -The language is derived automatically from the **first character** of the voice id (`af_*` / `am_*` is American English, `bf_*` / `bm_*` is British English, `ef_*` / `em_*` is Spanish, and so on). Available voice families: +### Local and model-server execution -| Prefix | Language | Examples | -|---------------|------------------|---------------------------------------------| -| `af_` / `am_` | American English | `af_heart` (default), `af_bella`, `am_adam` | -| `bf_` / `bm_` | British English | `bf_emma`, `bm_george`, `bm_fable` | -| `jf_` / `jm_` | Japanese | `jf_alpha`, `jm_kumo` | -| `zf_` / `zm_` | Mandarin | `zf_xiaoxiao`, `zm_yunxi` | -| `ef_` / `em_` | Spanish | `ef_dora`, `em_alex` | -| `ff_` | French | `ff_siwis` | -| `hf_` / `hm_` | Hindi | `hf_alpha`, `hm_omega` | -| `if_` / `im_` | Italian | `if_sara`, `im_nicola` | -| `pf_` / `pm_` | Portuguese | `pf_dora`, `pm_alex` | +When a model-server address is configured, the node connects a `ModelClient`, +loads the `kokoro` loader there, and writes the returned base64 WAV data to a +temporary file. Without one, it installs the local requirements, ensures the +`en_core_web_sm` spaCy model is present when spaCy is importable, and creates a +local `KPipeline`. Both paths synthesize at speed 1; the local path writes +mono, 16-bit, 24 kHz WAV output. -The full list of voice ids is defined in `services.json`. +### Input and cleanup behavior ---- +Whitespace-only input produces no audio. Document input excludes documents of +type `Image`, `Audio`, and `Video`; the remaining page content is joined with +newlines. A temporary WAV file is removed after streaming and is also removed +if synthesis raises an exception. -## Troubleshooting (`Exception: 1` / wasabi) +## Upstream docs -If misaki/spaCy initialization fails (for example `Exception: 1` or a missing `wasabi` dependency), ensure the spaCy English model is installed: this node downloads `en_core_web_sm` automatically from the official spaCy GitHub release wheel, matched to the installed spaCy version. Verify that `numpy`, `kokoro`, and `soundfile` from `requirements.txt` are installed, and that the model download was not blocked by network restrictions. +- [Kokoro repository](https://github.com/hexgrad/kokoro) ---- diff --git a/nodes/src/nodes/background_removal/README.md b/nodes/src/nodes/background_removal/README.md new file mode 100644 index 0000000000..c3e184a416 --- /dev/null +++ b/nodes/src/nodes/background_removal/README.md @@ -0,0 +1,69 @@ +# background_removal + +A RocketRide image node that removes an image's background and produces an RGBA cutout plus alpha-coverage statistics. Choose it when a downstream image node needs the foreground isolated for re-compositing. + +## About BiRefNet + +BiRefNet is the image background-removal model family used by this node. The node can load the default BiRefNet model or its higher-resolution HR variant, using the configured model identifier and optional revision. + +## What it does + +The node accepts an image stream, runs background removal for each completed image, and can emit both a cutout image and JSON alpha statistics. The image output preserves the source RGB pixels and combines them with a straight, non-premultiplied alpha matte, so another node can composite it over a new background. Choose this node when you need that RGBA foreground asset rather than a caption, detections, or a depth map. + +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `image` | An RGBA PNG cutout using the input image's RGB data and the predicted alpha matte. | +| `image` | `text` | JSON statistics: `mean_alpha` and `alpha_coverage_pct`. | + +## Profiles + +Default: **BiRefNet — default, 1K (MIT)** (`birefnet-default`). + +| Profile | Model | Maximum input edge | +| --- | --- | --- | +| BiRefNet — default, 1K (MIT) **(default)** | `ZhengPeng7/BiRefNet` | `1024` px | +| BiRefNet HR — 2K, finer hair / edge detail (MIT) | `ZhengPeng7/BiRefNet_HR` | `2048` px | + +## Configuration + +Select the BiRefNet profile that gives the edge detail you need, then tune the maximum input edge only when latency or edge quality calls for it. The node falls back to its implementation default model if no model is configured. + +### Model + +The default profile uses the 1K BiRefNet variant; the HR profile uses the 2K variant for finer hair and edge detail. Start with the default profile and use HR when the foreground boundary needs more detail. A configured model identifier overrides the profile's model value, and the optional revision is passed to the model loader; use an override only when you intend to load a compatible BiRefNet model. + +### Max input edge (px) + +This caps the source image's long edge before inference. The default is `1024`; a lower value reduces inference work and memory use at the cost of a less detailed alpha matte, while a higher value can improve edges. The resulting alpha matte is restored to the original image dimensions before the RGBA cutout is created. The runtime clamps supplied values to `256` through `4096`, so values outside that range do not take effect as entered. + +## Requirements + +With a model server configured, inference runs there. Otherwise it runs locally on CPU, Apple Silicon (MPS), or CUDA, and a device lock serializes local inference so concurrent frames never run through the model at once. + +## Notes + +### Output and failures + +For each successful image, `mean_alpha` is the alpha matte's average after scaling alpha bytes to `0.0`–`1.0`; `alpha_coverage_pct` is the percentage of pixels with alpha greater than `0.5`. The PNG encoder uses compression level 1. If image decoding or inference fails, the node logs a warning, drops that frame, and emits neither output for it. + +## Upstream docs + +- [BiRefNet model page](https://huggingface.co/ZhengPeng7/BiRefNet) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `background_removal.maxEdge` | `number` | **Max input edge (px)**
Downscale source so long edge <= this value before inference; alpha is upsampled back to the original source size for compositing. Lower = faster + less VRAM; higher = sharper edges. | `1024` | +| `background_removal.model` | `string` | **Model**
HuggingFace model identifier for background removal (overrides the profile default) | | +| `background_removal.profile` | `string` | **Model**
BiRefNet variant — default is 1K, HR is 2K for finer edges. | `"birefnet-default"` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/background_removal) + diff --git a/nodes/src/nodes/background_removal/services.json b/nodes/src/nodes/background_removal/services.json index bd33917c6b..fb29c30fba 100644 --- a/nodes/src/nodes/background_removal/services.json +++ b/nodes/src/nodes/background_removal/services.json @@ -58,7 +58,7 @@ "background_removal.maxEdge": { "type": "number", "title": "Max input edge (px)", - "description": "Downscale source so long edge <= this value before inference; alpha is upsampled back to the (capped) source size for compositing. Lower = faster + less VRAM; higher = sharper edges.", + "description": "Downscale source so long edge <= this value before inference; alpha is upsampled back to the original source size for compositing. Lower = faster + less VRAM; higher = sharper edges.", "default": 1024, "minimum": 256, "maximum": 4096 diff --git a/nodes/src/nodes/caption/README.md b/nodes/src/nodes/caption/README.md new file mode 100644 index 0000000000..a1351ca912 --- /dev/null +++ b/nodes/src/nodes/caption/README.md @@ -0,0 +1,54 @@ +# caption + +A RocketRide image node that turns each input image into a natural-language caption. Choose it when downstream text processing needs an image description rather than boxes, masks, or extracted text. + +## About Florence-2 + +Florence-2 is the image captioning model used by this node. The node loads the configured model identifier and optional revision, and sends its configured caption task to the model for each image. + +## What it does + +The node buffers each incoming image stream, captions the completed image, and writes the resulting string on the `text` lane. It offers short, detailed, and more-detailed caption tasks through its configuration. Choose it for a general natural-language description of image content; reach for Object Detection when you need boxes and OCR when you need the text in the image. + +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `text` | Caption string for the completed input image. | + +## Configuration + +The single supplied profile selects the Florence-2 Base model, so most use cases only need a granularity choice. The implementation uses its default model and task if either value is empty. + +### Granularity + +Granularity selects one of three configured tasks: `caption` (the default, shown as Short), `detailed_caption`, or `more_detailed_caption`. Keep the default when a concise description is sufficient; use a more detailed task when the next node needs richer text to reason over. The task is passed directly to the captioning facade, so choose one of the values exposed in the configuration panel. + +## Requirements + +With a model server configured, captioning runs there. Otherwise it runs locally on CPU, Apple Silicon (MPS), or CUDA, and a device lock serializes local caption generation. + +## Notes + +### Empty captions on failure + +The node runs captioning only when the `text` lane has a listener. If image decoding or caption inference raises an exception, it logs a warning and writes an empty string to that listener instead of propagating a result from the failed image. + +## Upstream docs + +- [Florence-2 Base model page](https://huggingface.co/microsoft/Florence-2-base) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `caption.profile` | `string` | **Model** | `"florence-base"` | +| `caption.task` | `string` | **Granularity**
How detailed the caption should be. | `"caption"` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/caption) + diff --git a/nodes/src/nodes/depth_estimate/README.md b/nodes/src/nodes/depth_estimate/README.md new file mode 100644 index 0000000000..86288124c5 --- /dev/null +++ b/nodes/src/nodes/depth_estimate/README.md @@ -0,0 +1,55 @@ +# depth_estimate + +A RocketRide image node that estimates a dense depth map from one image and emits both a colorized map and summary statistics. Choose it when you need per-pixel depth information instead of object bounding boxes or an image caption. + +## About Depth Anything V2 + +Depth Anything V2 is the monocular depth-estimation model used by this node. The node loads the configured model identifier and optional revision, using the supplied V2 Small profile by default. + +## What it does + +The node buffers each input image, estimates depth, and restores the dense result to the input's original dimensions. It can send a JPEG colorized depth map on the `image` lane and the depth array's minimum, maximum, and mean as JSON on the `text` lane. Choose it for a frame-wide depth signal; pair it with Object Detection when you need boxes as well as rough distance context for detected objects. + +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `image` | JPEG colorized depth map, where red is near and blue is far. | +| `image` | `text` | JSON depth statistics: `min`, `max`, and `mean`. | + +## Configuration + +The supplied V2 Small profile is the default model, leaving the maximum input edge as the main performance and detail control. The implementation uses its default model if no model is configured. + +### Max input edge (px) + +This setting limits the input image's long edge before depth inference; its default is `1024`. Lower it when local inference needs less work and memory, accepting less spatial detail in the predicted depth; raise it when sharper depth boundaries matter. The node upsamples the dense result to the original dimensions after inference. At runtime, malformed values use the default and all values are clamped to the range `256`–`4096`. + +## Requirements + +With a model server configured, inference runs there. Otherwise it runs locally on CPU, Apple Silicon (MPS), or CUDA, and a device lock serializes local model use. Output values are relative depth, not a calibrated physical distance. + +## Notes + +### Output and failures + +The colorized map is encoded as JPEG after the depth result is restored to the original image size. The emitted statistics are calculated from that restored depth array. If image decoding or depth inference fails, the node logs a warning, drops the frame, and does not emit image or text output for it. + +## Upstream docs + +- [Depth Anything V2 Small model page](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `depth_estimate.maxEdge` | `number` | **Max input edge (px)**
Downscale input so the long edge <= this value before inference; dense output is upsampled back to original. Lower = faster + less VRAM, higher = sharper depth. | `1024` | +| `depth_estimate.profile` | `string` | **Model** | `"v2-small"` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/depth_estimate) + diff --git a/nodes/src/nodes/detect/README.md b/nodes/src/nodes/detect/README.md new file mode 100644 index 0000000000..425883f9a9 --- /dev/null +++ b/nodes/src/nodes/detect/README.md @@ -0,0 +1,74 @@ +# detect + +A RocketRide image node that detects objects in each input frame and emits labeled bounding boxes plus an annotated image. Choose it when downstream work needs object locations; use Segmentation when it needs pixel masks instead. + +## About RF-DETR and MM-Grounding-DINO + +RF-DETR and MM-Grounding-DINO are the two detection backends exposed by this node. RF-DETR is the default common-object profile, while MM-Grounding-DINO is the prompt-driven open-vocabulary profile; the node supplies the selected backend, threshold, prompt, and optional revision to its detector. + +## What it does + +The node buffers each completed image stream, runs the selected detector, and can emit the detections as JSON and a JPEG annotated with boxes and labels. RF-DETR detects the configured profile's common-object class set; MM-Grounding-DINO lets the prompt define the desired objects or attributes. Choose it as a per-frame gate or whenever location, confidence, and centroid data matter, rather than choosing Caption for a prose description or Segmentation for masks. + +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `image` | JPEG copy of the source image annotated with detection boxes and labels. | +| `image` | `text` | JSON array of detections with `label`, `score`, `box`, and `centroid`. | + +## Profiles + +Default: **RF-DETR — Common Objects (COCO, 80 classes, default)** (`rfdetr`). + +| Profile | Backend | Use | +| --- | --- | --- | +| RF-DETR — Common Objects (COCO, 80 classes, default) **(default)** | `rfdetr` | Closed-set common-object detection. | +| MM-Grounding-DINO — Open-Vocabulary (any prompt) | `mmgdino` | Prompt-driven open-vocabulary detection. | + +## Configuration + +Start with the RF-DETR profile for the provided common-object detector. Switch to MM-Grounding-DINO when the sought object must be expressed in a prompt, then set a threshold to control which matched regions reach downstream nodes. + +### Model + +The RF-DETR profile is the default and supplies the `rfdetr` engine. The MM-Grounding-DINO profile supplies the `mmgdino` engine and makes the detection prompt visible in the configuration panel. If the runtime receives an unrecognized engine value, it logs a warning and uses its default backend instead; use one of the supplied profiles to avoid silently falling back. + +### Detection prompt + +This field is required for the prompt-driven MM-Grounding-DINO backend: startup fails when that backend has no non-empty prompt. It accepts period- or comma-separated terms, such as `person . car . dog`; the runtime splits either separator into the classes it passes to the detector. Use it to name objects or describe attributes such as `red car` or `person in a hat`. The field description explicitly warns that spatial relationships are not supported, so a prompt such as `the car on the left` can return all cars rather than only the left one. + +### Confidence threshold + +The threshold is a `0.0`–`1.0` minimum confidence for including a detection, with `0.3` as the default in both supplied profiles. Raise it when low-confidence boxes are creating noisy downstream work; lower it when missed candidates matter more than precision. If the runtime receives a non-numeric or out-of-range value, it logs a warning and uses its detector default threshold. + +## Requirements + +With a model server configured, detection runs there. Otherwise it runs locally on CPU, Apple Silicon (MPS), or CUDA, and a device lock serializes local inference. + +## Notes + +### Output and failures + +The image output draws lime rectangles and labels containing each detection's label and score. If image decoding, detection, or output generation fails for a frame, the node logs a warning, drops that frame, and emits no image or text output for it. + +## Upstream docs + +- [RT-DETR fallback model page](https://huggingface.co/PekingU/rtdetr_r50vd) +- [Grounding DINO model page](https://huggingface.co/IDEA-Research/grounding-dino-tiny) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `detect.profile` | `string` | **Model** | `"rfdetr"` | +| `detect.prompt` | `string` | **Detection prompt**
Example: "person . car . dog" (period or comma-separated list) or "red car" / "person in a hat" (described object). Returns all matching regions. Matches objects and attributes — not spatial relationships, so "the car on the left" returns all cars, not just the left one. | | +| `detect.threshold` | `number` | **Confidence threshold**
Minimum confidence score (0.0–1.0) to include a detection | `0.3` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/detect) + diff --git a/nodes/src/nodes/detect_segment/README.md b/nodes/src/nodes/detect_segment/README.md new file mode 100644 index 0000000000..49a1b2e3e9 --- /dev/null +++ b/nodes/src/nodes/detect_segment/README.md @@ -0,0 +1,79 @@ +# detect_segment + +A RocketRide image node that produces pixel-level instance or semantic segmentation, with a JSON result and an annotated image. Choose it when a bounding box alone is not enough and a downstream step needs the pixels belonging to each object or class region. + +## About Hugging Face + +This node's service definition identifies its Mask2Former engines as HuggingFace-native. Its configured profiles select pretrained Mask2Former models for object-instance and scene-region segmentation. + +## What it does + +The node consumes each image frame, runs the configured Mask2Former segmentation mode, and can emit both a JPEG overlay and a JSON mask payload. Instance mode returns separate masks, labels, scores, and boxes; semantic mode returns a per-pixel class map and class information. Use it instead of an object-detection node when the shape or extent of an object matters, rather than only its location. + +## Lanes + +| Lane in | Lane out | Description | +| ------- | -------- | ----------- | +| `image` | `image` | Annotated JPEG frame with translucent mask overlays; instance overlays also include boxes and labels. | +| `image` | `text` | JSON segmentation result: an instance list or a semantic-map object, according to the selected mode. | + +## Profiles + +Default: **Mask2Former — Object Instances (COCO, 80 classes, default)** (`mask2former-instance`). + +| Profile | Mode | Default confidence threshold | Default max input edge | +| ------- | ---- | ---------------------------- | ---------------------- | +| Mask2Former — Object Instances (COCO, 80 classes, default) **(default)** | `instance` | `0.3` | `1024` px | +| Mask2Former — Scene Regions (ADE20K, 150 classes) | `semantic` | `0.0` | `1024` px | + +## Configuration + +Start with the profile that matches the kind of result a downstream node needs. The profile supplies a matching mode and engine; adjust the confidence and image-size controls only when the expected objects are missing, noisy, or too expensive to process. + +### Mode and Engine + +`instance` produces a separate mask for each detected object, while `semantic` produces a class map across the image. Keep the engine paired with its mode: the available instance and semantic engines are gated that way in the configuration. Pick instance mode for object-by-object handling, such as extracting individual people; pick semantic mode when classifying every pixel of a scene is more useful. An unrecognised mode falls back to the implementation default at startup. + +### Confidence threshold + +The threshold determines which masks are included, from `0.0` through `1.0`. The instance preset starts at `0.3`; raise it when low-confidence masks clutter the result, or lower it when a relevant object is being omitted. The semantic preset uses `0.0`, so its output retains all class assignments. Invalid threshold values are replaced with the implementation default rather than stopping the node. + +### Max input edge (px) + +Before inference, the node downscales an image whose long edge exceeds this value, then restores masks to the source dimensions for the overlay. The profiles use `1024` px. Lower it to reduce inference time and memory use on large frames; raise it when small boundaries need more detail, within the `256`–`4096` px range. Values outside that range are clamped at startup. + +## Requirements + +This node declares GPU capability and uses its configured transformer engine through a shared device lock when inference runs locally. Its profile metadata states that the engines can run through Transformers on CPU, MPS, or CUDA; use an accelerator when throughput or large images make CPU processing inadequate. The node also loads its declared dependencies before inference, including the mask decoder used to render returned masks. + +## Notes + +### Output and failures + +The text and image outputs are produced only when those lanes have listeners. If image decoding or segmentation raises an error for a frame, the node logs a warning, drops that frame, and suppresses normal image forwarding rather than emitting a partial result. + +## Upstream docs + +- [Hugging Face Transformers documentation](https://huggingface.co/docs/transformers/) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `detect_segment.engine` | `string` | **Engine**
Backing engine. Gated by mode: instance -> mask2former-instance; semantic -> mask2former-semantic. | `"mask2former-instance"` | +| `detect_segment.maxEdge` | `number` | **Max input edge (px)**
Downscale source so long edge <= this value before inference; masks are upsampled back to the source size. Lower = faster + less VRAM. | `1024` | +| `detect_segment.mode` | `string` | **Mode**
Segmentation mode. instance: per-instance masks (default). semantic: per-pixel class map. Both use Mask2Former under the hood. | `"instance"` | +| `detect_segment.profile` | `string` | **Profile**
Segmentation preset. Runs on CPU/MPS/CUDA via transformers. | `"mask2former-instance"` | +| `detect_segment.threshold` | `number` | **Confidence threshold**
Minimum score (0.0-1.0) to include a mask | `0.3` | + +## Dependencies + +- `pycocotools` + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/detect_segment) + diff --git a/nodes/src/nodes/face_detection/README.md b/nodes/src/nodes/face_detection/README.md new file mode 100644 index 0000000000..eb14f39a59 --- /dev/null +++ b/nodes/src/nodes/face_detection/README.md @@ -0,0 +1,66 @@ +# face_detection + +A RocketRide image node that finds faces in each frame with MediaPipe BlazeFace and returns face boxes, scores, and optional alignment keypoints. Choose it as a lightweight face-presence or face-framing stage rather than a general-purpose object detector. + +## About MediaPipe + +MediaPipe Tasks is the local face-detection runtime used by this node. The implementation uses its FaceDetector interface with a cached BlazeFace model artifact. + +## What it does + +The node consumes an image stream, detects faces in each completed frame, and emits a JSON array plus an annotated JPEG. Each detection has the `face` label, a confidence score, a bounding box, and a centroid; the optional landmark setting adds six coarse keypoints. Use a general object-detection node for arbitrary object classes, and use this node when face-specific boxes and alignment points are what the pipeline needs. + +## Lanes + +| Lane in | Lane out | Description | +| ------- | -------- | ----------- | +| `image` | `image` | JPEG copy of the input frame annotated with cyan face boxes, labels, and any emitted landmark dots. | +| `image` | `text` | JSON array of face detections with `label`, `score`, `box`, `centroid`, and optional `landmarks`. | + +## Configuration + +The short-range BlazeFace profile is the available model choice. Tune the threshold to match the acceptable false-positive rate, then enable landmarks only for consumers that need their extra alignment information. + +### Confidence threshold + +The node passes this `0.0`–`1.0` value to MediaPipe as the minimum detection confidence; the default is `0.5`. Raise it when textured backgrounds or image artifacts are incorrectly reported as faces. Lower it when genuine faces are missed, accepting that more weak detections may be returned. The detector downsizes the long edge to 1333 px for inference and rescales its coordinates back to the source image, so the JSON remains in source-image coordinates. + +### Emit 6 alignment keypoints + +When enabled, each detection can include named points for the right eye, left eye, nose tip, mouth center, and both ear tragions. Keep it on for face-aware crops or alignment; turn it off when only boxes and centroids are needed, so downstream consumers do not have to handle landmark data. These are coarse alignment points, not a dense facial-landmark mesh. + +## Notes + +### Local runtime and model setup + +The implementation runs MediaPipe Tasks on CPU/XNNPACK; it does not enable a GPU delegate. On first use it downloads the pinned short-range BlazeFace model to the engine model cache and verifies its SHA-256 checksum before retaining it. On Linux, MediaPipe may need the system libraries `libGLESv2.so.2` and `libEGL.so.1`; missing-library errors identify the needed soname and give a Debian/Ubuntu package hint. + +### Output and failures + +The node writes its text and image results only if the corresponding output lane has a listener. If a frame cannot be decoded or detection fails, it logs a warning, drops that frame, and suppresses default image forwarding rather than emitting a partial result. + +## Upstream docs + +- [MediaPipe Tasks documentation](https://ai.google.dev/edge/mediapipe/solutions/vision/face_detector) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `face_detection.emit_landmarks` | `boolean` | **Emit 6 alignment keypoints**
Include coarse 6-point keypoints per face (eyes, nose, mouth, ear tragions) for face-aware framing and alignment. | `true` | +| `face_detection.profile` | `string` | **Model** | `"short"` | +| `face_detection.threshold` | `number` | **Confidence threshold**
Minimum detection confidence (0.0-1.0). Default 0.5 - higher than object detect to suppress false faces on textured backgrounds. | `0.5` | + +## Dependencies + +- `mediapipe` `>=0.10.35` +- `Pillow` +- `numpy` + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/face_detection) + diff --git a/nodes/src/nodes/frame_grabber/README.md b/nodes/src/nodes/frame_grabber/README.md index a5f3ef0c1f..6eb311df9e 100644 --- a/nodes/src/nodes/frame_grabber/README.md +++ b/nodes/src/nodes/frame_grabber/README.md @@ -1,108 +1,124 @@ # frame_grabber -A RocketRide filter node that extracts still frames from streamed video and passes them downstream as images, documents, or a timing table. +A RocketRide video node that extracts selected still frames from an incoming +video stream and sends them downstream as PNG images, image documents, or a +frame-timestamp table. Choose it when later pipeline stages need individual +frames rather than the original video stream. -## What it does - -Receives video on its `video` input lane and extracts still frames using one of three modes: fixed interval (the default), scene transition detection, or keyframe (I-frame) selection. Each extracted frame is emitted as a PNG image together with its frame number and presentation timestamp. - -Extraction is performed by FFmpeg, spawned as a subprocess via the `imageio-ffmpeg` bundled binary, so no host `ffmpeg` installation is required on the machine running the engine. Video bytes are piped to FFmpeg's stdin, a mode-specific `select` or `fps` video filter picks the frames, and complete PNG images are parsed back from the `image2pipe` output stream. Frame numbers and timestamps are derived from FFmpeg's `showinfo` filter. - -Each output lane is produced only when a downstream node is listening on it, so unused outputs cost nothing at runtime. - ---- - -## Modes +## About Pillow -The active mode is selected by `grabber.profile` (default: `interval`). Timestamps on all output lanes are relative to the start of the video; the configured start time is factored back into FFmpeg's relative output. +Pillow is an image library that frame_grabber imports for optional watermark +rendering. It opens a selected frame, draws the watermark text, and encodes the +result as PNG. -### Interval (default) +## What it does -Extracts frames at a fixed time interval using FFmpeg's `fps` filter. The configured interval in seconds is converted to `fps = 1 / interval` at startup; a zero or negative interval raises an error. +The node receives video on the `video` lane and selects frames in interval, +scene-transition, or keyframe mode. It can produce each selected frame as a +PNG image stream or as an image document; it can also collect frame numbers and +timestamps into a table when the video closes. Each output is generated only +when its corresponding downstream listener is present. Pick this node over a +single-image transform when a pipeline needs a controlled set of frames and +their timestamps from a video. -| Field | services.json key | Type / Default | Description | -|-----------------------------|----------------------------|----------------|-----------------------------------------| -| Interval (seconds) | `grabber.second.interval` | number / `5` | Seconds between extracted frames. | -| Start time (seconds) | `grabber.start_time` | number / `0` | Where to begin extraction (0 = start). | -| Duration (seconds) | `grabber.duration` | number / `0` | How long to extract (0 = full video). | +## Lanes -### Transition +| Lane in | Lane out | Description | +| --- | --- | --- | +| `video` | `image` | Emit each selected frame as an `image/png` stream. | +| `video` | `table` | Emit a Markdown table of selected frame numbers and timestamps when the video closes. | +| `video` | `documents` | Emit each selected frame as an image document with frame metadata. | -Extracts a frame whenever the scene changes by more than a pixel-change threshold, using FFmpeg's `select='gt(scene,)'` filter. +## Profiles -| Field | services.json key | Type / Default | Description | -|-----------------------------|----------------------------|----------------------|-----------------------------------------------------------------------------------------------------| -| Percentage change | `grabber.percent` | number / `0.4` (40%) | Pixel-change threshold that triggers extraction; selectable from 10% to 100% in steps of 10%. | -| Minimum scene gap (seconds) | `grabber.min_scene_gap` | number / `0` | Minimum time between extracted frames; reduces burst detections in high-motion segments. 0 = off. | -| Start time (seconds) | `grabber.start_time` | number / `0` | Where to begin extraction (0 = start). | -| Duration (seconds) | `grabber.duration` | number / `0` | How long to extract (0 = full video). | -| Maximum frames | `grabber.max_frames` | number / `0` | Cap on total frames extracted (0 = unlimited). | +Default: **Extract video frames at intervals** (`interval`). -### Keyframe +| Profile | Selection mode | Context | +| --- | --- | --- | +| `interval` **(default)** | Extract video frames at intervals | Shows the interval, start-time, duration, and watermark settings. | +| `transition` | Extract video frames at scene transitions | Shows the change percentage, minimum scene gap, time bounds, frame cap, and watermark settings. | +| `key` | Extract video frames at keyframes | Shows the time bounds, frame cap, and watermark settings. | -Extracts only video keyframes (I-frames) using FFmpeg's `select='eq(pict_type,I)'` filter. +## Configuration -| Field | services.json key | Type / Default | Description | -|-----------------------------|----------------------------|----------------|------------------------------------------------| -| Start time (seconds) | `grabber.start_time` | number / `0` | Where to begin extraction (0 = start). | -| Duration (seconds) | `grabber.duration` | number / `0` | How long to extract (0 = full video). | -| Maximum frames | `grabber.max_frames` | number / `0` | Cap on total frames extracted (0 = unlimited). | +Start by selecting the frame-grabber mode. The default interval profile is the +right starting point for regular sampling; use the other profiles only when +their selection criteria match the video. The generated schema below lists the +available fields and defaults. ---- +### Frame grabber mode -## Configuration +**Frame grabber mode** selects `interval`, `transition`, or `key` and changes +which settings the configuration panel exposes. Keep the default `interval` +when frames should be sampled at a regular cadence. Choose `transition` when +scene changes should control selection, or `key` when keyframes are the desired +selection points. The transition profile supplies `0.4` as its configured +percentage default. -### Lanes +### Interval between frames -Input lane: `video`. Output lanes: +**Interval (in seconds) between frames** is available in interval mode and +defaults to `5`. At global initialization, the node converts this interval to +frames per second as `1 / interval`; it must therefore be greater than zero, +or startup raises a `ValueError`. Lower it for more frequent sampling and raise +it when fewer frames are sufficient. -| Lane out | Description | -|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------| -| `image` | Each extracted frame streamed as a raw `image/png` payload. | -| `table` | One markdown table per video, written when the video closes: columns `Frame`, `Seconds`, `Time Stamp` (formatted as `HH:MM:SS.ss`). | -| `documents` | One document per frame: `type: "Image"`, base64-encoded PNG as content, with `chunkId` set to the frame number, `time_stamp` (seconds), and a `source` object carrying the originating video's media detail (see Source provenance below) in the metadata. | +### Percentage change for frame and minimum gap -### Fields +In transition mode, **Percentage change for frame** defaults to `0.4` and can +be set from `0.1` to `1.0` in the configuration panel. Use a lower percentage +when smaller changes should select frames; use a higher percentage to require +a larger change. **Minimum gap between scenes** defaults to `0` (disabled) and +can be set from `0.5` to `5` seconds to reduce burst detections in +high-motion segments. Pair a nonzero gap with a low percentage when a busy +video would otherwise produce closely spaced selections. -The `grabber.profile` field selects the active mode and controls which sub-fields are shown in the UI. +### Extraction window and frame cap -| Field | Type | Description | -|---|---|---| -| `percent` | number | Default 0.4. | -| `interval` | number | Default 5. | -| `start_time` | number | Default 0. | -| `duration` | number | Default 0. | -| `max_frames` | number | Default 0. | -| `min_scene_gap` | number | Default 0. Minimum time gap between extracted frames. Helps reduce burst detections in high-motion segments. Set to 0 to disable. | -| `profile` | string | Default "interval". | - -Each profile has a title defined in `preconfig.profiles`: "Extract video frames at intervals" (interval), "Extract video frames at scene transitions" (transition), and "Extract video frames at keyframes" (key). The `transition` profile also sets `percent` to `0.4` as a profile-level default. +**Start time** and **Duration** are available in every profile. Their default +of `0` starts at the beginning and continues to the end of the video, +respectively; set them to limit work to a relevant segment. **Maximum number of +frames** is available in transition and keyframe modes and defaults to `0` for +no limit. Set a cap when downstream storage or review must be bounded. ### Watermark -When `grabber.watermark` is set to `Add watermark`, each extracted frame is stamped with a text label before it is emitted on every output lane. The label is white text with a black outline (legible on any background) at a font size scaled to the frame height. +**Watermark extracted frames** is off by default. When enabled, the node draws +the selected filename and timestamp parts before sending the frame to any +output. Choose a corner with **Watermark position**; keep the default +bottom-right placement unless it covers important content. The filename and +timestamp switches both default to `yes`; turn either off when that part of the +label is unwanted. If both are off, the frame is returned unchanged even when +watermarking is enabled. -| Field | services.json key | Default | Description | -|---|---|---|---| -| Watermark extracted frames | `grabber.watermark` | `no` | Enable the watermark; the fields below appear only when enabled. | -| Watermark position | `grabber.watermark_location` | `bottom_right` | Corner for the label (top/bottom × left/right). | -| Include file name | `grabber.watermark_filename` | `yes` | Add the source file name (from the stream descriptor). | -| Include timestamp | `grabber.watermark_timestamp` | `yes` | Add the frame timestamp (`HH:MM:SS.ss`). | +The timestamp is formatted as `HH:MM:SS.ss`. When the incoming descriptor +identifies media extracted from a container, the filename label is rendered as +`file @ container`; otherwise it uses the available file or container name. +Watermark rendering is best effort: if Pillow raises an exception, the node +logs the error and emits the original frame. + +## Notes -The file name comes from the stream descriptor attached to the video on `BEGIN`; when it is unavailable the label falls back to the timestamp alone. +### Output metadata -### Source provenance +On the documents lane, every frame is emitted as a `Doc` with type `Image`, +base64-encoded PNG content, the frame number as `chunkId`, and its timestamp as +`time_stamp`. The node attaches source-video provenance and gives frames a +derived PNG name when a source descriptor is available. -Every extracted frame carries a backlink to the video it came from. The frame's own metadata already identifies the source object (`objectId`, `parent`); a nested `source` object adds the video's media detail that the frame itself does not have: +On the image lane, each frame is a separate `image/png` stream. Its begin +payload includes the frame's byte size and, when available, its dimensions, +derived name, and nested source provenance. -`source`: `source_mime`, `duration`, `fps`, `width`, `height`, `resource_name`, `container_mime`, `size`, `stream_index`. +### Table output -- On the `documents` lane the `source` object is attached to each frame document's metadata. -- On the `image` lane the same detail (plus `origin: extracted`) is sent as the frame's stream-descriptor enrichment on `BEGIN`, so the raw image stream is self-describing too. +The table lane accumulates a row for every selected frame and writes one +Markdown table only when the video closes and at least one row was collected. +Its columns are `Frame`, `Seconds`, and `Time Stamp`. -`stream_index` distinguishes frames from different videos embedded in one source (e.g. two clips in one presentation). The block is omitted entirely when the incoming video carried no stream descriptor. +## Upstream docs -Each frame document is also given a human-readable `metadata.name` of the form `.frame.png` (e.g. `BBC - Tear down this wall.frame0.png`), where `` is the source video's file name without extension and `N` is the frame index — so frames are individually identifiable downstream. Omitted when the source has no name. +- [Pillow documentation](https://pillow.readthedocs.io/) --- diff --git a/nodes/src/nodes/image_cleanup/README.md b/nodes/src/nodes/image_cleanup/README.md index b02cfac2cc..38babbad13 100644 --- a/nodes/src/nodes/image_cleanup/README.md +++ b/nodes/src/nodes/image_cleanup/README.md @@ -1,31 +1,69 @@ # image_cleanup -A RocketRide image filter node that pre-processes images for OCR, significantly improving recognition accuracy on scanned or low-quality input. +A RocketRide image filter node that prepares incoming images for OCR by +normalizing their format, enhancing contrast, correcting skew, and closing small +gaps in characters. Use it before OCR when scans are noisy, low quality, or +tilted, rather than where downstream processing needs to retain the original +image bytes or color. -## What it does +## About OpenCV -Sits between an image source and a downstream OCR node and rewrites every image that flows through the `image` lane. The incoming bytes are buffered until the full image is received, then transformed through four sequential stages before being forwarded: +OpenCV supplies the image operations used by this node. Here it decodes the +normalized PNG, converts it to grayscale, applies blur, contrast enhancement, +rotation, and morphological closing, then encodes the result as PNG. -1. **PNG normalization**: if the input is not already `image/png`, it is converted via **Pillow** (RGBA mode). PNG inputs pass through this stage untouched. -2. **Grayscale and contrast enhancement**: decodes the PNG with **OpenCV**, converts to grayscale, applies a 5x5 Gaussian blur to suppress noise, then applies CLAHE (clip limit 2.0, 8x8 tile grid) to lift local contrast without blowing out the grayscale range. -3. **Deskew**: inverts the grayscale image to make text white on black, locates all non-zero pixels, fits a minimum-area bounding rectangle to compute the text angle, then rotates the image back to horizontal using cubic interpolation with replicated borders. -4. **Morphological cleanup**: applies morphological closing with a 2x2 square kernel to fill small holes inside character shapes (such as the interior of an `o`) and remove speckle noise. +## What it does -The output is always `image/png` regardless of the input format. When the incoming image carried a stream descriptor, the forwarded image keeps the source-provenance chain (`source`) and inherited `name` on its `BEGIN`, so downstream consumers still know where the image came from. There are no configuration fields and no agent-callable tools; the cleanup pipeline runs automatically on every image. +Receives a streamed image on the `image` lane, buffers it until the stream ends, +then emits a cleaned replacement on the same lane. It is a pipeline-only filter: +there are no agent-callable functions or configuration fields. Pick it for an OCR +path that benefits from a grayscale, contrast-enhanced and deskewed image; place +another image node in the path when preserving color or the input representation +matters. ---- +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `image` | Receives an image stream and emits its cleaned PNG replacement. | ## Configuration -### Lanes +This node has no configuration fields. Add it between the image source and the +OCR consumer; it applies the same cleanup sequence to every complete image +stream. + +## Notes + +### Processing and output + +Inputs whose MIME type is not `image/png` are opened with Pillow, converted to +RGBA, and saved as PNG. The cleanup stages then decode the PNG with OpenCV, +convert it to grayscale, apply a 5×5 Gaussian blur and CLAHE contrast +enhancement (clip limit 2.0; 8×8 tile grid), deskew it, and apply morphological +closing with a 2×2 kernel. The output is therefore a grayscale `image/png`. + +Deskewing inverts the grayscale image, finds its non-zero pixels, and uses their +minimum-area rectangle to choose a rotation angle. If no such pixels are found, +the angle remains zero. Rotation uses cubic interpolation and replicated image +borders, so edge pixels are extended rather than filled with a constant border. + +### Streaming and failures + +The node stores every `WRITE` chunk in memory and performs processing only on +`END`; it suppresses the original stream instead of forwarding it alongside the +replacement. It parses the source descriptor received at `BEGIN` and supplies it +when forwarding the cleaned image. -| Lane in | Lane out | Description | -|---------|----------|-----------------------------------------------------| -| `image` | `image` | Receives the raw image, emits the cleaned PNG | +There is no local fallback to the original input. A non-PNG image that Pillow +cannot convert raises `ValueError`; the cleanup stages also raise when OpenCV +cannot decode or encode their PNG data. Use valid, complete image streams and +account for the full image being buffered before selecting this node for very +large inputs. -Images are streamed in chunks using `AVI_ACTION` signals (`BEGIN` / `WRITE` / `END`). The node buffers all `WRITE` chunks, processes the complete image on `END`, then re-emits it as a new `BEGIN` / `WRITE` / `END` sequence. The default pass-through is suppressed so only the cleaned image continues down the pipeline. +## Upstream docs -No configuration fields. Drop the node into the pipeline between an image source and an OCR node and it works as-is with no further setup. The node registers a single empty `default` profile. +- [OpenCV documentation](https://docs.opencv.org/) --- diff --git a/nodes/src/nodes/pose_estimation/README.md b/nodes/src/nodes/pose_estimation/README.md new file mode 100644 index 0000000000..dd5c009469 --- /dev/null +++ b/nodes/src/nodes/pose_estimation/README.md @@ -0,0 +1,74 @@ +# pose_estimation + +A RocketRide image node that estimates human poses and produces a per-person set of COCO keypoints with an annotated skeleton. Choose it when a pipeline needs body position or joints, rather than only person detections or face locations. + +## About RTMPose + +RTMPose is the human-pose-estimation engine used here through the rtmlib ONNX wrapper. This node first detects people, then uses RTMPose to predict the keypoints for each retained person crop. + +## What it does + +For every input image frame, the node estimates people and emits a JSON array of person boxes with 17 COCO keypoints, as well as an annotated JPEG with skeleton lines and points. The configured threshold controls what is drawn, and the retained-person limit bounds crowded-frame processing. Use it instead of face detection when full human pose matters, and instead of object detection when downstream work needs joint locations. + +## Lanes + +| Lane in | Lane out | Description | +| ------- | -------- | ----------- | +| `image` | `image` | JPEG copy of the frame annotated with person boxes, skeleton edges, and qualifying keypoints. | +| `image` | `text` | JSON array of people, each with a `person` label, a box, and 17 named COCO keypoints containing coordinates and scores. | + +## Profiles + +Default: **RTMPose Medium** (`rtmpose-medium`). + +| Profile | Default keypoint threshold | Default maximum persons | +| ------- | -------------------------- | ----------------------- | +| RTMPose Tiny | `0.3` | `20` | +| RTMPose Medium **(default)** | `0.3` | `20` | +| RTMPose Large | `0.3` | `20` | + +## Configuration + +Select a pose profile first, then use the threshold and person cap to balance useful annotation against work on each frame. The implementation normalizes invalid values at startup, so keeping the configuration within its declared range makes the chosen behavior explicit. + +### Model + +The Tiny, Medium, and Large profiles select their corresponding RTMPose modes; the default is Medium. Use a smaller profile where response time matters more than pose detail, and a larger profile where the available compute justifies it. An unknown profile falls back to the implementation's default mode. + +### Keypoint score threshold + +This `0.0`–`1.0` threshold controls skeleton rendering: keypoints below it are not drawn, and an edge is omitted if either endpoint is below it. The default `0.3` gives a moderately permissive overlay. Raise it to remove uncertain joints from the image output; lower it to show more tentative joints. The text output still receives the estimator's keypoint array, so consumers that make their own confidence decision should use the scores in JSON rather than relying on the drawing. + +### Max persons per frame + +The node keeps at most this many people after sorting by detection score. The default is `20`; reduce it for crowd scenes when latency or memory usage matters, and increase it when every person in a dense image is important. The runtime clamps the value to the `1`–`200` range. + +## Requirements + +This node declares GPU capability. It creates the pose-estimation facade with no fixed device: with a model server configured, inference is delegated there; otherwise it runs locally and serializes device access with a shared lock. Plan accelerator capacity for local high-throughput processing, especially when retaining many people per frame. + +## Notes + +### Output and failures + +The text and image outputs are emitted only when their lanes have listeners. If an image cannot be decoded or inference fails, the node logs a warning, drops the frame, and prevents default forwarding instead of producing a partial pose result. + +## Upstream docs + +- [RTMPose documentation](https://mmpose.readthedocs.io/) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `pose_estimation.max_persons` | `number` | **Max persons per frame**
Cap on persons retained per frame (sorted by detection score). Limits memory + compute on crowd scenes. | `20` | +| `pose_estimation.profile` | `string` | **Model** | `"rtmpose-medium"` | +| `pose_estimation.threshold` | `number` | **Keypoint score threshold**
Minimum per-keypoint confidence (0.0–1.0). Keypoints below this score are skipped when drawing the skeleton. | `0.3` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/pose_estimation) + diff --git a/nodes/src/nodes/thumbnail/README.md b/nodes/src/nodes/thumbnail/README.md index 65fb35a81d..e95862a31a 100644 --- a/nodes/src/nodes/thumbnail/README.md +++ b/nodes/src/nodes/thumbnail/README.md @@ -1,61 +1,68 @@ # thumbnail -A RocketRide filter node that generates a 128×128 PNG thumbnail from an input image. +A RocketRide image filter that turns an existing image or Image document into a +compact 128×128 PNG thumbnail. Choose it when downstream preview, storage, or +vision work needs a fixed-size image rather than the source resolution. -## What it does +## About Pillow -Resizes incoming images into compact, fixed-size thumbnails. It accepts either a raw -image stream or Image documents, and outputs the resized result downstream. Useful for -creating lightweight image previews before passing them to a vision LLM, reducing token -usage and processing time compared to full-resolution images. +Pillow is a Python imaging library. This node uses the shared Pillow-based image +helper to decode input images, resize them, crop a thumbnail, and encode PNG +output. -Uses **Pillow** (via the shared `ai.common.image.ImageProcessor` helper, the node itself -declares no extra requirements). Thumbnails are produced by stepwise downscaling (halving -until the larger side is ≤ 256 px, which avoids aliasing), an aspect-ratio-preserving -resize, and a final **center crop to exactly 128×128 pixels**. Output is always encoded -as PNG, preserving transparency. +## What it does -The node is registered as a `filter` with class type `image` and has no configurable -fields. +The node accepts raw images on the `image` lane and Image documents on the +`documents` lane, then produces thumbnail versions for the lanes declared by the +pipeline. It is a pure transform with no configuration or agent-tool surface. +Use it after an image is already available and you need a compact, consistently +sized derivative; it does not select frames or acquire images from another +source. ---- +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| `image` | `documents` | Emit the thumbnail as an Image document. | +| `image` | `image` | Emit the thumbnail as a raw PNG image stream. | +| `documents` | `documents` | Replace each supported Image document with its thumbnail. | ## Configuration -### Lanes +This node has no configuration fields. Its output size is fixed by the +implementation, so connect the input and only attach the output lane or lanes +your pipeline needs. + +## Notes + +### Thumbnail output -| Lane in | Lane out | Description | -| ----------- | ----------- | ------------------------------------- | -| `image` | `image` | Resize raw image to thumbnail | -| `image` | `documents` | Resize raw image and emit as document | -| `documents` | `documents` | Resize image documents to thumbnail | +The shared image helper works on a copy, progressively reduces very large +images, preserves aspect ratio during its resize step, then center-crops the +result to exactly 128×128 pixels. Both raw-image and document paths encode the +result as PNG; the raw-image output therefore announces `image/png` even if the +input used a different MIME type. -### `image` lane in +### Raw image streams -Raw image data arrives as a chunked stream (`BEGIN` / `WRITE` / `END`). The node buffers -all chunks, loads the complete image once the stream ends, and generates the thumbnail. -It then emits on whichever output lanes have listeners attached: +The `image` lane buffers the input from `BEGIN` through `WRITE` and processes +it at `END`. If `documents` is connected, it emits one base64-encoded `Image` +document with new thumbnail metadata and preserved source provenance. If +`image` is connected, it emits an enriched PNG stream with the thumbnail width +and height. When both lanes have listeners, it emits both forms. -- **`documents` listener**: the thumbnail is base64-encoded (PNG) and emitted as a - single `Doc` of type `Image` with fresh metadata (`chunkId`, `isTable: false`, - `tableId: 0`, `isDeleted: false`), plus a `metadata.source` provenance chain (the input - image's media detail, nesting the input's own `source` underneath, so the chain grows one - hop per transform) and `metadata.name` (inherited from the input, else `.png`) - when the input carried a stream descriptor. -- **`image` listener**: the thumbnail is emitted as raw PNG bytes in a new - `BEGIN` / `WRITE` / `END` stream as `image/png` (the encoded format, regardless of the input - mime); the `BEGIN` carries the same `source` provenance chain and `name` as an enrichment - descriptor, so both lanes stay consistent. +### Image documents -### `documents` lane in +Only documents whose `type` is `Image` and whose `page_content` is non-empty +are processed. Unsupported documents, empty content, and document-level decode +or processing errors produce a warning and are skipped, while the node +continues processing the remaining documents in the batch. The emitted document +keeps a copy of the original metadata and adds an image-source layer for the +new thumbnail. -Each incoming `Doc` of type `Image` has its base64 content decoded, thumbnailed, and -re-emitted as a new `Image` document with the **original metadata preserved**. Documents -are skipped with a warning (not an error) when they are not of type `Image`, have empty -content, or fail to decode/process, the remaining documents in the batch are still -processed. +## Upstream docs -None. The node's shape defines no properties; the thumbnail size (128×128) is fixed. +- [Pillow documentation](https://pillow.readthedocs.io/) --- diff --git a/nodes/src/nodes/twelvelabs/README.md b/nodes/src/nodes/twelvelabs/README.md index a69e361c85..5d8b3f3371 100644 --- a/nodes/src/nodes/twelvelabs/README.md +++ b/nodes/src/nodes/twelvelabs/README.md @@ -1,64 +1,70 @@ # twelvelabs -A RocketRide filter node that sends a video to TwelveLabs along with instructions and returns the generated text response. +A RocketRide video node that sends an incoming video to TwelveLabs and emits +the service's generated text response. -## What it does +## About TwelveLabs -Analyzes video using the TwelveLabs **Pegasus** model (`pegasus1.2`, with both `visual` and `audio` model options) and emits the generated text. For each incoming video stream the node: +TwelveLabs is the video-analysis service that this node contacts. RocketRide +uses it to index a submitted video temporarily and request text generated from +that indexed video. -1. Buffers the video chunks in memory and writes them to a temporary file when the stream ends. -2. Creates a temporary TwelveLabs index named `rocketride-`. -3. Uploads the video as an indexing task and polls every 5 seconds until the task is `ready` (raises an error if the task reports `failed`, or after a **15-minute timeout**). -4. Calls the TwelveLabs analyze endpoint with your configured instructions as the prompt and writes the result to the `text` lane. -5. Deletes the temporary index and the temporary file, even on failure. +## What it does -Uses the official **`twelvelabs`** Python SDK. If no instructions are configured, the prompt defaults to `Describe this video.`. If TwelveLabs returns an empty result, the node outputs the string `No data from TwelveLabs`. +The node receives `video` data, collects the stream, submits the resulting +temporary file for analysis, and writes the returned text to `text` when that +lane has a listener. Pick it when a pipeline needs a hosted analysis response +for a complete video, rather than local playback, transcription, or speech +generation. There is no agent-tool interface. -Supported input formats (by MIME type): MP4, MOV (QuickTime), AVI, WebM, MKV, and MPG. Unrecognized video MIME types are written to a `.mp4` temp file before upload. +## Lanes ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `video` | `text` | Submit the completed video stream and emit its generated text response. | ## Configuration -### Lanes - -| Lane in | Lane out | Description | -| ------- | -------- | ------------------------------------ | -| `video` | `text` | Analyze video and return text output | - -The text result is only emitted when a downstream node is listening on the `text` lane. +Set the API key and add instructions only when the default description prompt +is not enough. The node has one `default` profile, so profile selection itself +does not change its behavior. -### Fields +### Instructions -| Field | Type | Description | -|---|---|---| -| `apikey` | string | TwelveLabs API key. | -| `instructions` | array | Instructions to guide TwelveLabs when analyzing the video. | -| `profile` | string | Default "default". | - -The node ships a single `default` profile containing both fields. - ---- +The instruction strings are joined with newlines and sent as the analysis +prompt. Leave the list empty to use `Describe this video.`. Add instructions +when the response needs to answer a particular question or follow a particular +format; use separate entries for distinct constraints so they remain readable +in the configuration panel. ## Authentication -Provide your TwelveLabs API key in the `apikey` field. The key is loaded once when the pipeline starts and shared by all instances of the node. Get a key from the [TwelveLabs dashboard](https://docs.twelvelabs.io). +Set `apikey` to the API key accepted by the TwelveLabs client. The key and +instructions are loaded once when the pipeline starts and shared by the node's +instances. + +## Notes ---- +### Submission lifecycle and failures -## Operational notes +Every stream is buffered in memory, written to a temporary file when it ends, +and analyzed through a new index named `rocketride-`. The node polls +the indexing task every five seconds, fails if that task reports `failed`, and +raises `TwelveLabs task timed out` after 15 minutes. It attempts to delete both +the temporary index and the local temporary file even when processing fails; +an empty service response is emitted as `No data from TwelveLabs`. -- Each video processed creates and then deletes its own TwelveLabs index, no persistent index is reused, so nothing accumulates in your TwelveLabs account, but indexing happens from scratch on every video. -- Indexing time depends on video length; tasks that take longer than 15 minutes raise `TwelveLabs task timed out`. -- Videos are currently buffered fully in memory before being written to the temp file, so very large videos increase memory usage accordingly. +### Input file suffix ---- +The local temporary-file suffix follows the input MIME type for MP4, QuickTime, +AVI, WebM, Matroska, and MPEG video. Any other MIME type is written with an +`.mp4` suffix. This only determines the temporary filename; choose a MIME type +that accurately describes the input so the upload is not mislabeled. ## Upstream docs - [TwelveLabs documentation](https://docs.twelvelabs.io) ---- diff --git a/nodes/src/nodes/video_composer/README.md b/nodes/src/nodes/video_composer/README.md new file mode 100644 index 0000000000..98ca030af8 --- /dev/null +++ b/nodes/src/nodes/video_composer/README.md @@ -0,0 +1,68 @@ +# video_composer + +A RocketRide video node that collects image frames and encodes them as an MP4 clip with FFmpeg. Place it after an image-producing filter when those frames should become a playable video rather than continue through an image lane. + +## About FFmpeg + +FFmpeg is the encoder this node invokes to turn an image pipe into an MP4 stream. The implementation sends input and receives output through pipes, so it does not create temporary video files. + +## What it does + +The node receives image frames, buffers them until the current object closes, then encodes the sequence as `video/mp4` and sends it down the video lane. It also sends the completed data to the UI in base64-encoded SSE chunks. Use it after image filters or frame-generation steps; use a video-analysis node instead when the goal is to inspect an existing video rather than compose one. + +## Lanes + +| Lane in | Lane out | Description | +| ------- | -------- | ----------- | +| `image` | `video` | MP4 video encoded from the accumulated input frames. | + +## Profiles + +Default: **Standard quality (H.264, CRF 23)** (`standard`). + +| Profile | Codec | Quality (CRF) | Frame rate | +| ------- | ----- | ------------- | ---------- | +| Standard quality (H.264, CRF 23) **(default)** | `libx264` | `23` | `1.0` fps | +| High quality (H.264, CRF 18) | `libx264` | `18` | `1.0` fps | + +## Configuration + +Choose the standard profile for the smaller default output or the high-quality profile when image detail matters more than file size. Then set playback rate and quality for the way the generated clip will be viewed; the configured codec is passed to FFmpeg as the video codec. + +### Output frame rate (fps) + +This value determines both the FFmpeg input framerate and the playback speed of the MP4. The profiles default to `1.0` fps. Set it to match the timing of the upstream frames: raise it for a faster playback sequence, or lower it when each frame should remain on screen longer. Encoding returns no video if the runtime value is not greater than zero and no more than `240` fps. + +### Quality (CRF) + +CRF is the H.264 constant-rate-factor setting, from `0` through `51`; lower values preserve more detail and produce larger files. The standard profile uses `23`, while the high-quality profile uses `18`. Lower it when artifacts in generated or processed frames are unacceptable, and raise it when output size matters more than visual fidelity. Out-of-range values cause encoding to return no video. + +## Notes + +### Frames, formats, and memory + +Frames are buffered in memory until the input object closes, so peak memory grows roughly with frame count times frame size; the implementation notes that typical resolutions below 500 frames are its intended scale. Send one supported image MIME type per sequence: PNG, JPEG, WebP, BMP, and TIFF are mapped to FFmpeg input codecs, while an unknown type is treated as PNG. The encoder chooses that codec from the most recently received frame's MIME type. + +### Encoding failures + +An empty frame sequence produces no video. FFmpeg failures, a missing or non-executable FFmpeg binary, unsupported codecs, and an encode timeout all result in no lane output rather than a partial MP4. Successful output is written in 48 KiB AVI and SSE chunks, followed by a `video_complete` SSE event. + +## Upstream docs + +- [FFmpeg documentation](https://ffmpeg.org/documentation.html) + + + + +## Schema + +| Field | Type | Description | Default | +|---|---|---|---| +| `composer.crf` | `number` | **Quality (CRF)**
Constant Rate Factor for H.264. Lower = better quality, larger file. Range 0-51. | `23` | +| `composer.fps` | `number` | **Output frame rate (fps)**
Playback speed of the output video. Should match the upstream frame rate. | `1` | +| `composer.profile` | `string` | **Output quality**
Video encoding quality preset | `"standard"` | + +## Source + +[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/video_composer) +