diff --git a/nodes/src/nodes/aparavi_aql/README.md b/nodes/src/nodes/aparavi_aql/README.md index b39b1340a3..b386c58e72 100644 --- a/nodes/src/nodes/aparavi_aql/README.md +++ b/nodes/src/nodes/aparavi_aql/README.md @@ -1,147 +1,114 @@ # aparavi_aql -A RocketRide tool node that lets an AI agent query the Aparavi data governance platform in plain English using AQL (Aparavi Query Language). +A RocketRide tool node that lets an agent retrieve Aparavi STORE metadata by +asking questions in plain English. Pick it when the data is governed by an +Aparavi server rather than stored in one of RocketRide's managed databases. -## What it does - -Translates natural-language questions into AQL SELECT statements, executes them against -the Aparavi REST API, and returns file-metadata rows from the Aparavi **STORE** table. -The agent never needs to know AQL or the schema: it just asks a question, and the node -handles schema knowledge, query generation, and execution internally. - -AQL generation is delegated to a connected LLM node via the node's required **`llm`** -invoke connection. The full STORE column schema is fixed (no dynamic introspection) and -is injected into the LLM prompt together with AQL syntax rules and few-shot examples. - -Execution goes over HTTP using the **requests** library: `POST /server/api/v3/database/query` -on the configured Aparavi server with HTTP Basic Auth, a 30-second timeout, and a default -row limit of 250. Timestamp columns returned in milliseconds are normalized to seconds. - -Safety behavior: every generated query is checked before it touches the network. Only a -single SELECT statement is allowed; multi-statement input is rejected, and the keywords -`INSERT`, `UPDATE`, `DELETE`, `DROP`, `TRUNCATE`, `ALTER`, `CREATE`, `EXEC`, `EXECUTE` -are blocked anywhere in the query. A query that fails the safety check returns an error -immediately. If LLM generation or execution fails, the node makes up to 3 attempts in -total, feeding the failed AQL and the server's error message back to the LLM so it can -correct the query. - -This is a pure tool node: it defines no pipeline lanes and is used only through its -agent-callable tools. - ---- - -## Configuration - - - -| Field | Type | Description | -|---|---|---| -| `url` | string | Default empty. Base URL of the Aparavi server, e.g. https://aparavi.example.com | -| `user` | string | Default empty. Aparavi login username | -| `password` | string | Aparavi login password | -| `db_description` | string | Default empty. What is this data used for? Describe its content and purpose, this helps the LLM generate more accurate AQL queries. | -| `profile` | string | Default "default". | +## About Aparavi +Aparavi is a data governance platform whose server exposes a query API over +metadata in its STORE table. This node uses Aparavi Query Language (AQL), an +SQL-like language, to retrieve those rows without requiring the calling agent +to construct the query itself. +## What it does -The node has a single preconfig profile (`default`). - -### Invoke connections - -| Connection | Min | Description | -|------------|-----|-------------| -| `llm` | 1 | LLM used to generate AQL queries from natural language. | - ---- - -## Available tools - -### get_data - -The primary tool for all Aparavi data retrieval. Takes a natural-language `question` -(required string), generates AQL via the connected LLM, runs the safety check, executes -the query, and returns: - - +The node is a pure tool node: it has no pipeline lanes. Its required LLM turns +a natural-language request into a SELECT-only AQL query over the fixed STORE +schema, then the node sends that query to the configured Aparavi server. Pick +it over rocketride_sql when the authoritative data is Aparavi file metadata; +use the SQL node for the managed relational tenant database. -| Tool | Description | -|---|---|---| -| `get_data` | Translate natural language to AQL, execute against Aparavi, and return rows. | -| `get_aql` | Convert a natural-language question to an AQL SELECT statement without executing it. Use only when the user explicitly asks to see the query. | -| `get_schema` | FALLBACK ONLY: returns the fixed column schema for the Aparavi STORE table. Do NOT call this preemptively; only use if get_data fails or returns unexpected results. | +## Connections +| Connection | Required | Description | +| --- | --- | --- | +| llm | yes | Generates AQL from natural-language questions. | +## As a tool -### get_aql +The functions are registered under the bare names below; this node defines no +configurable server-name prefix. question arguments must be non-empty strings +in a JSON object. -Converts a natural-language `question` (required string) to an AQL SELECT statement -without executing it. Returns `{ "aql": "" }`. Intended only for when the -user explicitly asks to see the query. +| Function | Description | +| --- | --- | +| get_data | Generates safe AQL for required question, executes it, and returns STORE rows. | +| get_aql | Generates AQL for required question without executing it. | +| get_schema | Returns the fixed STORE table schema; use only after retrieval fails or is unexpected. | -### get_schema +get_data returns {rows, aql, count} when successful. If the client was not +initialized, a generated query is unsafe, generation never succeeds, or all +execution retries fail, it returns {error, aql, rows: []}. Invalid tool input +raises ValueError. The node tries generation and execution up to three times; +after an API error, the next LLM attempt receives the failed AQL and error. -Fallback only: returns the fixed column schema for the STORE table as -`{ "store": "STORE", "columns": [{ name, type, description }, ...] }`. Column types are -`STRING`, `NUMBER`, `DATE`, or `OBJECT`. Agents are instructed not to call this -preemptively; it exists for when `get_data` fails or returns unexpected results. +get_aql returns {aql} and does not run the safety check or contact the API; +invalid input or an LLM failure raises. get_schema accepts an empty object and +returns {store: "STORE", columns: [...]}, where each column has a name, type, +and description. ---- +## Configuration -## AQL generation +The single built-in profile has no endpoint defaults. Supply the Aparavi server +connection, then describe the data in terms that help the required LLM select +the correct fixed STORE fields. Query retry count and result limit are fixed in +the implementation rather than configurable fields. -The LLM prompt enforces these rules when generating queries: +### Aparavi Server URL -- `STORE` is the only table, no JOINs. -- Structure: `SELECT cols FROM STORE [WHERE cond] [WHICH CONTAIN 'term'] [GROUP BY col] [HAVING cond] [ORDER BY col ASC|DESC] [LIMIT n]`. -- `LIMIT 250` is added unless the user specifies a different limit. -- Size units are supported in conditions: `10 MB`, `5 GB`, `100 KB`. -- Date functions: `NOW()`, `TODAY()`, `YEAR()`, `MONTH()`, `DAY()`. `NOW()` returns seconds since the Unix epoch; DATE columns are compared in seconds (e.g. last 30 days = `NOW() - (30 * 86400)`). -- Aggregates (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`), string functions (`UPPER`, `LOWER`, `TRIM`, `LENGTH`, `SUBSTR`, `CONCAT`), `CAST`, and `CASE WHEN` are available. -- Column aliases are always double-quoted to avoid reserved-word conflicts, e.g. `COUNT(*) AS "count"`. +Provide the base URL of the Aparavi server, including its scheme, for example +https://aparavi.example.com. The node removes a trailing slash before adding +its database-query path. Change this value when the target Aparavi deployment +changes; an empty or unreachable URL leaves the tool client uninitialized or +causes its HTTP calls to return errors after a 30-second timeout. -The LLM is instructed to output only the raw AQL string; any accidental markdown fences -are stripped from the response before the safety check runs. +### Username and password -Example generations: +Username and Password are sent as HTTP Basic Auth on every Aparavi +database-query request. Set them to credentials accepted by the configured +server. Change both together when switching servers or users; a mismatched +endpoint and credentials becomes an API HTTP or connection error returned by +get_data after its retry loop. -```sql --- "Find all PDF files larger than 10 MB" -SELECT name, parentPath, size, modifyTime FROM STORE WHERE extension = 'pdf' AND size > 10 MB LIMIT 250 +### Data description --- "Count files by extension" -SELECT extension, COUNT(*) AS "count" FROM STORE GROUP BY extension ORDER BY "count" DESC LIMIT 250 +Data description is empty by default and is added as context to the LLM that +writes AQL. Describe the collection's purpose, vocabulary, and useful metadata +conventions when they are not evident from the fixed schema. This can improve +query selection without giving the model a new table: STORE remains the only +table in the generated-query prompt. --- "Files modified in the last 7 days" -SELECT name, parentPath, size, modifyTime FROM STORE WHERE modifyTime > NOW() - (7 * 86400) LIMIT 250 -``` +## Authentication ---- +The node uses HTTP Basic Auth with the configured username and password for +the Aparavi database-query API. It sends POST requests to +/server/api/v3/database/query and does not use a separate token field. -## STORE schema +## Limitations -The schema is hard-coded in `aql_schema.py` (mirroring Aparavi's server column -definitions) and covers roughly 100 columns: identity (`objectId`, `uniqueId`, -`dupKey`), file attributes (`name`, `parentPath`, `extension`, `size`, `mimeType`), -timestamps (`createTime`, `modifyTime`, `accessTime`, all in Unix epoch seconds), -document and email metadata, cost and storage metrics (`storageCost`, `dupCount`, `dri`), -paths, tags, datasets, classifications, ownership and permissions, audit messages, -search and classification hits, and 0/1 status flags (`isContainer`, `isDeleted`, -`isObject`, `isIndexed`, `isClassified`, `isSigned`). +This node runs on the RocketRide engine host and does not support remote +execution. The engine host must be able to reach the configured Aparavi server +using the supplied Basic Auth credentials. It only retrieves from the fixed +STORE schema: the safety check permits one SELECT statement, rejects embedded +multi-statement input, and blocks mutation and execution keywords before +sending AQL to the server. The API request uses a fixed 250-object limit and a +30-second timeout. -Timestamp normalization: values in `createTime`, `modifyTime`, `accessTime`, -`docCreateTime`, `docModifyTime`, `instanceMessageTime`, and `objectMessageTime` larger -than 10^10 are treated as milliseconds and divided by 1000, so results are always in -epoch seconds. +## Notes ---- +### Query generation and result normalization -## Authentication +The LLM prompt supplies the fixed STORE schema, AQL syntax rules, and examples. +It asks for LIMIT 250 unless the question specifies another limit, while the +HTTP client itself requests at most 250 objects. Accidental Markdown fences in +the LLM output are removed. Timestamp values above 10,000,000,000 in the known +Aparavi date fields are treated as milliseconds and normalized to seconds in +returned rows. -The node authenticates to the Aparavi server with HTTP Basic Auth using the configured -`user` and `password` on every API request. Credentials are held in memory for the -lifetime of the pipeline and released when it ends. +## Upstream docs ---- +- [Aparavi Data Suite documentation](https://aparavi.com/docs/data-suite/reports/create-a-new-report/) diff --git a/nodes/src/nodes/autopipe/README.md b/nodes/src/nodes/autopipe/README.md index e80db31acf..9cdc5d5e59 100644 --- a/nodes/src/nodes/autopipe/README.md +++ b/nodes/src/nodes/autopipe/README.md @@ -12,7 +12,7 @@ The node registers as a `filter` with class type `other` and capability `interna --- -## What gets inserted +### Filter assembly | Mode | Filters inserted | | ---- | ---------------- | @@ -44,7 +44,9 @@ Defaults come from the `default` preconfig profile in `services.json`. The prepr --- -## Remote processing +## Notes + +### Remote processing The implementation distinguishes local and remote filter placement, and the default profile carries a `remote` sub-configuration (host, port, apikey, `mode: local`). However, the remote-pipeline assembly path is currently commented out in `IGlobal.endGlobal` and the remote queue is never dispatched: all inserted filters run locally regardless of the `remote` setting. diff --git a/nodes/src/nodes/core/README.md b/nodes/src/nodes/core/README.md index 20abed03a1..3d1820a0ba 100644 --- a/nodes/src/nodes/core/README.md +++ b/nodes/src/nodes/core/README.md @@ -16,7 +16,7 @@ The `hash/` and `parser/` subdirectories carry the per-service documentation pag --- -## Services +### Protocol-bearing services | Service | File | Protocol | Class type | Lanes | |---------|------|----------|------------|-------| @@ -72,7 +72,21 @@ An internal no-op endpoint registered as both a source shape and a target shape --- -## Shared field libraries +## Lanes + +| Lane in | Lane out | Description | +|---------|----------|-------------| +| `_source` | `tags` | Local File System emits source tags for downstream processing. | +| `tags` | `tags` | The Fingerprinter preserves the tags lane while adding its deterministic content fingerprint. | +| `source` | `tags` | The internal null endpoint forwards a source lane into tags without an external system. | + +The parser also accepts `tags` and emits `text`, `table`, `image`, `video`, and `audio`; its protocol-specific documentation is in the `parser/` subdirectory. + +## Configuration + +This directory supplies several built-in services as well as shared field definitions used by other nodes. Configure the protocol-bearing service selected in a pipeline; the generated schema below is the field reference. The shared field files do not register a selectable service themselves. + +### Shared field libraries These files define common fields that are merged into a service definition as required. Field names below are exact. @@ -150,7 +164,7 @@ Combines services into single selectable types for pipelines that pick one provi --- -## Google access helper (`google_access.py`) +### Google access helper (`google_access.py`) A single reader that turns a Google tool node's `access` enum and capability toggles into one resolved object: the OAuth scopes to request, plus the write/destructive gates the node's tool functions check at invoke time. @@ -176,12 +190,20 @@ Bundled specs: --- -## Running the tests +### Running the tests ```bash pytest nodes/test/core/test_google_access.py -v ``` +## Limitations + +The Local File System service reads local paths and is marked for filesystem access, security-sensitive use, non-remote execution, and non-SaaS deployment. Run pipelines that use it where the intended files are locally accessible; it is not available in hosted RocketRide deployments and cannot be moved to a remote execution host. + +## Notes + +The internal Word indexer, ZIP Creation, and null endpoint are protocol-bearing engine services but are not normal user-selectable nodes. The `core` directory also contains reusable JSON field fragments; those fragments are included in the generated schema but do not themselves register pipeline protocols. + --- diff --git a/nodes/src/nodes/guardrails/README.md b/nodes/src/nodes/guardrails/README.md index 26f8d020dc..ad517c1c87 100644 --- a/nodes/src/nodes/guardrails/README.md +++ b/nodes/src/nodes/guardrails/README.md @@ -1,10 +1,10 @@ # guardrails -A RocketRide filter node that screens questions before they reach the LLM and answers before they reach your users. +A RocketRide filter node that checks questions before they reach an LLM and answers before they reach users. Pick it when a pipeline needs configurable local rule checks at its boundary rather than another generation or retrieval step. ## What it does -Sits in the pipeline as a guard filter, evaluating questions on the way in and answers on the way out. On the input side it catches prompt injection, enforces topic rules (blocked and allowed keyword lists), and caps input length or estimated token count. On the output side it checks answers for hallucination (keyword grounding against source documents), flags harmful content, detects PII leaks (emails, phones, SSNs, credit cards, IP addresses), and validates the output format. +Evaluates `questions` before forwarding them, `answers` before forwarding them, and collects `documents` as grounding context for answer checks. Input checks cover prompt-injection patterns, optional topic keywords, and optional size limits; output checks cover configured grounding, content-safety, PII, and format rules. Unlike an LLM moderation or retrieval node, it decides from the text and local configuration, then either forwards or suppresses the original pipeline item. All checks are pure stdlib and regex: the node has no external dependencies, no model calls, and adds no network latency. @@ -14,9 +14,7 @@ Text that is empty or whitespace-only is forwarded without checks. --- -## Configuration - -### Lanes +## Lanes | Lane in | Lane out | Description | |-------------|-------------|-----------------------------------------------------------------------| @@ -26,50 +24,42 @@ Text that is empty or whitespace-only is forwarded without checks. Question text is assembled from both the question objects and any attached context before evaluation. Collected document content resets per pipeline object. -### Fields - -| Field | Type | Description | -|---|---|---| -| `policy_mode` | string | Default "warn". How to handle violations: block (reject), warn (log + continue), log (silent) | -| `enable_prompt_injection` | boolean | Default true. Detect and flag prompt injection attempts in input | -| `enable_content_safety` | boolean | Default true. Detect harmful or unsafe content in output | -| `enable_pii_detection` | boolean | Default true. Detect personal identifiable information (emails, phones, SSNs, credit cards) in output | -| `enable_hallucination_check` | boolean | Default false. Verify that output claims are grounded in source documents | -| `max_input_length` | number | Default 0. Maximum character count for input text (0 = no limit) | -| `max_tokens_estimate` | number | Default 0. Maximum estimated token count for input text (0 = no limit) | -| `expected_format` | string | Default empty. Validate that output matches this format (empty = no check) | -| `blocked_topics` | array | Keywords for topics that should be rejected | -| `allowed_topics` | array | If set, input must contain at least one of these keywords | -| `profile` | string | Default "basic". Guardrails profile | +## Profiles ---- +Default: **Basic: Prompt injection + PII detection** (`basic`). -## Profiles +Start with `basic` to observe violations without interrupting a pipeline, or +`strict` when rejected content must not continue. Choose `custom` when the +checks need to be selected individually. -Three built-in profiles control which fields are exposed in the UI and set sensible starting defaults. +| Profile | Behaviour | +| --- | --- | +| `basic` **(default)** | Prompt injection + PII detection, `warn` mode. Only `policy_mode` is configurable in the UI. | +| `strict` | All checks enabled, `block` on violation, `max_input_length` 50000, `max_tokens_estimate` 4096. Exposes `policy_mode`, `max_tokens_estimate`, and `expected_format`. | +| `custom` | All checks enabled with no size limit, `warn` mode. Exposes every individual check, limit, topic, format, and policy control. | -| Profile | Behaviour | -|--------------------|-------------------------------------------------------------------------------------------------------------| -| Basic *(default)* | Prompt injection + PII detection, `warn` mode. Only `policy_mode` is configurable in the UI. | -| Strict | All checks enabled, `block` on violation, `max_input_length` 50000, `max_tokens_estimate` 4096. Exposes `policy_mode`, `max_tokens_estimate`, and `expected_format`. | -| Custom | All checks enabled, `warn` mode. Every field is configurable individually. | +## Configuration ---- +The generated schema is the field reference. Use the checks below to select what to enforce, then choose the policy mode that determines whether an observed violation should stop the pipeline. -## Input checks +### Input checks Run on the `questions` lane before the question is forwarded: +Enable prompt-injection checking for untrusted questions. Use an allowed-topic list to constrain a focused workflow, or a blocked-topic list for specific unacceptable terms; both are case-insensitive substring checks, not semantic classification. Set one or both size limits when unusually large questions should be stopped before later nodes consume them. + - **Prompt injection** (rule `prompt_injection`, critical severity): regex patterns covering instruction-override attempts ("ignore all previous instructions"), system-prompt extraction, role-play jailbreaks (DAN and similar), delimiter/token injection (`<|system|>`, `[INST]`, etc.), and encoding-evasion commands; plus weighted keyword scoring (keywords such as `jailbreak`, `bypass`, `ignore safety`) that triggers when the combined score reaches 0.7. Topic restriction only runs when `blocked_topics` or `allowed_topics` is non-empty. - **Topic restriction** (rule `topic_restriction`): blocked-keyword matches are high severity; failing to match any allowed keyword is medium severity. Matching is case-insensitive substring. - **Input length** (rule `input_length`, medium severity): only runs when a limit is set (`max_input_length > 0` or `max_tokens_estimate > 0`). Tokens are estimated as word count times 1.3, so treat `max_tokens_estimate` as a rough budget rather than an exact tokenizer count. --- -## Output checks +### Output checks Run on the `answers` lane before the answer is forwarded: +Enable hallucination checking only when relevant documents arrive on the `documents` lane before the answer. It is a lexical grounding test, so use it to flag potentially unsupported output rather than as a factual verifier. Select an expected format only when a downstream consumer requires that shape; an unrecognized format value is skipped. + - **Hallucination** (rule `hallucination`, high severity): sentence-level grounding check. Each output sentence is evaluated for keyword overlap (3+ character non-stop words) against the combined source documents; sentences with less than 30% coverage are flagged. The check is skipped when no documents have been received on the `documents` lane. - **Content safety** (rule `content_safety`, critical severity): regex patterns across three categories: self-harm, violence (weapon and explosive construction), and illegal activity (hacking, theft, counterfeiting). - **PII leak** (rule `pii_leak`, high severity): pattern matches for `email`, `phone_us`, `ssn`, `credit_card`, and `ip_address`. @@ -77,7 +67,7 @@ Run on the `answers` lane before the answer is forwarded: --- -## Policy modes +### Policy modes When any enabled check fails, `policy_mode` decides the outcome: diff --git a/nodes/src/nodes/local_text_output/README.md b/nodes/src/nodes/local_text_output/README.md index 75d17f1c8b..e8911fb12a 100644 --- a/nodes/src/nodes/local_text_output/README.md +++ b/nodes/src/nodes/local_text_output/README.md @@ -1,57 +1,46 @@ # local_text_output -A RocketRide target node that writes pipeline text output to the local filesystem as `.txt` files. +A RocketRide target node that writes pipeline text to files on the machine running the pipeline; choose it when local filesystem output is required instead of an SMB export. ## What it does -Receives text arriving on its `text` lane and saves each source object as a `.txt` file under a configured output directory, preserving the source directory structure. Text is accumulated per object while the object is open and flushed in a single UTF-8 write when the object closes. This is a sink node with no output lane. +The node accumulates text for each object and, when the object closes, writes it as a `.txt` file below the configured output directory. It consumes the `text` lane and produces no output lane. Choose `text_output` instead when the destination is an SMB network share rather than the pipeline host’s filesystem. -Uses only the Python standard library (`os`) with no external dependencies. +## Lanes -The node is restricted to self-hosted deployments (capability `nosaas`). It also carries the `filesystem` and `security` capabilities. +| Lane in | Lane out | Description | +| --- | --- | --- | +| `text` | — | Text content to write to the configured local output directory. | -Safety behavior that applies to every object: - -- Objects that failed upstream (`objectFailed`) are skipped entirely. -- A path-traversal guard rejects any resolved target path that escapes the configured output directory. -- Directory creation failures and write errors log a warning and skip the object rather than failing the whole pipeline. -- Files are only written when the accumulated text is non-empty. -- If `storePath` is not set, every object is skipped with a warning. +## Configuration ---- +Set the output directory, then decide whether a prefix should be removed from each source path before it becomes the destination-relative path. Most users can leave **exclude** at its default of `N/A`. -## Configuration +### Destination path -### Lanes +The node reads `storePath` as the output directory. It resolves this directory and each candidate target path before writing, then rejects a candidate that would resolve outside the output directory. Use an explicit directory the pipeline process is allowed to create and write to; an empty output path causes a warning and the object is skipped. -| Lane in | Description | -| ------- | ----------------------------- | -| `text` | Text content to write to disk | +On Windows, configuration validation rejects `storePath` values containing `< > : " / | ? *`. The node also shortens over-long path components before the write and uses extended-length local paths on Windows. -No output lanes: this is a terminal (target) node. +### Exclude -### Fields +The default `N/A` preserves the complete source path below the output directory. Set a prefix only when that leading source-path portion should not appear in the output hierarchy. If an object path does not begin with the configured prefix, the node logs a warning and skips that object rather than guessing a relative path. -Both fields live under the node's `parameters` key (shown as "Destination path" in the UI). +When a prefix is accepted, the node removes it, replaces the source extension with `.txt`, and creates the necessary directory hierarchy. On Windows, the same invalid-character validation applies to a non-`N/A` exclude value. -| Field | Type | Description | -|---|---|---| -| `exclude` | string | Default "N/A". Which paths to exclude from the output path, if not required put N/A. e.g Users/Downloads/ or N/A | +## Limitations -On Windows, config validation rejects a `storePath` or `exclude` value that contains any of the characters `< > : " / | ? *`. An `exclude` value of `N/A` is exempt from this check. +This node is unavailable in SaaS deployments and writes directly to the pipeline host’s filesystem. Its filesystem and security capabilities mean the configured output location must be intentionally writable by that process; path traversal outside the resolved output directory is rejected. ---- +## Notes -## Output path resolution +### Write behavior -For each object the target file path is constructed from the object's source path: +Objects already marked failed upstream are not written. The node writes only non-empty accumulated text in UTF-8. Directory-creation and write failures are logged as warnings and leave the pipeline running; the per-object state is then reset. -1. If `exclude` is `N/A`, the full source path is used as the relative sub-path. Otherwise the `exclude` prefix is stripped from the source path. If the source path does not start with the `exclude` prefix, the node logs a warning and skips the object. -2. The source file extension is replaced with `.txt` (for example, `Hackathon/folder1/report.pdf` becomes `Hackathon/folder1/report.txt`). -3. The relative path is joined under `storePath` (for example, `/Users/username/Desktop/Hackathon/folder1/report.txt`). The fully resolved path must remain inside `storePath`; any path that escapes it is rejected with a path-traversal error. -4. All necessary subdirectories are created automatically before the file is written. +## Upstream docs ---- +- [Python `os` module](https://docs.python.org/3/library/os.html) diff --git a/nodes/src/nodes/remote/README.md b/nodes/src/nodes/remote/README.md index 165e658369..d7f11b74ca 100644 --- a/nodes/src/nodes/remote/README.md +++ b/nodes/src/nodes/remote/README.md @@ -26,7 +26,7 @@ deployments only, not on RocketRide Cloud. --- -## How it works +### How it works `preparePipeline` (in `client/prepare_pipeline.py`) rewrites the user's simplified pipeline before execution, based on the selected profile: @@ -56,9 +56,19 @@ All HTTP and WebSocket requests carry an `Authorization: Bearer ` header --- +## Profiles + +Default: `local`, which inlines the sub-pipeline without opening a network +connection. + +| Profile | Description | +|---------|-------------| +| `local` **(default)** | Inlines the sub-pipeline locally; no network connection is created. | +| `remote` | Uses the preset `localhost:5565` and `xxx` API key as placeholders; replace them with the remote server details. | + ## Configuration -### Lanes +### Pipeline and lane forwarding `services.client.json` declares no static lanes (`"lanes": {}`); lane wiring is derived from the sub-pipeline by `preparePipeline`. At runtime the client forwards these calls @@ -76,7 +86,7 @@ pipeline tracks the local object lifecycle. Responses from the remote pipeline (`writeText`, `writeDocuments`, etc.) are dispatched back into the local pipeline as they arrive. -### Fields +### Remote endpoint and sub-pipeline Settings live under the node's `remote` configuration block; the sub-pipeline lives under `pipeline`. @@ -96,7 +106,7 @@ or the job's task ID is missing. --- -## Profiles +### Profile presets | Profile | Description | |---------|-------------| @@ -105,7 +115,11 @@ or the job's task ID is missing. --- -## Error handling and limits +## Limitations + +Remote Processing and its internal Remote Server counterpart are marked `nosaas`. They run only on self-hosted RocketRide deployments, because remote execution creates HTTP and WebSocket connections to another RocketRide server and transfers the configured sub-pipeline and its data across that boundary. + +### Error handling and limits - Every forwarded call is acknowledged: after processing, each side sends an `error` lane message containing an `APERR` result. A non-success code is re-raised on the diff --git a/nodes/src/nodes/rerank_cohere/README.md b/nodes/src/nodes/rerank_cohere/README.md index cbd2e7e347..edb9558cfe 100644 --- a/nodes/src/nodes/rerank_cohere/README.md +++ b/nodes/src/nodes/rerank_cohere/README.md @@ -2,6 +2,10 @@ A RocketRide rerank node that reorders retrieved documents by relevance to the query using Cohere's Rerank API. +## About Cohere + +Cohere provides the Rerank API used by this node to score a query against a list of document strings. The node sends the configured model, query, document list, and requested result count to that API, then uses the returned relevance scores to construct RocketRide documents. + ## What it does Takes questions that already carry retrieved documents and reorders those documents by how well they match the query. Cohere scores each document, results are sorted by relevance, cut to the top N, and anything below the minimum score threshold is dropped. Put it downstream of a retrieval or vector-store node so it can rerank the documents attached to each question. @@ -19,9 +23,7 @@ Key behavior to know: --- -## Configuration - -### Lanes +## Lanes | Lane in | Lane out | Description | |-------------|-------------|-----------------------------------------------| @@ -30,28 +32,33 @@ Key behavior to know: The `documents` lane is written only when at least one document survives the `min_score` filter. The `answers` lane is **always** written, so downstream nodes receive a result even when every document was filtered out: the answer text is the surviving documents' content joined by blank lines (empty string if none survived). -### Fields - -| Field | Type | Description | -|---|---|---| -| `model` | string | Default "rerank-english-v3.0". Cohere rerank model name | -| `top_n` | number | Number of top results to return | -| `min_score` | number | Minimum relevance score threshold (0.0-1.0) | -| `profile` | string | Default "rerank-english-v3.0". Rerank model | +## Profiles -`top_n` must be a whole number >= 1 (whole-number floats like `5.0` are accepted; booleans and fractional values are rejected). `min_score` must be a number between 0.0 and 1.0. - -### Profiles - -The **Model** dropdown selects a preconfigured profile: +Default: **Rerank v3.0** (`rerank-english-v3.0`). The **Model** dropdown +selects a preconfigured profile: | Profile | Title | Model | |-----------------------|-------------|---------------------------------| -| `rerank-english-v3.0` | Rerank v3.0 | `rerank-english-v3.0` (default) | +| `rerank-english-v3.0` **(default)** | Rerank v3.0 | `rerank-english-v3.0` | | `rerank-v3.5` | Rerank v3.5 | `rerank-v3.5` | -| `custom` | Custom | free-form `model` field | +| `custom` | Custom | _(free-form model name)_ | + +All preset profiles expose `top_n`, `min_score`, and the API key; `custom` +additionally takes a free-form model name. + +## Configuration + +Select a preset for the supplied model name, or use Custom to enter a model name. Most tuning is the trade-off between how many candidates Cohere considers worth returning and how selective the local score filter should be. + +### Model + +The named profiles provide their declared model values. Use Custom only when the API key is authorized for the name entered; the node rejects an empty or whitespace-only model before it can make a request. + +### Top N and Min Score -All profiles expose `top_n`, `min_score`, and the API key. The Custom profile additionally exposes the `model` field. +`top_n` determines how many results are requested from Cohere before local filtering. It must be an integer of at least 1; whole-number numeric values are accepted, while booleans and fractional values are rejected. Keep the default of 5 for a small candidate set, increase it when later pipeline stages need more alternatives, and lower it to limit the resulting context. + +`min_score` is then applied to those returned results and must be between 0.0 and 1.0. The default 0.0 preserves all returned results. Raise it to remove weak matches, but expect fewer than `top_n` documents because the threshold is applied after Cohere's top-N result set has been returned. --- @@ -63,7 +70,9 @@ The built-in pipeline test cases require the `ROCKETRIDE_RERANK_COHERE_KEY` envi --- -## Error handling +## Notes + +### API failures Cohere API errors are mapped to a custom exception hierarchy whose class names let retry/circuit-breaker heuristics classify them correctly: @@ -75,6 +84,10 @@ Cohere API errors are mapped to a custom exception hierarchy whose class names l | `InternalServerError` | `RerankServerError` | Yes | | Any other exception | `RerankServerError` | Yes | +## Upstream docs + +- [Cohere Rerank API reference](https://docs.cohere.com/reference/rerank) + --- diff --git a/nodes/src/nodes/response/README.md b/nodes/src/nodes/response/README.md index 13b464ad7b..0fe339dfc4 100644 --- a/nodes/src/nodes/response/README.md +++ b/nodes/src/nodes/response/README.md @@ -20,7 +20,7 @@ The node has no Python dependencies of its own (`requirements.txt` is empty); it --- -## Service variants +### Service variants The same implementation is registered as ten services. The generic **HTTP Results** service (`response://`) accepts all nine lane types and lets you map each lane to its own result key. Nine single-lane variants accept exactly one lane each and expose a single `laneName` field: @@ -41,9 +41,7 @@ All variants are `classType: infrastructure` and register as a `filter`. The gen --- -## Configuration - -### Lanes +## Lanes All lanes are inputs; the node produces no output lanes. @@ -59,27 +57,19 @@ All lanes are inputs; the node produces no output lanes. | `video` | - | Captured under the configured key (base64-encoded stream + descriptor `metadata`) | | `image` | - | Captured under the configured key (base64-encoded stream + descriptor `metadata`) | -### HTTP Results (generic service) - -| Field | Type | Description | -|---|---|---| -| `laneId` | string | | -| `laneName` | string | | -| `lanes` | array | Each lane maps pipeline data to a custom JSON key in the response. Select the data type (text, documents, answers, etc.) for Lane Name, and enter a custom JSON key name (1-32 characters) for Result Key. | - -Multiple lane-to-key mappings can be added to return several outputs in a single response. When no mapping is configured for a lane, its data is stored under the lane type name as the default key. +## Configuration -### Single-lane variants (Return Answers, Return Text, ...) +Choose the generic HTTP Results service when one response needs several lane types, and use a single-lane variant when a pipeline has one terminal output. The generated schema contains the field definitions; the guidance below covers the effect of the result-key setting. -| Field | Type / Default | Description | -|------------|-----------------------------------|-------------| -| `laneName` | string, defaults to the lane type | The JSON key under which this lane's data appears in the response body (1-32 characters). | +### Lane name and result key -When `laneName` is set at the top level of the node config (the style used by all single-lane variants), it overrides the `lanes` array and every lane type arriving at the node is written under that one key. The per-lane `lanes` mapping only applies when no top-level `laneName` is configured. +For HTTP Results, each `lanes` entry maps a lane identifier to a result key of one to 32 characters. Use distinct keys when a client needs to distinguish returned types without consulting `result_types`; leave a lane unmapped to use its lane type as the key. In a single-lane service, the top-level `laneName` takes precedence over the `lanes` mapping, so set it only when every arriving result should use the same key. --- -## Response format +## Notes + +### Response format The JSON object returned to the client has the following structure: diff --git a/nodes/src/nodes/rocketride_graph/README.md b/nodes/src/nodes/rocketride_graph/README.md index d8ce1700df..f13577b934 100644 --- a/nodes/src/nodes/rocketride_graph/README.md +++ b/nodes/src/nodes/rocketride_graph/README.md @@ -1,45 +1,116 @@ # rocketride_graph -A RocketRide-managed graph database node backed by PostgreSQL + Apache AGE in your own provisioned RocketRide cloud database — with **zero database setup**. +A RocketRide graph database node for natural-language Cypher queries against +the Apache AGE graph in the signed-in tenant's managed database. Pick it over +rocketride_sql when relationships and traversals are the data model. ## What it does -Mirrors the `graph_neo4j` node: as a pipeline node it takes natural-language questions on the `questions` lane, asks a connected LLM to translate them to Cypher, executes, and emits results; as a tool node agents call `get_data`, `get_schema`, `get_query`, `execute`, and `dialect` (dialect: `age`). +The node accepts questions, asks its required LLM to generate read-only Cypher, +translates it to Apache AGE SQL, and returns rows as a table, text, or answer. +It also exposes graph discovery and query functions to an agent. Use it for +graph labels, relationships, and multi-hop traversal; use rocketride_sql for +relational SQL or rocketride_vector for embedding-backed document retrieval. -Two differences from the generic graph nodes: +## Connections -1. **No connection fields.** The per-tenant DSN is resolved from the account layer (`Account.resolve_db_dsn(client_id)`), keyed by the authenticated connection identity — the same seam as `rocketride_sql` and `rocketride_vector` (one database per tenant backs all three). Requires signing into RocketRide cloud; the open-source build without a cloud identity fails with `RocketRide cloud DB nodes require signing into RocketRide cloud`. -2. **Cypher → AGE translation.** Apache AGE cannot run bare Cypher, so every query path routes through the translation layer at `ai.common.graph.age` (openCypher ANTLR parse → firewall → dialect capability gate → `cypher()` envelope with synthesized column list → prepared-statement parameter binding → agtype decode). Even the raw EXECUTE path translates — only the *semantic* firewall is skipped there, never the resource caps. +| Connection | Required | Description | +| --- | --- | --- | +| llm | yes | Produces Cypher from a natural-language question. | -## Safety model +## Lanes -- **Safe path** (LLM/tool reads): runs in a server-side **READ ONLY transaction** (writes are refused by Postgres itself), plus the layer's semantic firewall (no write clauses, no CALL) and the base's `is_cypher_safe` regex as defence-in-depth. -- **Resource caps** (both paths): query length limit, variable-length traversal depth cap (unbounded `*` patterns are rejected), and a per-transaction `statement_timeout`. -- **EXECUTE** is gated by `allow_execute` (default off); isolation for raw writes is the database-per-tenant boundary. -- All per-query settings are `SET LOCAL` — the cloud endpoint is a transaction-mode pooler, so session-level `SET` would bleed across tenants. AGE is preloaded server-side (no `LOAD`). +| Lane in | Lane out | Description | +| --- | --- | --- | +| questions | table | Returns an executed graph-query result as a Markdown table. | +| questions | text | Returns the graph-query result as text. | +| questions | answers | Returns the graph-query result on the answers lane. | -## Graph provisioning (open) +## As a tool -Ownership of per-tenant `create_graph` is **pending** (cloud provisioner vs node). Until decided, the node fails fast at pipeline start when the configured graph does not exist rather than creating one silently. +The inherited graph functions are registered under the bare names below; this +node defines no configurable server-name prefix. Inputs are JSON objects. -## Configuration - -### Fields +| Function | Description | +| --- | --- | +| get_data | Converts required natural-language question to a safe read-only Cypher query and returns rows; limit is optional. | +| get_schema | Returns the discovered labels, sampled node properties, and relationships. | +| get_query | Converts required natural-language question to read-only Cypher without executing it; limit is optional. | +| execute | Runs required raw Cypher query when direct execution is enabled. | +| dialect | Returns {"dialect": "age"}. | -| Field | Type | Description | -|---|---|---| -| `graph` | string | Default "rocketride". Name of the AGE graph to query | -| `db_description` | string | Default empty. What the graph contains; improves LLM query quality | -| `max_attempts` | integer | Default 5. LLM re-ask ceiling when validation rejects generated Cypher | -| `max_rows` | integer | Default 1000. Row ceiling for the read path | -| `query_timeout_ms` | integer | Default 30000. Per-transaction statement timeout | -| `allow_execute` | boolean | Default false. Enables the raw EXECUTE path | +get_data defaults to the shared read limit, then clamps the requested limit to +the configured Max read rows ceiling. A successful result contains +{valid, rows, query, row_limit, truncated}. Generation, validation, or +execution failure returns {valid: false, error, query, rows: []}; a non-graph +question may instead carry an LLM answer with valid: false. -There are intentionally no `host` / `user` / `password` / `database` fields. +get_query returns {query, valid: true} only after its safe-query checks. +execute bypasses the read-only gate but still passes Cypher through the AGE +translation and resource limits; it raises if direct execution is disabled or +the input is invalid, and otherwise returns {rows, affected_rows}. -### Dialect notes (AGE 1.5.0) +## Configuration -The capability table (see `ai.common.graph.age.capabilities`) rejects constructs the cloud's AGE 1.5.0 cannot run with actionable messages: `datetime()` (store ISO-8601 strings or epoch numbers), `RETURN *` (list columns explicitly), `ORDER BY` on a projection alias (order by the expression), `MERGE ... ON CREATE/MATCH SET` (plain `MERGE` then a separate `SET`), label predicates in `WHERE` (put the label in the `MATCH` pattern), multi-labels like `(n:A:B)` (model the second label as a property or category-node edge), and `shortestPath()` (use a bounded variable-length match). All cells are empirically verified against the exact cloud pin — none pass through unverified. +The single built-in profile supplies the default graph name. RocketRide +provisions a per-tenant database for its managed database nodes, and this node +resolves it from the signed-in RocketRide identity instead of a host, user, +password, or database name you enter. Start with the defaults, then tune the +graph context and read limits around the size and shape of the graph your LLM +must query. + +### Graph name and graph description + +Graph name defaults to rocketride and must name an existing AGE graph; startup +fails instead of creating a missing graph. Graph description is empty by +default and becomes LLM context, so describe labels, relationship meaning, and +domain vocabulary when those are not obvious from reflected schema. Change the +graph name when the tenant database contains multiple AGE graphs; update the +description with it so generated Cypher targets the right model. + +### Validation attempts and read rows + +The node retries failed Cypher validation up to five times by default. Raise +Max validation attempts for a complex schema where the returned validation error +is likely to let the LLM repair its query, or lower it to fail faster. Max read +rows defaults to 1,000 and is an owner-controlled cap for both the questions +lane and get_data; increase it only when agents truly need larger result sets, +since results beyond the cap are intentionally truncated. + +### Query timeout + +Query timeout (ms) defaults to 30,000 and is applied inside each query's +transaction. Reduce it to protect an interactive pipeline from expensive +traversals; increase it only for known queries that legitimately need more +time. It works with the row cap: one limits execution time and the other limits +returned data. + +### Allow direct query execution + +This setting is off by default. Turning it on permits raw Cypher through the +execute tool and QuestionType.EXECUTE; those calls skip LLM translation and the +safe read-only gate. They still use the AGE translator and its resource +controls. Enable it only for trusted callers that require writes or raw Cypher. + +## Limitations + +This node runs on the RocketRide engine host and does not support remote +execution. The engine host must have access to the signed-in RocketRide +identity used to resolve the tenant DSN. The tenant database must already have +Apache AGE installed and contain the configured graph. Read paths are +intentionally read-only; raw execution remains disabled until explicitly +enabled. + +## Notes + +### Translation and schema discovery + +Apache AGE cannot execute bare Cypher. Every query path is translated, and the +safe path runs in a server-side read-only transaction with a semantic firewall, +the base Cypher safety check, and a transaction-local statement timeout. +Schema reflection is best effort: it lists AGE labels, samples node properties, +and samples relationship endpoints; an individual reflection failure warns and +returns partial rather than blocking all schema output. diff --git a/nodes/src/nodes/rocketride_sql/README.md b/nodes/src/nodes/rocketride_sql/README.md index 2964c883fe..cda462a3ff 100644 --- a/nodes/src/nodes/rocketride_sql/README.md +++ b/nodes/src/nodes/rocketride_sql/README.md @@ -1,53 +1,112 @@ # rocketride_sql -A RocketRide-managed database node that answers natural-language questions against your own provisioned RocketRide cloud database and inserts structured pipeline data into tables — with **zero database setup**. +A RocketRide database node for asking natural-language questions of, and writing +structured pipeline data to, the relational database provisioned for the signed-in +RocketRide tenant. Pick it instead of a connection-configured PostgreSQL node when +the data belongs in that managed tenant database. ## What it does -The same two roles as the generic `db_postgres` node. As a pipeline node, it receives natural-language questions on the `questions` lane, asks a connected LLM to translate them into SQL, executes the query, and emits the results; it also accepts structured data on the `answers` lane and inserts it into the configured table. As a tool node, agents call it directly through `get_data`, `get_schema`, `get_sql`, `execute`, and `dialect`. +On the questions lane, the node asks its connected LLM to produce a SQL query, +validates a safe query with EXPLAIN, and returns the result as a table, text, or +answer. On the answers lane, it inserts structured rows into the configured +table. Use it for relational queries and writes against the tenant database; +rocketride_vector is the sibling for document embeddings and rocketride_graph is +the sibling for Cypher graph queries. -The defining difference: **there are no connection fields**. Instead of host/user/password, the node resolves a ready per-tenant DSN from the account layer (`Account.resolve_db_dsn(client_id)`), keyed by the authenticated connection identity. The RocketRide cloud provisions one database per tenant; the same database backs `rocketride_sql`, `rocketride_vector`, and `rocketride_graph`, so raw SQL over the vector tables also goes through this node. +## Connections -Requires signing into RocketRide cloud. On the open-source build without a cloud identity the node fails at start with `RocketRide cloud DB nodes require signing into RocketRide cloud`. +| Connection | Required | Description | +| --- | --- | --- | +| llm | yes | Produces SQL from a natural-language question. | + +## Lanes + +| Lane in | Lane out | Description | +| --- | --- | --- | +| answers | — | Inserts structured pipeline rows into the configured table. | +| questions | table | Returns an executed query result as a Markdown table. | +| questions | text | Returns the query result as text. | +| questions | answers | Returns the query result on the answers lane. | + +## As a tool + +The inherited database functions are registered under the bare names below; this +node defines no configurable server-name prefix. Input must be a JSON object +unless a function explicitly permits an empty object. + +| Function | Description | +| --- | --- | +| get_data | Converts a required natural-language question into safe SQL, executes it, and returns rows. | +| get_schema | Returns reflected tables, columns, primary keys, and foreign keys; table is optional. | +| get_sql | Converts a required question into SQL without executing it. | +| execute | Runs required raw SQL, with optional positional params and a transaction session_id. | +| begin | Opens a raw-SQL transaction and returns its session_id. | +| commit | Commits the required session_id. | +| rollback | Rolls back the required session_id. | +| dialect | Returns {"dialect": "postgres"}. | + +get_data returns {valid, rows, sql, row_limit} for a successful query. A +generation or execution problem returns valid: false with error, SQL, or an LLM +answer as applicable. It defaults to 250 rows; a supplied limit is clamped to +the shared maximum. get_schema reports an unknown requested table as an error +value rather than throwing. + +get_sql returns {sql, valid: true} only for safe generated SQL; unsafe SQL +returns {error, sql, valid: false}. execute, begin, commit, and rollback raise +for invalid input, an unknown or expired transaction, or when direct execution +is disabled. A successful raw execution returns {rows, affected_rows}; begin +returns {session_id} and transaction completion returns {ok: true}. -Safety defaults match `db_postgres`: only `SELECT` statements are permitted for LLM-generated queries, generated SQL is validated with `EXPLAIN` before execution, and raw SQL execution (`QuestionType.EXECUTE`) is disabled by default via `allow_execute`. Isolation for raw execution comes from the database-per-tenant boundary, not query inspection. +## Configuration ---- +There is one built-in profile and no connection panel. RocketRide provisions a +per-tenant database for its managed database nodes, and this node resolves it +from the signed-in RocketRide identity instead of a host, user, password, or +database name you enter. Configure the table and the context supplied to the +LLM; leave direct execution disabled unless a trusted caller needs it. -## Connections +### Table name and database description -| Connection | Required | Description | -| ---------- | -------- | ---------------------------------------------- | -| `llm` | yes | LLM used to generate SQL from natural language | +Table name defaults to table and is the target used for structured answers-lane +inserts. Database description is empty by default and is included as context +when the node asks the LLM to write SQL. Change it when the database or table +has domain-specific meanings that a column name alone cannot convey; a concise +description helps the LLM choose relevant tables and predicates without +changing the actual schema. ---- +### Max validation attempts -## Configuration +The node defaults to five LLM attempts when EXPLAIN rejects generated SQL. +Raise it when a complex, well-described schema produces repairable SQL errors; +lower it when fast failure matters more than another LLM round trip. It affects +only the natural-language path, not raw execute calls. -### Lanes +### Allow direct query execution -| Lane in | Lane out | Description | -| ----------- | --------- | -------------------------------------------------------------- | -| `questions` | `table` | Translate question to SQL, execute, return as a markdown table | -| `questions` | `text` | Translate question to SQL, execute, return as text | -| `questions` | `answers` | Translate question to SQL, execute, return as answers | -| `answers` | (none) | Parse structured rows and insert into the table | +This setting is off by default. When enabled, QuestionType.EXECUTE on the +questions lane and the execute, begin, commit, and rollback tools can run raw +SQL without LLM translation or SQL safety checks. Enable it only for a trusted +application that needs write statements or explicit transactions; otherwise +keep it off so those entry points fail rather than executing input. -Two special question types are handled on the `questions` lane: +## Limitations -- **`QuestionType.DIALECT`**: emits `{"dialect": "postgres"}` on the `answers` lane so SDK callers can branch on the underlying engine. -- **`QuestionType.EXECUTE`**: runs the question text as raw SQL (read or write, no LLM, no safety check). Gated by `allow_execute`; when disabled the request is logged and dropped. `SELECT` results are capped at 25,000 rows; write statements report `affected_rows`. +This node is marked noremote and depends on a signed-in RocketRide identity to +resolve the per-tenant DSN. It cannot start where that cloud identity is not +available; DSN resolution is deliberately not replaced by host or credential +fields. LLM-generated queries are limited to the safe SQL path, while raw SQL +is unavailable until direct execution is explicitly enabled. -### Fields +## Notes -| Field | Type | Description | -|---|---|---| -| `table` | string | Default "table". Name of the table to read from or write to | -| `db_description` | string | Default empty. What is this database used for? Helps the LLM generate more accurate queries. | -| `max_attempts` | integer | Default 5. Maximum number of times to re-ask the LLM if EXPLAIN rejects the generated SQL | -| `allow_execute` | boolean | Default false. Permit QuestionType.EXECUTE callers to run raw SQL without LLM translation or safety checks. | +### Query paths -There are intentionally no `host` / `user` / `password` / `database` fields — the connection is resolved from your signed-in RocketRide identity. +The node inherits PostgreSQL schema reflection and its structured query surface. +For QuestionType.DIALECT, the questions lane emits the PostgreSQL dialect on +answers. For QuestionType.EXECUTE, a disabled direct-execution setting logs and +drops the request; successful raw SELECT results are bounded by the shared +execution-row maximum, while writes report affected_rows. diff --git a/nodes/src/nodes/rocketride_vector/README.md b/nodes/src/nodes/rocketride_vector/README.md index 622821a96e..e5e48e1009 100644 --- a/nodes/src/nodes/rocketride_vector/README.md +++ b/nodes/src/nodes/rocketride_vector/README.md @@ -1,42 +1,84 @@ # rocketride_vector -A RocketRide-managed vector store backed by PostgreSQL + pgvector in your own provisioned RocketRide cloud database — with **zero database setup** and a real vector index from the first write. +A RocketRide vector store node that stores embedded document chunks and retrieves +them by keyword or semantic similarity from the managed tenant database. Pick it +over the SQL and graph nodes for retrieval-augmented document search. ## What it does -Mirrors the generic `vectordb_postgres` store node: accepts documents (with embeddings from a bound `vectorizer`) on the `documents` lane, upserts them into a pgvector table, and serves keyword and semantic search on the `questions` lane. +The node writes documents from the documents lane into a pgvector-backed table, +replacing existing chunks for the same object IDs. Questions can produce +matching documents, answers, or enriched questions through the three configured +question outputs. Use it when retrieval should flow through a pipeline; unlike +the tool-capable sibling stores, this node registers no agent tools or raw-SQL +execution surface. -Two differences from the generic node: +## Lanes -1. **No connection fields.** Instead of host/user/password, the node resolves a ready per-tenant DSN from the account layer (`Account.resolve_db_dsn(client_id)`), keyed by the authenticated connection identity. Requires signing into RocketRide cloud; on the open-source build without a cloud identity the node fails at start with `RocketRide cloud DB nodes require signing into RocketRide cloud`. -2. **Default HNSW index.** The generic node creates no index, so every semantic search is a sequential scan. This node creates an HNSW index over the embedding column when the table is first created. The operator class is derived from the `similarity` config so Postgres actually uses the index (`cosine → vector_cosine_ops`, `l2 → vector_l2_ops`, `inner_product → vector_ip_ops`), with build parameters `m = 16`, `ef_construction = 64` (overridable). pgvector's HNSW supports at most 2000 dimensions; for wider vectors the index is skipped with a warning and search falls back to a sequential scan. +| Lane in | Lane out | Description | +| --- | --- | --- | +| documents | — | Stores document chunks in the configured vector table. | +| questions | documents | Returns matching documents. | +| questions | answers | Returns matching documents as answers. | +| questions | questions | Enriches a question with matching documents. | -There is **no direct-execute path** — vector stores are structured (search/upsert), and raw SQL over the vector tables is covered by `rocketride_sql` (same tenant database). +## Configuration -Embeddings come from the separate `vectorizer` binding — not in-node. +The single cloud profile provides a table, cosine similarity, a score threshold, +and HNSW index defaults. RocketRide provisions a per-tenant database for its +managed database nodes, and this node resolves it from the signed-in RocketRide +identity instead of a host, user, password, or database name you enter. Bind an +embedding module for semantic search; the embedding dimension is taken from the +first stored document rather than from a configuration field. ---- +### Table -## Configuration +Table defaults to rocketride and is the PostgreSQL table that holds the chunks +and embeddings. Choose a distinct table when separate corpora need separate +retrieval indexes or retention behavior. The node accepts only an unquoted +PostgreSQL identifier: it must start with a letter or underscore, use only +letters, digits, and underscores, and be at most 63 characters. Invalid names +are rejected during configuration validation and at startup. + +### Score threshold and similarity metric + +Score threshold defaults to 0.5; it is the minimum returned similarity score. +Raise it when loose matches are polluting downstream context, and lower it when +relevant documents are being excluded. Scores are calculated from the +configured metric: cosine uses 1 - distance, L2 uses 1 / (1 + distance), and +inner product negates the returned distance. Regardless of this setting, the +store drops results below its fixed 0.20 minimum similarity floor. + +Similarity Metric defaults to cosine; l2 and inner_product are also accepted. +Select the metric that matches the embeddings and expected notion of closeness +before the table is first written, because it chooses the HNSW operator class +used for the index. An unsupported value prevents startup. + +### HNSW m and ef_construction -### Lanes +The table's HNSW index is created on first write. HNSW m defaults to 16 and +controls the graph degree; HNSW ef_construction defaults to 64 and controls the +candidate list used while building it. Higher values can improve search quality +at the cost of a more expensive, larger index. Values are clamped to pgvector's +supported ranges (m 2–100 and ef_construction 4–1,000), and ef_construction is +raised to at least twice m. These values do not rebuild an index that already +exists. -| Lane in | Lane out | Description | -| ----------- | ----------- | ---------------------------------------------------- | -| `documents` | (none) | Upsert document chunks into the vector table | -| `questions` | `documents` | Keyword / semantic search, results emitted as documents | +## Notes -### Fields +### Storage and retrieval behavior -| Field | Type | Description | -|---|---|---| -| `collection` | string | Default "rocketride". Name of the table to store vectors | -| `score` | number | Default 0.5. Minimum similarity score for a document to be returned | -| `similarity` | string | Default "cosine". One of `cosine`, `l2`, `inner_product`. Also selects the HNSW index operator class | -| `hnsw_m` | integer | Default 16. HNSW graph degree used when the index is first created | -| `hnsw_ef_construction` | integer | Default 64. HNSW build-time candidate list size used when the index is first created | +The node creates the table on the first write and creates a metric-compatible +HNSW index then. pgvector cannot create that index for embeddings wider than +2,000 dimensions, so the node warns and searches without the index in that +case. Keyword search uses a content LIKE match; semantic search needs an +embedding bound to the question and raises if none is available. Missing tables +produce empty search results rather than an error. -There are intentionally no `host` / `port` / `user` / `password` / `database` fields — the connection is resolved from your signed-in RocketRide identity. The vector dimension is not configured; it is derived from the first document's embedding at write time. +Deleted objects are excluded by default, while document rendering reassembles +stored chunks by chunkId. The store removes all existing chunks for incoming +object IDs before inserting the replacement chunks, preventing duplicate data +for a re-ingested object. diff --git a/nodes/src/nodes/search_exa/README.md b/nodes/src/nodes/search_exa/README.md index 13cb875853..ce4ffae955 100644 --- a/nodes/src/nodes/search_exa/README.md +++ b/nodes/src/nodes/search_exa/README.md @@ -2,38 +2,40 @@ A RocketRide search node that submits a user question to the Exa web search API and returns the raw results as pipeline output. +## About Exa + +Exa provides the search API called by this node. The node posts a query and the configured search options to Exa, then returns the response body as formatted JSON after applying its local URL-safety checks. + ## What it does Takes a question from the `questions` lane, sends it as a single query to the Exa REST endpoint (`https://api.exa.ai/search`), and writes the pretty-printed Exa JSON response to the `answers` and `text` lanes. Uses the **requests** library directly against the Exa REST API; no Exa SDK is required. Each HTTP request has a 30-second timeout. The node expects exactly one question per invocation: an empty question or a multi-question payload raises an error immediately. -Result URLs are sanitized before leaving the node. Every `url`, `image`, and `favicon` field in the Exa response is validated to be an `http`/`https` URL whose host resolves to a public IP address. Results whose primary `url` resolves to a private, loopback, link-local, reserved, multicast, or unspecified address are dropped entirely; invalid `image` and `favicon` fields are removed from the result without dropping the whole result. This guards downstream nodes against SSRF via attacker-influenced search content. Sanitization is skipped when the `ROCKETRIDE_MOCK` environment variable is set, which is useful for local testing. +Result URLs are sanitized before leaving the node, guarding downstream nodes against SSRF via attacker-influenced search content; see **URL safety** under Notes for what is checked and dropped. --- -## Configuration - -### Lanes +## Lanes | Lane in | Lane out | Description | |-------------|-----------|----------------------------------------------------------------| | `questions` | `answers` | Search results as answers (pretty-printed Exa JSON) | | `questions` | `text` | Search results as plain text | -| `questions` | `questions` | Original question passed through unchanged (when a downstream listener is connected) | -### Fields +## Configuration + +The single default profile starts with automatic search, five results, and highlights enabled. Supply credentials, then tune the search options only when the result style or response size needs to change. + +### Search Type -| Field | Type | Description | -|---|---|---| -| `profile` | string | Default "default". | -| `apikey` | string | Default empty. Exa API key | -| `type` | string | Default "auto". | -| `numResults` | integer | Default 5. | -| `includeHighlights` | boolean | Default true. | -| `highlightChars` | integer | Default 600. | +Choose `auto` for the default request behavior. Select `keyword` or `neural` when the query should explicitly use that Exa search type. The selected value is passed as the API request's `type`; it does not change how RocketRide interprets the returned payload. -The node ships a single `default` profile containing the search settings (`type`, `numResults`, `includeHighlights`, `highlightChars`). The `apikey` field sits outside the profile and is set once per connection config. +### Results and highlights + +`numResults` controls how many results the request asks for and accepts values from 1 through 20. Keep the default 5 for a compact response; increase it when later stages need a broader result set, bearing in mind that the entire raw response is sent to both output lanes. + +When highlights are enabled, the node sends a `contents.highlights` request with the configured maximum character count. The default is 600 characters; the allowed range is 100 through 4,000. Turn highlights off when result metadata is sufficient, or reduce their size when downstream context is more valuable than excerpts. --- @@ -49,7 +51,9 @@ If none of these sources provides a non-empty value, the pipeline fails at start --- -## Error handling +## Notes + +### Request failures Exa HTTP errors are mapped to descriptive failures: @@ -59,6 +63,14 @@ Exa HTTP errors are mapped to descriptive failures: - Network timeout: `search_exa: Exa request timed out` - Connection failure: `search_exa: Unable to reach Exa` +### URL safety + +Before serializing results, the node validates every `url`, `image`, and `favicon` field. Each must use `http` or `https` and resolve to a public IP address. A result whose primary `url` resolves to a private, loopback, link-local, reserved, multicast, or unspecified address is dropped entirely; invalid `image` and `favicon` fields are removed while their result is retained. This sanitization is skipped when `ROCKETRIDE_MOCK` is set. That bypass is intended only for controlled tests and must never be enabled in production. + +## Upstream docs + +- [Exa documentation](https://docs.exa.ai/) + --- diff --git a/nodes/src/nodes/telegram/README.md b/nodes/src/nodes/telegram/README.md index 1d61ddc317..0c76c7d3ea 100644 --- a/nodes/src/nodes/telegram/README.md +++ b/nodes/src/nodes/telegram/README.md @@ -1,84 +1,58 @@ # telegram -A RocketRide source node that connects a Telegram bot to your pipeline, routing incoming messages to typed lanes and returning pipeline answers to the sender. +A RocketRide source node that connects a Telegram bot to a pipeline; choose it when a bot’s messages and media should initiate pipeline work and receive a reply. -## What it does - -A `source` node (`telegram://`) that authenticates with a bot you create via @BotFather and listens for incoming messages. It handles text and media alike: photos, audio, voice notes, video, and documents are each downloaded (up to Telegram's 20 MB bot file limit) and routed to the matching pipeline lane. The first answer produced by the pipeline is sent back to the originating chat via `sendMessage` automatically. - -Talks to the Telegram Bot API directly over **aiohttp** with no Telegram SDK dependency. In webhook mode the incoming POST route is registered on the shared FastAPI web server that `ai/node.py` starts for the pipeline subprocess — the node does not create a server of its own. - -Both new and edited messages are processed. Unsupported message types (stickers, locations, polls, and so on) are silently ignored. - ---- - -## Configuration - -### Lanes +## About Telegram -The node is a pipeline source. Its `_source` lane emits to `text`, `image`, `audio`, `video`, and `tags`. Each Telegram message type maps to one output lane: +Telegram is a messaging service with a Bot API. This node uses that API to receive updates, download attached files, and send text replies to the originating chat. -| Telegram message | Output lane | Notes | -|------------------|-------------|-------| -| Text | `text` | Written as plain text. | -| Photo | `image` | The largest available photo size is downloaded; MIME type `image/jpeg`. | -| Audio | `audio` | MIME type from the message, default `audio/mpeg`. | -| Voice note | `audio` | MIME type from the message, default `audio/ogg`. | -| Video | `video` | MIME type from the message, default `video/mp4`. | -| Document (PDF, Word, etc.) | `tags` | Written as tagged stream data; connect a Parser node downstream. | - -Entry URLs are built as `telegram:///` for text messages and `telegram:///` for files. - -### Fields - -| Field | Type | Description | -|---|---|---| -| `botToken` | string | Telegram bot token from @BotFather (e.g. 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11) | -| `mode` | string | Default "polling". Polling works anywhere without a public URL. Webhook requires a public HTTPS endpoint. | -| `webhookUrl` | string | Public HTTPS URL Telegram will POST updates to (e.g. https://your-server.com/telegram/webhook). Required for webhook mode. | +## What it does -The node tile in the UI shows the currently configured mode. +Telegram receives text or supported attachments, sends each item to the matching pipeline lane, and returns the first pipeline answer to the sender. Choose it instead of a generic webhook when the input and response should follow Telegram’s bot update and reply flow. It supports polling for outgoing connections to the Bot API and webhook delivery for a publicly reachable callback. -The monitor info panel shows the last 6 characters of the configured bot token so you can verify which bot is connected without exposing the full secret. +## Lanes ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `_source` | `text` | Text messages are written as text. | +| `_source` | `image` | Photos are downloaded and written as JPEG image data. | +| `_source` | `audio` | Audio files and voice messages are written as audio data. | +| `_source` | `video` | Videos are written as video data. | +| `_source` | `tags` | Documents are written as tagged data for a downstream parser. | -## Connection modes +## Configuration -### Polling +Set a bot token, then choose the delivery mode that your deployment can support. Polling is the default and needs no public callback. Webhook mode additionally needs a reachable callback URL. -The default. The node long-polls the Telegram `getUpdates` API in a background task using a 30-second server-side timeout and a batch size of up to 100 updates. The offset is advanced after each processed update, so acknowledged messages are never re-delivered. On network or API errors the loop sleeps 5 seconds then retries. Any previously registered webhook is cleared at startup because Telegram refuses `getUpdates` while a webhook is active. +### Connection Mode -### Webhook +`polling` is the default. It clears any previously registered webhook, repeatedly calls Telegram’s update endpoint with a 30-second server timeout and up to 100 updates, and retries after errors; use it where the pipeline process can make outbound requests but cannot expose a public route. -For production deployments with a public HTTPS endpoint. At startup the node registers `telegram.webhookUrl` with Telegram via `setWebhook` along with a freshly generated random secret token. Incoming POSTs are validated against the `X-Telegram-Bot-Api-Secret-Token` header; requests with a wrong or missing secret are rejected with HTTP 403. Each accepted update is handled concurrently as a background task. In-flight handlers are awaited during shutdown, and the webhook is deregistered via `deleteWebhook` when the pipeline stops. +Choose `webhook` only when Telegram can POST to a public HTTPS URL that reaches this pipeline. The node registers that URL and derives its local POST route from its path (or `/telegram/webhook` when the URL has no path). In webhook mode, incoming requests without the generated secret header are rejected with HTTP 403; malformed request bodies return HTTP 500. -The local POST route is derived from the path portion of `telegram.webhookUrl`, falling back to `/telegram/webhook` if the URL contains no path. Your reverse proxy or tunnel must forward that path to the shared web server's port (the `--data_port` the engine assigns to the pipeline subprocess). +### Webhook URL ---- +This value is used only in webhook mode. Provide the complete public URL, including the path your reverse proxy or tunnel forwards to the pipeline’s shared web-server port. Leaving it empty makes webhook registration fail; use polling when a public route is unavailable. -## Replies +## Authentication -After a message runs through the pipeline, the first answer in the pipeline response is sent back to the originating chat. Replies longer than Telegram's 4096-character limit are truncated with a trailing ellipsis (`...`). If the pipeline produces no answers, nothing is sent. Reply failures are logged via `debug()` and never crash the update handler. +Set **Bot Token** to the token for the bot that should receive updates. The node uses it in its Bot API requests and displays only its final six characters in the monitor information. A missing token aborts startup after publishing a missing-token status. ---- +## Notes -## Limits & behavior notes +### Message handling and replies -- **20 MB file cap** -- Telegram's Bot API limit for downloads. The node checks the size reported by `getFile` and skips larger files silently. -- **One answer per message** -- only the first pipeline answer is returned to the chat; additional answers are discarded. -- **Missing token** -- if `telegram.botToken` is empty the node reports `Telegram Bot: missing bot token` in the monitor and stays idle. -- **Byte accounting** -- processed message and file sizes are reported to the monitor as completed or failed bytes via `monitorCompleted` / `monitorFailed`. +Both new and edited messages are considered. Unsupported updates are ignored. Files reported larger than 20 MiB are skipped before download, and download or pipeline failures return no reply while recording failed bytes in the monitor. ---- +Only the first answer in the pipeline response is sent back. Replies longer than 4,096 characters are truncated with an ellipsis; errors from Telegram’s send operation are logged and do not crash the update handler. -## Authentication +### Webhook reachability -This node requires a Telegram bot token. Create a bot by messaging @BotFather on Telegram and following the `/newbot` flow. Copy the token BotFather provides and paste it into the `telegram.botToken` field. +Webhook delivery uses the pipeline’s shared web server and requires the process entry point to provide it. The implementation notes that a multi-tenant cloud engine cannot expose an individual pipeline subprocess port to Telegram, so use polling there; the local POST route can still be used by a self-hosted deployment that forwards it publicly. -No additional OAuth or API key registration is needed beyond the bot token. +## Upstream docs ---- +- [Telegram Bot API](https://core.telegram.org/bots/api) diff --git a/nodes/src/nodes/text_output/README.md b/nodes/src/nodes/text_output/README.md index 977ca397f4..9a21aed48f 100644 --- a/nodes/src/nodes/text_output/README.md +++ b/nodes/src/nodes/text_output/README.md @@ -1,96 +1,54 @@ # text_output -A RocketRide target node that writes the pipeline's extracted text to an SMB network share, with optional anonymization of classified (sensitive) data. +A RocketRide target node that writes a pipeline object’s text to an SMB share; choose it when the destination is network storage rather than the engine’s local filesystem. -## What it does - -Saves your pipeline's text to networked storage over SMB. Each upstream object becomes a -`.txt` file, mirroring the source directory layout under the store path. It is the end of -the line, it consumes the `text` lane and emits nothing. +## About SMB -Uses **smbclient / smbprotocol**: a pure-Python SMB client, so no host SMB mount or -`smbclient` binary is required on the machine running the engine. +SMB is the network file-sharing protocol used by this node to reach its target. The implementation connects to the configured server and share through Python SMB client libraries. -Output is UTF-8, the source file extension is replaced with `.txt`, and target -subdirectories are created automatically. Empty objects are skipped ("no text extracted"), -and so are objects unchanged since the last run, so the same file is never rewritten twice. +## What it does -Optionally, the node can **anonymize PII**: classification hits in the text are replaced -with a masking character before the file is written. +The node consumes text, writes non-empty output as UTF-8 `.txt` files on the configured SMB share, and produces no downstream lane. It retains the source-derived directory structure and skips unchanged objects, making it appropriate for incremental exports to a reachable share. Choose `local_text_output` instead when the desired destination is the pipeline host’s local filesystem rather than a network server. -Requires the `network` capability and is not available in remote (`noremote`) or SaaS -(`nosaas`) deployments. +## Lanes ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `text` | — | Text content to write to the SMB share. | ## Configuration -### Lanes - -| Lane in | Description | -| ------- | -------------------------------------- | -| `text` | Text content to write to the SMB share | +Configure the SMB server and destination path first. Authentication is optional for a guest-accessible share; anonymization settings only matter when sensitive content should be masked before export. -The node is a pure target (`classType: ["target"]`), it produces no output lanes. +### SMB destination -### Fields +The node validates the server name as a hostname or IP address and requires a store path between 3 and 256 characters. The first store-path segment identifies the share; it may be followed by folders. The path cannot be rooted, contain empty or dot folders, or contain `<>:"|?*`. -| Field | Type / Default | Description | -| --------------- | ------------------- | ---------------------------------------------------------------------------------------------------- | -| `server` | string, required | SMB server hostname or IP address (validated against RFC-1123). | -| `username` | string | SMB user in domain format: `DOMAIN\user`. Required when `password` is set. | -| `password` | string | SMB password, max 127 characters. Required when `username` is set. | -| `storePath` | string, required | Share name plus optional subfolders, e.g. `share/folder/subfolder`. 3–256 characters. | -| `anonymize` | boolean, `false` | Mask sensitive data in the text before writing. Enabling it reveals the two fields below. | -| `anonymizeChar` | string (1 char), `█` | The character used to mask each character of a classification hit. Required when `anonymize` is on. | -| `anonymizeAll` | boolean, `false` | Collapse every hit to a fixed length instead of masking character-for-character (`SSN: ***` vs `SSN: ***********`). | +At action startup, the node configures SMB credentials only when both username and password are supplied, verifies the share, and checks the full destination path. Use a `DOMAIN\\user` username when credentials are needed; supplying only one credential fails validation. Choose a share that the running task can reach, because configuration validation deliberately does not make the SMB connection. -### storePath rules +### Anonymization -The path must not be rooted, must not contain empty or dot (`.` / `..`) folders, and must -not contain the characters `<>:"|?*`. The first segment is the share name (1–80 -characters). When the pipeline starts, the node verifies that `//server/share` is -reachable; missing subfolders under the share are created on first write. +With anonymization off (the default), the node writes the incoming text unchanged. Turn it on when the pipeline’s classification output must be masked before the file is written: the node requests classification and applies the `anonymize_text` filter. The masking character defaults to `█`; it must be exactly one character when anonymization is enabled. ---- +Enable **anonymize all** when every classified match should become a fixed three-character mask instead of a character-for-character replacement. Changing the classification policies, masking character, or this setting changes the stored settings key and causes all objects to be transformed again, including objects that would otherwise be unchanged. -## Anonymization - -When `anonymize` is enabled, the node injects the classification filter plus an -`anonymize_text` pipe filter, so classification hits in the incoming text are replaced -with `anonymizeChar` before the file is written. With `anonymizeAll` enabled, each hit is -collapsed to a fixed length (3 masking characters) instead of being masked -character-for-character. - -Changing any anonymization setting (the classify policies, `anonymizeChar`, or -`anonymizeAll`) changes the settings key the node keeps in its key-value store, which -forces **all** objects to be re-transformed on the next run, not just new and changed ones. +## Authentication ---- +Credentials are optional. For a protected SMB share, set both username and password; the username must contain a domain-like prefix and the password may be at most 127 characters. For a guest-accessible share, leave both blank. -## Change detection +## Limitations -The node performs incremental writes. For each object it builds a transform key of the form -`flags;sourceChangeKey;targetChangeKey`, where the source change key comes from the -object's change key (or its modify time and size) and the target change key from the -existing target file's mtime and size (`0;0` when the file does not exist yet). The key is -stored in the object's instance tags under `text-output:////status`. +This node is excluded from SaaS and remote deployments. It opens a direct SMB connection from the running task to the configured server, so the task environment must have network access to that share and the required SMB dependencies; use a deployment located where that share is reachable. -On the next run an object is skipped ("object transformed and not changed") when its -transform key matches the stored one and the anonymization settings have not changed. -Failed objects record the exception as their completion code instead of writing a file. +## Notes ---- +### Incremental writes -## Authentication +For each object, the node derives a target path, hash-truncates over-long path components, and records a transformation key containing object flags, the source change key, and the target file’s modification time and size. An object is skipped when that key still matches and anonymization settings have not changed. Empty text is skipped, while write or SMB errors are stored as the object’s completion error. -Authentication is optional, leave `username` and `password` blank for shares that allow -anonymous/guest access. When credentials are provided, both fields are required and the -username must use the domain format `DOMAIN\user`. Credentials are registered with the SMB -client globally at connection time; the connection (and reachability of the share) is -tested when the action starts, not during configuration validation. +## Upstream docs ---- +- [smbprotocol project](https://github.com/jborean93/smbprotocol) diff --git a/nodes/src/nodes/webhook/README.md b/nodes/src/nodes/webhook/README.md index 422fa0833a..7151b446cc 100644 --- a/nodes/src/nodes/webhook/README.md +++ b/nodes/src/nodes/webhook/README.md @@ -1,67 +1,53 @@ # webhook -A RocketRide source node that lets external input reach a pipeline over HTTP: three variants (Webhook, Chat, and Dropper) served by a single shared implementation. +A RocketRide source-node directory offering raw HTTP intake, browser chat and upload interfaces, and a tool-only endpoint; choose the service that matches how a client should enter the pipeline. ## What it does -Exposes an HTTP endpoint and forwards incoming data into the attached pipeline. All three variants are `source` nodes registered as endpoints and share the same code (`nodes.webhook`); they differ only in protocol and in the surface they expose: +The directory provides four endpoint services that share a long-running, engine-provided web server: Webhook, Chat, Dropper, and Tools. Choose **Webhook** for programmatic HTTP intake, **Chat** for browser-submitted questions, **Dropper** for browser file uploads, or **Tools** when an application needs to call connected tools without adding an agent or data lane. Each service is a source, so it starts work rather than consuming a preceding pipeline lane. -- **Webhook** (`webhook://`): a raw HTTP intake. External tools, scripts, or services POST documents, media, or data to the URL, triggering the pipeline to process the uploaded content. The same endpoint also backs the RocketRide DataToolchain (`adtoolchain`) flow. -- **Chat** (`chat://`): serves a web-based chat UI. Users open the chat URL in a browser and type questions; each submission flows through the pipeline and results are returned in the chat window. -- **Dropper** (`dropper://`): serves a web-based drag-and-drop file upload UI. Users drop files onto the page; each upload is sent through the pipeline, and results are displayed in the browser across JSON, text, table, and image tabs. +## Connections -The server is a **FastAPI / Uvicorn** wrapper (`ai.web.WebServer`), but the node no longer creates it. One shared server is started per pipeline subprocess by `ai/node.py`, bound to the address the engine passes on the command line (`--data_host`, `--data_port`). That server loads the `data` module, which registers a `/task/data` websocket as the data plane between the public endpoint and the pipeline; the node simply registers its target endpoint on it. +### Tools (`tools://`) -Because the shared server only exists when `node.py` is the process entry point and `--data_port` is supplied — which is what the task manager does for every pipeline subprocess — a pipeline sourced by this node cannot run outside that path. If the shared server is missing, the node fails immediately with an error saying so rather than a `NoneType` attribute error. +| Connection | Required | Description | +| --- | --- | --- | +| `tool` | no | Tool nodes hosted by this endpoint through the control-plane invoke channel. | -The node keeps running until the pipeline is stopped; the source task completes only when the process shuts down. +The Tools service has no data outputs. It keeps the endpoint alive so a client can reach the connected tools directly. -After the pipeline starts, the Project Log displays a stable, pipe-specific interface URL, the public authorization key, and the private token, so callers know how to reach the endpoint. The URL forms are `{host}/chat/{project_id}/{source}?auth={public_auth}`, `{host}/dropper/{project_id}/{source}?auth={public_auth}`, and `{host}/webhook/{project_id}/{source}`. The legacy `/chat`, `/dropper`, `/webhook`, and `/task/data` URLs continue to work for existing integrations. +## Lanes ---- +| Lane in | Lane out | Description | +| --- | --- | --- | +| `_source` | `questions` | **Chat** emits each browser-submitted message as a question. | +| `_source` | `tags` | **Dropper** emits uploaded files as tagged data. | +| `_source` | — | **Tools** is an invoke host and emits no data lane. | +| `_source` | `tags` | **Webhook** can emit tagged input. | +| `_source` | `text` | **Webhook** can emit text input. | +| `_source` | `json` | **Webhook** can emit JSON input. | +| `_source` | `audio` | **Webhook** can emit audio input. | +| `_source` | `video` | **Webhook** can emit video input. | +| `_source` | `image` | **Webhook** can emit image input. | +| `_source` | `questions` | **Webhook** can emit question input. | ## Configuration -### Lanes +These services expose no node-specific settings beyond the standard source properties. Select the service whose interaction model fits the caller, then wire its declared output lane to the downstream node that handles that data type. -Each variant takes the internal `_source` input and emits to its declared output lanes: +### Endpoint service -| Variant | Lane in | Lanes out | -| -------- | ------- | ---------------------------------------------------- | -| Webhook | - | `tags`, `text`, `json`, `audio`, `video`, `image`, `questions` | -| Chat | - | `questions` | -| Dropper | - | `tags` | - -- **Webhook**: data received from the HTTP request, routed by content type. -- **Chat**: each message submitted via the chat UI becomes a question. -- **Dropper**: each uploaded file enters the pipeline for processing. - -None. There are no node-specific config fields, the shape exposes only the standard source properties (`source.mode` and an empty `parameters` object), and the single `default` profile is empty. The endpoint URL, public authorization key, and private token are generated automatically when the pipeline starts. - ---- - -## Startup status - -When the server is up, the node emits a ready status to the monitor. Because this is the source component, the message also means every downstream component (embedding, LLM, etc.) has already been initialized: - -| Variant | Status message | -| -------- | ------------------------------------------------------- | -| Webhook | `Webhook ready - system is ready to accept requests` | -| Chat | `Chat ready - system is ready to accept questions` | -| Dropper | `Dropper ready - system is ready to process files` | - ---- +Use **Webhook** when another system can send an HTTP request, **Chat** when people should enter questions in a web UI, and **Dropper** when people should upload files in a web UI. Use **Tools** only to host invoke-connected tool nodes for direct client calls: it intentionally has neither a user-facing URL nor a data lane. ## Authentication -Two credentials are published to the Project Log on startup: +On startup, Webhook, Chat, and Dropper publish an interface URL plus a public authorization key and private token in the monitor information. The Chat and Dropper button URLs include the public authorization key as the `auth` query parameter; Tools does not publish a user-facing URL. Treat the private token as a secret and restrict access to monitor information that contains it. -- **Public authorization key**: passed by clients reaching the public interface (e.g. the `auth` query parameter on the chat URL). -- **Private token**: the private credential for the endpoint. +## Notes -Both are generated per pipeline; there is nothing to configure on the node. +### Shared server lifecycle ---- +The endpoint registers its target with the shared web server initialized by `ai.node` and waits until shutdown. It does not create a server itself. If that shared server is unavailable, startup raises an explanatory error; run these services through the RocketRide pipeline process that provides it. The published interface is reachable only where deployment networking, firewalls, and reverse proxies permit access.