Skip to content

docs(nodes): migrate the infrastructure and I/O nodes #1978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dylan-savage merged 10 commits into from
Aug 25, 2026
Merged

docs(nodes): migrate the infrastructure and I/O nodes #1978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
93933ff
docs(nodes): migrate local_text_output, telegram, text_output, webhoo…
Aug 15, 2026
5e21e87
docs(nodes): migrate autopipe, core, remote, response to the README s…
Aug 15, 2026
4815e3f
docs(nodes): migrate aparavi_aql, rocketride_graph, rocketride_sql, r…
Aug 15, 2026
649bedb
docs(nodes): migrate guardrails, rerank_cohere, search_exa to the REA…
Aug 15, 2026
48ad28d
fix(docs): explain local-only node execution
kgarg2468 Aug 22, 2026
7aa388e
docs(guardrails): document custom profile
kgarg2468 Aug 22, 2026
c3ecb10
docs(search-exa): warn against production mock bypass
kgarg2468 Aug 22, 2026
737b38a
docs(webhook): clarify endpoint security constraints
kgarg2468 Aug 22, 2026
2ce1907
Merge remote-tracking branch 'origin/fix/docs' into docs/nodes-infra
dylan-savage Aug 25, 2026
6faf7b6
docs(nodes): bring the infrastructure nodes up to the new Profiles co…
dylan-savage Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 81 additions & 114 deletions nodes/src/nodes/aparavi_aql/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,147 +1,114 @@
# aparavi_aql

A RocketRide tool node that lets an AI agent query the Aparavi data governance platform in plain English using AQL (Aparavi Query Language).
A RocketRide tool node that lets an agent retrieve Aparavi STORE metadata by
asking questions in plain English. Pick it when the data is governed by an
Aparavi server rather than stored in one of RocketRide's managed databases.

## What it does

Translates natural-language questions into AQL SELECT statements, executes them against
the Aparavi REST API, and returns file-metadata rows from the Aparavi **STORE** table.
The agent never needs to know AQL or the schema: it just asks a question, and the node
handles schema knowledge, query generation, and execution internally.

AQL generation is delegated to a connected LLM node via the node's required **`llm`**
invoke connection. The full STORE column schema is fixed (no dynamic introspection) and
is injected into the LLM prompt together with AQL syntax rules and few-shot examples.

Execution goes over HTTP using the **requests** library: `POST /server/api/v3/database/query`
on the configured Aparavi server with HTTP Basic Auth, a 30-second timeout, and a default
row limit of 250. Timestamp columns returned in milliseconds are normalized to seconds.

Safety behavior: every generated query is checked before it touches the network. Only a
single SELECT statement is allowed; multi-statement input is rejected, and the keywords
`INSERT`, `UPDATE`, `DELETE`, `DROP`, `TRUNCATE`, `ALTER`, `CREATE`, `EXEC`, `EXECUTE`
are blocked anywhere in the query. A query that fails the safety check returns an error
immediately. If LLM generation or execution fails, the node makes up to 3 attempts in
total, feeding the failed AQL and the server's error message back to the LLM so it can
correct the query.

This is a pure tool node: it defines no pipeline lanes and is used only through its
agent-callable tools.

---

## Configuration



| Field | Type | Description |
|---|---|---|
| `url` | string | Default empty. Base URL of the Aparavi server, e.g. https://aparavi.example.com |
| `user` | string | Default empty. Aparavi login username |
| `password` | string | Aparavi login password |
| `db_description` | string | Default empty. What is this data used for? Describe its content and purpose, this helps the LLM generate more accurate AQL queries. |
| `profile` | string | Default "default". |
## About Aparavi

Aparavi is a data governance platform whose server exposes a query API over
metadata in its STORE table. This node uses Aparavi Query Language (AQL), an
SQL-like language, to retrieve those rows without requiring the calling agent
to construct the query itself.

## What it does

The node has a single preconfig profile (`default`).

### Invoke connections

| Connection | Min | Description |
|------------|-----|-------------|
| `llm` | 1 | LLM used to generate AQL queries from natural language. |

---

## Available tools

### get_data

The primary tool for all Aparavi data retrieval. Takes a natural-language `question`
(required string), generates AQL via the connected LLM, runs the safety check, executes
the query, and returns:


The node is a pure tool node: it has no pipeline lanes. Its required LLM turns
a natural-language request into a SELECT-only AQL query over the fixed STORE
schema, then the node sends that query to the configured Aparavi server. Pick
it over rocketride_sql when the authoritative data is Aparavi file metadata;
use the SQL node for the managed relational tenant database.

| Tool | Description |
|---|---|---|
| `get_data` | Translate natural language to AQL, execute against Aparavi, and return rows. |
| `get_aql` | Convert a natural-language question to an AQL SELECT statement without executing it. Use only when the user explicitly asks to see the query. |
| `get_schema` | FALLBACK ONLY: returns the fixed column schema for the Aparavi STORE table. Do NOT call this preemptively; only use if get_data fails or returns unexpected results. |
## Connections

| Connection | Required | Description |
| --- | --- | --- |
| llm | yes | Generates AQL from natural-language questions. |

## As a tool

### get_aql
The functions are registered under the bare names below; this node defines no
configurable server-name prefix. question arguments must be non-empty strings
in a JSON object.

Converts a natural-language `question` (required string) to an AQL SELECT statement
without executing it. Returns `{ "aql": "<query>" }`. Intended only for when the
user explicitly asks to see the query.
| Function | Description |
| --- | --- |
| get_data | Generates safe AQL for required question, executes it, and returns STORE rows. |
| get_aql | Generates AQL for required question without executing it. |
| get_schema | Returns the fixed STORE table schema; use only after retrieval fails or is unexpected. |

### get_schema
get_data returns {rows, aql, count} when successful. If the client was not
initialized, a generated query is unsafe, generation never succeeds, or all
execution retries fail, it returns {error, aql, rows: []}. Invalid tool input
raises ValueError. The node tries generation and execution up to three times;
after an API error, the next LLM attempt receives the failed AQL and error.

Fallback only: returns the fixed column schema for the STORE table as
`{ "store": "STORE", "columns": [{ name, type, description }, ...] }`. Column types are
`STRING`, `NUMBER`, `DATE`, or `OBJECT`. Agents are instructed not to call this
preemptively; it exists for when `get_data` fails or returns unexpected results.
get_aql returns {aql} and does not run the safety check or contact the API;
invalid input or an LLM failure raises. get_schema accepts an empty object and
returns {store: "STORE", columns: [...]}, where each column has a name, type,
and description.

---
## Configuration

## AQL generation
The single built-in profile has no endpoint defaults. Supply the Aparavi server
connection, then describe the data in terms that help the required LLM select
the correct fixed STORE fields. Query retry count and result limit are fixed in
the implementation rather than configurable fields.

The LLM prompt enforces these rules when generating queries:
### Aparavi Server URL

- `STORE` is the only table, no JOINs.
- Structure: `SELECT cols FROM STORE [WHERE cond] [WHICH CONTAIN 'term'] [GROUP BY col] [HAVING cond] [ORDER BY col ASC|DESC] [LIMIT n]`.
- `LIMIT 250` is added unless the user specifies a different limit.
- Size units are supported in conditions: `10 MB`, `5 GB`, `100 KB`.
- Date functions: `NOW()`, `TODAY()`, `YEAR()`, `MONTH()`, `DAY()`. `NOW()` returns seconds since the Unix epoch; DATE columns are compared in seconds (e.g. last 30 days = `NOW() - (30 * 86400)`).
- Aggregates (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`), string functions (`UPPER`, `LOWER`, `TRIM`, `LENGTH`, `SUBSTR`, `CONCAT`), `CAST`, and `CASE WHEN` are available.
- Column aliases are always double-quoted to avoid reserved-word conflicts, e.g. `COUNT(*) AS "count"`.
Provide the base URL of the Aparavi server, including its scheme, for example
https://aparavi.example.com. The node removes a trailing slash before adding
its database-query path. Change this value when the target Aparavi deployment
changes; an empty or unreachable URL leaves the tool client uninitialized or
causes its HTTP calls to return errors after a 30-second timeout.

The LLM is instructed to output only the raw AQL string; any accidental markdown fences
are stripped from the response before the safety check runs.
### Username and password

Example generations:
Username and Password are sent as HTTP Basic Auth on every Aparavi
database-query request. Set them to credentials accepted by the configured
server. Change both together when switching servers or users; a mismatched
endpoint and credentials becomes an API HTTP or connection error returned by
get_data after its retry loop.

```sql
-- "Find all PDF files larger than 10 MB"
SELECT name, parentPath, size, modifyTime FROM STORE WHERE extension = 'pdf' AND size > 10 MB LIMIT 250
### Data description

-- "Count files by extension"
SELECT extension, COUNT(*) AS "count" FROM STORE GROUP BY extension ORDER BY "count" DESC LIMIT 250
Data description is empty by default and is added as context to the LLM that
writes AQL. Describe the collection's purpose, vocabulary, and useful metadata
conventions when they are not evident from the fixed schema. This can improve
query selection without giving the model a new table: STORE remains the only
table in the generated-query prompt.

-- "Files modified in the last 7 days"
SELECT name, parentPath, size, modifyTime FROM STORE WHERE modifyTime > NOW() - (7 * 86400) LIMIT 250
```
## Authentication

---
The node uses HTTP Basic Auth with the configured username and password for
the Aparavi database-query API. It sends POST requests to
/server/api/v3/database/query and does not use a separate token field.

## STORE schema
## Limitations

The schema is hard-coded in `aql_schema.py` (mirroring Aparavi's server column
definitions) and covers roughly 100 columns: identity (`objectId`, `uniqueId`,
`dupKey`), file attributes (`name`, `parentPath`, `extension`, `size`, `mimeType`),
timestamps (`createTime`, `modifyTime`, `accessTime`, all in Unix epoch seconds),
document and email metadata, cost and storage metrics (`storageCost`, `dupCount`, `dri`),
paths, tags, datasets, classifications, ownership and permissions, audit messages,
search and classification hits, and 0/1 status flags (`isContainer`, `isDeleted`,
`isObject`, `isIndexed`, `isClassified`, `isSigned`).
This node runs on the RocketRide engine host and does not support remote
execution. The engine host must be able to reach the configured Aparavi server
using the supplied Basic Auth credentials. It only retrieves from the fixed
STORE schema: the safety check permits one SELECT statement, rejects embedded
multi-statement input, and blocks mutation and execution keywords before
sending AQL to the server. The API request uses a fixed 250-object limit and a
30-second timeout.

Timestamp normalization: values in `createTime`, `modifyTime`, `accessTime`,
`docCreateTime`, `docModifyTime`, `instanceMessageTime`, and `objectMessageTime` larger
than 10^10 are treated as milliseconds and divided by 1000, so results are always in
epoch seconds.
## Notes

---
### Query generation and result normalization

## Authentication
The LLM prompt supplies the fixed STORE schema, AQL syntax rules, and examples.
It asks for LIMIT 250 unless the question specifies another limit, while the
HTTP client itself requests at most 250 objects. Accidental Markdown fences in
the LLM output are removed. Timestamp values above 10,000,000,000 in the known
Aparavi date fields are treated as milliseconds and normalized to seconds in
returned rows.

The node authenticates to the Aparavi server with HTTP Basic Auth using the configured
`user` and `password` on every API request. Credentials are held in memory for the
lifetime of the pipeline and released when it ends.
## Upstream docs

---
- [Aparavi Data Suite documentation](https://aparavi.com/docs/data-suite/reports/create-a-new-report/)

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->
Expand Down
6 changes: 4 additions & 2 deletions nodes/src/nodes/autopipe/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The node registers as a `filter` with class type `other` and capability `interna

---

## What gets inserted
### Filter assembly

| Mode | Filters inserted |
| ---- | ---------------- |
Expand Down Expand Up @@ -44,7 +44,9 @@ Defaults come from the `default` preconfig profile in `services.json`. The prepr

---

## Remote processing
## Notes

### Remote processing

The implementation distinguishes local and remote filter placement, and the default profile carries a `remote` sub-configuration (host, port, apikey, `mode: local`). However, the remote-pipeline assembly path is currently commented out in `IGlobal.endGlobal` and the remote queue is never dispatched: all inserted filters run locally regardless of the `remote` setting.

Expand Down
30 changes: 26 additions & 4 deletions nodes/src/nodes/core/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The `hash/` and `parser/` subdirectories carry the per-service documentation pag

---

## Services
### Protocol-bearing services

| Service | File | Protocol | Class type | Lanes |
|---------|------|----------|------------|-------|
Expand Down Expand Up @@ -72,7 +72,21 @@ An internal no-op endpoint registered as both a source shape and a target shape

---

## Shared field libraries
## Lanes

| Lane in | Lane out | Description |
|---------|----------|-------------|
| `_source` | `tags` | Local File System emits source tags for downstream processing. |
| `tags` | `tags` | The Fingerprinter preserves the tags lane while adding its deterministic content fingerprint. |
| `source` | `tags` | The internal null endpoint forwards a source lane into tags without an external system. |

The parser also accepts `tags` and emits `text`, `table`, `image`, `video`, and `audio`; its protocol-specific documentation is in the `parser/` subdirectory.

## Configuration

This directory supplies several built-in services as well as shared field definitions used by other nodes. Configure the protocol-bearing service selected in a pipeline; the generated schema below is the field reference. The shared field files do not register a selectable service themselves.

### Shared field libraries

These files define common fields that are merged into a service definition as required. Field names below are exact.

Expand Down Expand Up @@ -150,7 +164,7 @@ Combines services into single selectable types for pipelines that pick one provi

---

## Google access helper (`google_access.py`)
### Google access helper (`google_access.py`)

A single reader that turns a Google tool node's `access` enum and capability toggles into one resolved object: the OAuth scopes to request, plus the write/destructive gates the node's tool functions check at invoke time.

Expand All @@ -176,12 +190,20 @@ Bundled specs:

---

## Running the tests
### Running the tests

```bash
pytest nodes/test/core/test_google_access.py -v
```

## Limitations

The Local File System service reads local paths and is marked for filesystem access, security-sensitive use, non-remote execution, and non-SaaS deployment. Run pipelines that use it where the intended files are locally accessible; it is not available in hosted RocketRide deployments and cannot be moved to a remote execution host.

## Notes

The internal Word indexer, ZIP Creation, and null endpoint are protocol-bearing engine services but are not normal user-selectable nodes. The `core` directory also contains reusable JSON field fragments; those fragments are included in the generated schema but do not themselves register pipeline protocols.

---

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
Expand Down
Loading
Loading