Skip to content

Add trace-commons-worker, trace-commons-tenant, and operator runbook - #126

Closed
zmanian wants to merge 2 commits into
trace-commons-operator-binaries-review-adminfrom
trace-commons-operator-binaries-worker-tenant
Closed

zmanian wants to merge 2 commits into
trace-commons-operator-binaries-review-adminfrom
trace-commons-operator-binaries-worker-tenant

Conversation

@zmanian

@zmanian zmanian commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Context

PR #3 of the three-PR stack. Completes the trace-commons-server side of the operator-binaries migration kicked off by nearai/ironclaw#3738 (the prune). Adds the remaining 19 operator commands across two binaries, plus the operator runbook + CI smoke wiring.

Stack: #124 (foundation) → #125 (review + admin) → this PR (worker + tenant + docs + CI).

Merge order: top-down — #125 first (it's the head of the stack at this point), then this PR.

What lands

trace-commons-worker (8 cmds) — per-subcommand bearer env

Per CLAUDE.md, each worker route has its own bearer-token gate. Each subcommand exposes its own --bearer-token-env <NAME> with a per-route default:

Subcommand Default bearer env
worker-utility-credit TRACE_COMMONS_UTILITY_CREDIT_WORKER_BEARER
worker-retention-maintenance TRACE_COMMONS_RETENTION_WORKER_BEARER
worker-vector-index TRACE_COMMONS_VECTOR_WORKER_BEARER
worker-benchmark-convert TRACE_COMMONS_BENCHMARK_WORKER_BEARER
worker-replay-dataset-export TRACE_COMMONS_EXPORT_WORKER_BEARER
worker-ranker-training-candidates TRACE_COMMONS_RANKER_WORKER_BEARER
worker-ranker-training-pairs TRACE_COMMONS_RANKER_WORKER_BEARER
process-evaluation-submit TRACE_COMMONS_PROCESS_EVALUATION_WORKER_BEARER

Verified per-route by test `worker_per_route_bearer_resolution_uses_distinct_envs` (sets 8 distinct env vars to 8 distinct tokens, fires each handler against wiremock, asserts each routed the right `Authorization: Bearer ...`).

`trace-commons-tenant` (11 cmds) — single shared bearer

Default `TRACE_COMMONS_TENANT_BEARER`. Subcommands: `tenant-policy-get`, `tenant-policy-set`, `tenant-access-grants-list`, `tenant-principal-ref`, `tenant-access-grant-create`, `tenant-access-grant-revoke`, `ranker-training-candidates`, `ranker-training-pairs`, `audit-events`, `list-traces`, `privacy-filter-canary`.

Operator runbook — `docs/operator/operator-binaries.md`

214 lines. Covers: overview, install, env-var matrix, common workflows (claim/decide/release a review lease; admin vs. worker retention; rotate tenant policy + grants; benchmark convert + lifecycle), defense-in-depth, troubleshooting (one row per error kind). Linked from `docs/operator/README.md`.

CI smoke — `scripts/operator/operator-smoke.sh` + `.github/workflows/ci.yml`

New `operator-binaries-smoke` job, sibling to `pilot-bootstrap-smoke`. Builds all four binaries and exercises `--help` / `--version` plus a happy-path parse-through per binary. Required check on every PR.

Stubbed

  • `trace-commons-tenant privacy-filter-canary` — the Ironclaw implementation reached into a local sidecar via `privacy_filter_adapter_from_env()`. Porting that requires moving sidecar-adapter code into the operator-client model, which is out of scope. Currently `anyhow::bail!`s with a pointer to legacy Ironclaw tooling. The subcommand still parses cleanly with `--text` and `--timeout-seconds`. Documented in the runbook's binaries-at-a-glance section.

Divergence from Ironclaw reference

  • Default bearer env names use the `_BEARER` suffix to match PR GCS object backend + pluggable KEK trait (production gated) #2's convention (`TRACE_COMMONS_REVIEWER_BEARER`, `TRACE_COMMONS_ADMIN_BEARER`). Ironclaw used `_TOKEN`. The ingest server uses role-based tenant-token tables (`TRACE_COMMONS_TENANT_TOKENS`) rather than per-route bearer envs, so there is no authoritative server name to mirror — env-var names are entirely client-side organizational.
  • `ConsentScope` in the tenant binary distinguishes `server_value()` (snake_case for policy/grant bodies) from `query_value()` (kebab-case for trace-list and ranker-training-export query strings). The Ironclaw reference does the same; behavior preserved.

Tests

  • Worker: 27 tests — parse-through, per-route bearer resolution, wiremock happy-path per command family.
  • Tenant: 37 tests — parse-through (22), grant-body / policy-body validation, wiremock round-trips.

Verification

```
RUSTFLAGS="-D warnings" cargo check --workspace --all-targets # clean
RUSTFLAGS="-D warnings" cargo test --workspace --no-run # clean
cargo test -p trace-commons-server --bins # 109 passed (21+24+27+37)
cargo clippy -p trace-commons-server --bins -- -D warnings # clean
cargo build --bin trace-commons-worker # builds standalone
cargo build --bin trace-commons-tenant # builds standalone
bash scripts/operator/operator-smoke.sh # SmokeOperatorBinariesOK
```

Net diff

7 files, +3,647 LOC.

Stack now complete

After this PR lands, the four-binary operator surface in `trace-commons-server` matches the 39 commands Ironclaw lost in nearai/ironclaw#3738:

Binary Cmds Bearer model
`trace-commons-review` 8 Single `TRACE_COMMONS_REVIEWER_BEARER`
`trace-commons-admin` 12 Single `TRACE_COMMONS_ADMIN_BEARER`
`trace-commons-worker` 8 Per-subcommand defaults
`trace-commons-tenant` 11 Single `TRACE_COMMONS_TENANT_BEARER`
Total 39

Out of scope

  • Porting `privacy-filter-canary` to the operator-client model (tracked above).
  • Runbook updates renaming `ironclaw traces ...` → `ironclaw-reborn traces ...` for the contributor surface (lives on the Ironclaw side; tracked separately).

🤖 Generated with Claude Code

zmanian added a commit that referenced this pull request May 20, 2026
Lands the four-PR stack (#124, #125, #126, #127) as a single squash commit on main.

## What's in this squash

**1. `trace-commons-operator-client` foundation crate** (was #124)
- `Client` wrapper around `reqwest::Client` with bearer-token resolution from env, host allowlist enforcement, typed error mapping for `{"error": "<Label>"}` refusals.
- Error taxonomy: `BearerMissing`, `InvalidEndpoint`, `HostNotAllowed`, `Transport`, `HttpFailure`, `ServerLabel`, `MalformedResponse`. `Error::user_diagnostic()` strips query strings + fragments.
- `HostAllowlist` (CSV, exact-match, case-insensitive) sourced from `--allowed-hosts` or `TRACE_COMMONS_ALLOWED_HOSTS`.
- Output helpers: `JsonEnvelope` wrapper for `--json` mode, ASCII table renderer, key-value block renderer.

**2. `trace-commons-review` (8 cmds) + `trace-commons-admin` (12 cmds)** (was #125)
- Single shared bearer per binary (`TRACE_COMMONS_REVIEWER_BEARER`, `TRACE_COMMONS_ADMIN_BEARER`).
- 37 binary tests across both (parse-through, body validation, wiremock round-trips).

**3. `trace-commons-worker` (8 cmds) + `trace-commons-tenant` (11 cmds)** (was #126)
- Worker uses per-subcommand bearer envs (8 distinct env vars, one per route gate). Verified by `worker_per_route_bearer_resolution_uses_distinct_envs`.
- Tenant uses single shared `TRACE_COMMONS_TENANT_BEARER`.
- Operator runbook at `docs/operator/operator-binaries.md`.
- CI smoke job `operator-binaries-smoke` in `.github/workflows/ci.yml`, sibling to `pilot-bootstrap-smoke`.

**4. Privacy-filter-canary port** (was #127)
- New `trace_commons_operator_client::privacy_filter` module: `PrivacyFilterAdapter` trait, `CommandPrivacyFilterAdapter`, `SafePrivacyFilterRedaction`, `adapter_from_env`, `canary_leaked_tokens`. Wire shapes byte-identical to Ironclaw's sidecar protocol.
- Dual-name env-var compat: `TRACE_COMMONS_PRIVACY_FILTER_*` canonical, `IRONCLAW_TRACE_*` legacy fallback with warn-on-use.
- `trace-commons-tenant privacy-filter-canary` stub replaced with the real handler.

## Operator surface added

| Binary | Cmds | Bearer model |
|---|---:|---|
| `trace-commons-review` | 8 | Single `TRACE_COMMONS_REVIEWER_BEARER` |
| `trace-commons-admin` | 12 | Single `TRACE_COMMONS_ADMIN_BEARER` |
| `trace-commons-worker` | 8 | Per-subcommand defaults |
| `trace-commons-tenant` | 11 | Single `TRACE_COMMONS_TENANT_BEARER` |
| **Total** | **39** | |

These mirror the 39 operator commands removed from Ironclaw in nearai/ironclaw#3738.

## Verification

```
cargo check --workspace --all-targets   (-D warnings)   # clean
cargo test --workspace --no-run         (-D warnings)   # clean
cargo clippy --workspace --all-targets  (-D warnings)   # clean
cargo test -p trace-commons-operator-client            # 39 passed
cargo test -p trace-commons-server --bins              # all green
bash scripts/operator/operator-smoke.sh                # SmokeOperatorBinariesOK
```

Superseded PRs (closed): #124, #125, #126.
@zmanian

zmanian commented May 20, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #127. The trace-commons-server stack (#124 → #125 → #126 → #127) was squash-merged top-down per the project's stacked-PR merge convention; #127's squash captures this PR's full diff (trace-commons-worker + trace-commons-tenant operator binaries + runbook + CI smoke). No code is lost.

Closing as superseded.

@zmanian zmanian closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant