Skip to content

feat(harness): trace correlation via OTel baggage + iii.* span attrs - #138

Merged
andersonleal merged 1 commit into
mainfrom
feat/harness-traces
May 15, 2026
Merged

feat(harness): trace correlation via OTel baggage + iii.* span attrs#138
andersonleal merged 1 commit into
mainfrom
feat/harness-traces

Conversation

@andersonleal

@andersonleal andersonleal commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Every harness-wrapped function tags its span with iii.session.id / iii.message.id / iii.function.id and propagates the same triple as OTel baggage. iii-sdk's BaggageSpanProcessor materializes those attrs on every downstream worker span — the Developer Console TRACES tab can now group spans per user turn.

  • harness/src/otel.rs (new, 905 lines): with_harness_span / with_envelope_span / with_raw_span wrappers, single-walk ID extraction with header-injection + control-byte guards, traceparent + x-iii-message-id response-header echo. 72 unit tests covering payload shapes, security guards, baggage round-trip.
  • harness/src/lib.rs: every registered function (harness.status / harness.call / harness.info / ui.subscribe / ui.unsubscribe / harness.fs.read_inline) goes through a wrapper.
  • harness/tests/trace_correlation.rs (new): end-to-end coverage including cross-crate allowlist parity with iii_sdk::DEFAULT_ALLOWLIST.
  • hook-fanout: emits OTel span attributes (topic, replies, elapsed_ms, exit_reason) + new quiescence-based exit so collect loops don't always wait the full timeout.
  • provider-anthropic / provider-openai: iii_sdk::capture_otel_context() across tokio::spawn so HTTP child spans inherit baggage; HTTP request build wrapped in its own iii_sdk::run_in_span.
  • harness/Makefile: new build target — cargo build --release for each local worker, symlink into ~/.iii/workers/<name>. make all now runs build → config → observability → engine → verify.
  • harness/web/src/bridge.ts: toBridgeError walks common engine-error shapes so users see real messages instead of [object Object].
  • iii-sdk pinned to =0.11.7-next.3 (crates.io) for the 4 workers that need the new span_ops / run_with_baggage helpers; the 15 unrelated workers stay on =0.11.3.

Test plan

  • cargo test --lib in harness/ (72/72 passing)
  • cargo clippy --tests -- -D warnings clean in harness/ and hook-fanout/
  • cargo fmt --all --check clean across the 4 modified crates
  • make all from harness/ succeeds: build → config → observability → engine → verify
  • iii trigger --function-id harness::status returns 14 expected workers
  • Send a chat turn; confirm Developer Console TRACES tab shows iii.message.id on every downstream span (not just the harness root)
  • iii trigger --function-id engine::traces::list --payload '{"attributes":[["iii.message.id","<id>"]],"search_all_spans":true}' returns ≥1 span

Out of scope

  • Splitting hook-fanout's quiescence + observability changes into separate commits (intertwined; can rebase if desired).
  • Dropping the worker_path: line that iii worker add . writes for the harness entry into config.yaml — manual edit needed until the iii CLI is taught to prefer existing ~/.iii/workers/ symlinks.

Summary by CodeRabbit

Release Notes

  • New Features

    • Added trace correlation and OpenTelemetry instrumentation for improved observability and request tracking.
    • Introduced automatic message ID generation for tracing individual requests.
    • Enhanced error handling with clearer error messages in the UI.
  • Documentation

    • Updated workflow documentation with new build system targets.
    • Expanded observability and configuration guides.
  • Chores

    • Updated dependencies and build configuration.
    • Added integration tests for trace verification.

Review Change Stack

Every harness-wrapped function tags its span with iii.session.id /
iii.message.id / iii.function.id and propagates the same triple as OTel
baggage. iii-sdk's BaggageSpanProcessor materializes those attrs on every
downstream worker span, so the Developer Console TRACES tab can group
spans per user turn.

* harness/src/otel.rs: with_harness_span / with_envelope_span / with_raw_span
  wrappers, ID extraction with header-injection / control-byte guards, and
  traceparent + x-iii-message-id response-header echo.
* harness/src/lib.rs: every registered function now goes through a wrapper.
* harness/tests/trace_correlation.rs: end-to-end coverage including the
  allowlist lock-step with iii-sdk's DEFAULT_ALLOWLIST.
* hook-fanout: span attribute emission + quiescence-based exit.
* provider-anthropic / provider-openai: capture_otel_context across
  tokio::spawn so HTTP child spans inherit baggage.
* harness/Makefile: new `build` target compiles each local worker and
  symlinks the release binary into ~/.iii/workers/. `make all` now runs
  build + config + observability + engine + verify.
* harness/web/src/bridge.ts: surface engine error messages instead of
  collapsing to "[object Object]".
@coderabbitai

coderabbitai Bot commented May 15, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

This PR implements OpenTelemetry tracing integration for the harness with ID extraction and header injection, updates provider clients to preserve tracing context across async boundaries, adds quiescence-based collection termination to hook-fanout, improves web client error handling, and updates build orchestration with local worker symlink provisioning.

Changes

OpenTelemetry Tracing Infrastructure

Layer / File(s) Summary
OTel module with ID extraction and span wrapping
harness/src/otel.rs
New module defines WildcardMode/IdSource enums and HarnessIds struct for trace correlation. Implements extract_ids with source-specific extraction, safety validation (length/control-byte filtering), and wildcard precedence. Provides span-wrapping entrypoints (with_harness_span, with_envelope_span, with_raw_span) that create OTel spans, record attributes when tracing is active, propagate IDs via baggage, and inject traceparent/x-iii-message-id headers into envelope responses. Includes comprehensive unit tests for ID extraction precedence, wildcard handling, header merge behavior, and error tagging.
Harness handler instrumentation with spans
harness/src/lib.rs
Exposes otel module and wraps harness HTTP handlers (harness::status, harness::call, harness::info), UI subscription handlers (ui::subscribe, ui::unsubscribe), and harness::fs::read_inline with OTel spans via with_harness_span, using different IdSource/WildcardMode settings per handler and optionally extracting inner_function_id for harness::call span identification.
Trace correlation integration tests
harness/tests/trace_correlation.rs
Integration test suite against live engine with environment-controlled skip/panic gating. Verifies emitted correlation headers (x-iii-message-id echo behavior, traceparent when observability active), trace discovery via engine::traces::list and engine::traces::tree, and error span recording. Includes unit tests for skip/panic logic and OTel baggage allowlist validation.
Tracing and observability documentation
harness/README.md
Comprehensive OTel documentation including span tagging rules, HTTP header semantics, baggage propagation behavior, TRACES grouping limitations, envelope vs raw return-shape contract, and wildcard subscription semantics for session_id: null.

SDK Dependency Updates and Provider Tracing

Layer / File(s) Summary
SDK dependency version updates
harness/Cargo.toml, hook-fanout/Cargo.toml, provider-anthropic/Cargo.toml, provider-openai/Cargo.toml
Updates iii-sdk from =0.11.3 to =0.11.7-next.3 across all Cargo manifests; adds uuid crate with v4 feature to harness.
Anthropic provider OTel context and span instrumentation
provider-anthropic/src/lib.rs, provider-anthropic/crates/provider-base/src/openai_compat.rs
Captures OTel context before spawning stream_inner and attaches it to the task. Request construction and pre-HTTP marshaling wrapped in iii_sdk::run_in_span("anthropic.request.build"). HTTP execution switched to iii_sdk::execute_traced_request. SSE consumption wrapped in iii_sdk::run_in_span("anthropic.stream.consume") with early_return flag for error handling.
OpenAI provider OTel context and span instrumentation
provider-openai/crates/provider-base/src/openai_compat.rs
Captures OTel context before spawning stream_inner and attaches it. Request preparation (JSON body, client build, headers, request object) wrapped in openai.request.build span. SSE consumption wrapped in openai.stream.consume span with early_return boolean to exit on stream-read errors before final message construction.

Web Client Bridge Updates and Error Handling

Layer / File(s) Summary
App.tsx message ID tracking with harness bridge
harness/web/src/App.tsx
send() now generates unique crypto.randomUUID() per message and routes via bridge("harness::call", ...) instead of direct run::start. Passes message_id and session_id in bridge envelope while preserving run::start payload structure (provider, model, history, approval_required, cwd).
Bridge error normalization and test coverage
harness/web/src/bridge.ts, harness/web/src/bridge.test.ts
Adds toBridgeError(e, functionId) to normalize thrown values into BridgeError by extracting message from common engine error fields (message, error, error_message, description, error_id) and falling back to JSON.stringify to avoid "[object Object]". Includes Vitest test suite covering message preservation, field extraction, JSON fallback, and primitive/empty-string handling.

Hook-Fanout Quiescence and Collection Exit

Layer / File(s) Summary
Quiescence configuration and defaults
hook-fanout/src/config.rs
WorkerConfig adds quiescence_ms: u64 field with serde default wiring. Introduces default_quiescence_ms() function; Default impl updated to set quiescence_ms. Tests validate empty-YAML defaults and custom YAML overrides.
Collection exit decision logic with quiescence and precedence
hook-fanout/src/handler.rs
publish_collect now parses expected_replies and quiescence_ms from payload, tracks first/last reply timestamps and publish_failed flag. Uses new decide_exit helper implementing precedence: expected_replies fires immediately, quiescence after at least one reply and idle window (disabled when quiescence_ms == 0), then deadline vs deadline_no_replies depending on reply history. Sets span attributes including exit_reason and conditional publish_failed. Includes comprehensive unit tests for all exit branches.

Build System, Local Worker Provisioning, and Documentation

Layer / File(s) Summary
Gitignore patterns
.gitignore
Adds .serena/ and docs/ directory ignore patterns.
Makefile build and observability targets
harness/Makefile
Introduces WORKERS_REPO, WORKERS_BIN path variables and LOCAL_WORKERS list. Adds build target that loops over local workers, runs cargo build --release per crate, and symlinks binaries into ~/.iii/workers. Adds observability target that idempotently registers iii-observability worker via iii worker add. Updates ensure-dirs to create workers symlink directory. Changes all target to build config observability engine verify.
Architecture and README demo workflow documentation
harness/ARCHITECTURE.md, harness/README.md
ARCHITECTURE.md replaces scripts/demo.sh documentation with Makefile target listing. README.md top-level description updated to document harness HTTP bridge and new bridge::trigger/bridge::events functions with usage intent.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • iii-hq/workers#135: Both PRs make substantial, conflicting changes to harness/Makefile's demo orchestration—main PR adds new build/observability targets (and alters all), while retrieved PR reworks/strips build-style workflows and refactors the Makefile target graph.

Suggested reviewers

  • sergiofilhowz

Poem

🐰 A rabbit hops through traces bright,
With spans and IDs shining light,
The builders now can test their way,
While workers build and bridge the day,
A quiescence waits, then moves along—
The harness sings an OTel song! 🎵

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main feature added: trace correlation through OpenTelemetry baggage and iii.* span attributes, which aligns with the substantial refactoring across harness and provider modules.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/harness-traces

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

Copy link
Copy Markdown
Contributor

skill-check — worker

0 verified, 25 skipped (no docs/).

Layer Result
structure
vale
ai

Three for three. Nicely done.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
harness/tests/trace_correlation.rs (1)

304-312: 💤 Low value

Optional: extract find_named helper to reduce duplication.

The find_named function is defined identically in two test functions (lines 304-312 and 421-429). Consider extracting it to a test helper module to keep the test suite DRY.

♻️ Example extraction

Add to the test module before the test functions:

#[cfg(test)]
fn find_named<'a>(node: &'a Value, name: &str) -> Option<&'a Value> {
    if node["name"].as_str() == Some(name) {
        return Some(node);
    }
    node["children"]
        .as_array()
        .and_then(|cs| cs.iter().find_map(|c| find_named(c, name)))
}

Then remove the #[allow(clippy::items_after_statements)] blocks and the duplicate definitions.

Also applies to: 421-429

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@harness/tests/trace_correlation.rs` around lines 304 - 312, Two identical
helper functions named find_named are duplicated in tests; extract a single test
helper and remove the duplicates. Create one shared find_named function in the
test module (e.g., placed above the test functions or in a test helper
submodule) and have both tests call that function instead of defining their own;
remove the #[allow(clippy::items_after_statements)] duplicate definitions at the
original locations (the blocks containing find_named) so the tests use the
single shared find_named implementation.
harness/README.md (1)

130-130: 💤 Low value

Minor: Markdown heading level skip.

The static analyzer flagged line 130 for jumping from h2 to h4. Consider changing #### Operator observability of the wrapper itself to ### Operator observability of the wrapper itself for proper heading hierarchy.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@harness/README.md` at line 130, Change the markdown heading "#### Operator
observability of the wrapper itself" to use a level-3 heading so the document's
hierarchy is consistent: replace "#### Operator observability of the wrapper
itself" with "### Operator observability of the wrapper itself" (look for that
exact heading text in README.md) to fix the h2→h4 skip flagged by the static
analyzer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@harness/ARCHITECTURE.md`:
- Around line 101-109: Update the fenced command block in ARCHITECTURE.md to
match the Makefile by changing the "make all" description to include "build"
(i.e., "make all  # build + config + observability + engine + verify") and add a
language tag to the fenced block (use ```bash) so markdown lint MD040 is
satisfied; locate the block containing the listed make targets and the "make
all" line and apply these two small edits.

In `@harness/Makefile`:
- Around line 71-76: The build loop that assumes target/release/$(w) may create
broken symlinks if the crate emits a different binary name; modify the Makefile
loop (the block iterating over LOCAL_WORKERS using WORKERS_REPO and WORKERS_BIN)
to, after cargo build, locate the actual executable produced under
"$(WORKERS_REPO)/$$w/target/release" (e.g. search for an executable file
matching $$w or with a $$w-containing basename), verify that this resolved
binary path exists, remove any stale symlink at "$(WORKERS_BIN)/$$w" if present,
and only then create the symlink to the real binary; if no matching binary is
found, print an error and exit non‑zero instead of creating a dangling symlink.

In `@harness/src/lib.rs`:
- Around line 113-124: The code directly extracts inner_function_id and passes
it into crate::otel::with_harness_span without applying the safety validation
used by extract_ids_for_bridge_trigger, which can allow oversized or malicious
values into the iii.function.id span attribute; fix by replacing the direct
extraction with a call to extract_ids_for_bridge_trigger to obtain validated IDs
(including the function_id) and then invoke run_in_span_with_ids (instead of
with_harness_span) so the span is created with the validated IDs; update the
call sites referencing inner_function_id, with_harness_span, and ensure
run_in_span_with_ids receives the returned validated ids from
extract_ids_for_bridge_trigger.

---

Nitpick comments:
In `@harness/README.md`:
- Line 130: Change the markdown heading "#### Operator observability of the
wrapper itself" to use a level-3 heading so the document's hierarchy is
consistent: replace "#### Operator observability of the wrapper itself" with
"### Operator observability of the wrapper itself" (look for that exact heading
text in README.md) to fix the h2→h4 skip flagged by the static analyzer.

In `@harness/tests/trace_correlation.rs`:
- Around line 304-312: Two identical helper functions named find_named are
duplicated in tests; extract a single test helper and remove the duplicates.
Create one shared find_named function in the test module (e.g., placed above the
test functions or in a test helper submodule) and have both tests call that
function instead of defining their own; remove the
#[allow(clippy::items_after_statements)] duplicate definitions at the original
locations (the blocks containing find_named) so the tests use the single shared
find_named implementation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2b438df6-5788-4cfd-b9a2-893d069293dc

📥 Commits

Reviewing files that changed from the base of the PR and between 673f85b and c2bb288.

⛔ Files ignored due to path filters (4)
  • harness/Cargo.lock is excluded by !**/*.lock
  • hook-fanout/Cargo.lock is excluded by !**/*.lock
  • provider-anthropic/Cargo.lock is excluded by !**/*.lock
  • provider-openai/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (19)
  • .gitignore
  • harness/ARCHITECTURE.md
  • harness/Cargo.toml
  • harness/Makefile
  • harness/README.md
  • harness/src/lib.rs
  • harness/src/otel.rs
  • harness/tests/trace_correlation.rs
  • harness/web/src/App.tsx
  • harness/web/src/bridge.test.ts
  • harness/web/src/bridge.ts
  • hook-fanout/Cargo.toml
  • hook-fanout/src/config.rs
  • hook-fanout/src/handler.rs
  • provider-anthropic/Cargo.toml
  • provider-anthropic/crates/provider-base/src/openai_compat.rs
  • provider-anthropic/src/lib.rs
  • provider-openai/Cargo.toml
  • provider-openai/crates/provider-base/src/openai_compat.rs

Comment thread harness/ARCHITECTURE.md
Comment thread harness/Makefile
Comment thread harness/src/lib.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants