Propagate AIQ request tags to NAT spans

[AjayThorve]

Summary

• propagate resolved AIQ request tags into NAT-exported spans for HTTP, WebSocket, and async job workflows
• carry request tags into async Dask jobs so child workflow spans retain caller and access-channel context
• keep public channel tagging generic: ui, api, headless, internal, anonymous, unknown
• move request trace span injection into the API auth layer and document nat.aiq.* / nat.enduser.id span attributes

Validation

• ruff check
• ruff format --check
• python -m py_compile
• uv run pytest tests/ -q (881 passed, 2 skipped)
• uv run pytest tests/aiq_agent/async_api/test_websocket_reconnect.py -q (22 passed)

Local E2E

• verified curl /chat traces include nat.aiq.* and nat.enduser.id attributes
• verified UI websocket traces include NAT request tags after binding websocket request context

[greptile-apps]

Greptile Summary

This PR wires resolved AIQ request tags (caller type, auth transport, verified status, access channel, and optional pseudonymous identifiers) into NAT-exported workflow spans for HTTP, WebSocket, and async Dask job paths. It does so by introducing a new ContextVar in auth/request_trace.py that is populated by AuthMiddleware and scoped per-request, then consumed by an idempotent monkey-patch of SpanExporter._process_start_event that stamps each emitted NAT span with nat.*-prefixed attributes. Alongside this, the PR removes the now-redundant X-AIQ-Access-Channel UI frontend header forwarding and the skill channel value, and renames docs/env examples to reflect the actual nat.aiq.* attribute namespace.

Confidence Score: 5/5

PR is safe to merge; no P0/P1 issues found in the changed paths.

All three propagation paths (HTTP, WebSocket, Dask async) are correctly wired with appropriate context scoping. The monkey-patch is idempotent, the 7-tuple expansion in submit.py lines up with the run_agent_job signature, and the previously-flagged missing build_request_trace_tags symbol is now present. No security regressions or data-correctness bugs identified.

No files require special attention.

Important Files Changed

Filename	Overview
frontends/aiq_api/src/aiq_api/auth/request_trace.py	New module: ContextVar for request trace tags, NAT span injection via idempotent monkey-patch of SpanExporter._process_start_event, nat. prefix applied at injection time
frontends/aiq_api/src/aiq_api/auth/middleware.py	Adds public build_request_trace_tags and get_current_trace_tags helpers; AuthMiddleware now captures and scopes trace tags via request_trace_tag_context alongside user_context
frontends/aiq_api/src/aiq_api/auth/utils.py	Refactored attach_request_to_active_trace into build_request_trace_tags (returns tags) + attach_request_to_active_trace (tags + apply to spans); skill removed from TRACE_ACCESS_CHANNELS
frontends/aiq_api/src/aiq_api/jobs/runner.py	run_agent_job accepts request_trace_tags, installs span injection, and wraps exporter execution in request_trace_tag_context so Dask worker spans carry caller context
frontends/aiq_api/src/aiq_api/jobs/submit.py	_get_parent_trace_context extended to 7-tuple; get_current_trace_tags() captured at submission time and propagated to Dask worker via job_args splat
frontends/aiq_api/src/aiq_api/websocket_reconnect.py	process_workflow_request now builds and scopes request trace tags per WebSocket message using build_request_trace_tags with module-level _external_hostnames
frontends/aiq_api/tests/test_auth.py	Test data updated (starfleet/nvauth → oidc/service, skill → headless channel), no coverage gaps introduced
tests/aiq_agent/async_api/test_websocket_reconnect.py	New test test_process_workflow_request_binds_request_trace_tags verifies trace tag context is set correctly for authenticated WebSocket sessions
frontends/ui/src/app/api/chat/route.ts	Removes X-AIQ-Access-Channel: ui header forwarding from chat API proxy
frontends/ui/src/adapters/auth/constants.ts	Deleted: ACCESS_CHANNEL_HEADER constant no longer needed after frontend header forwarding removed

Sequence Diagram

sequenceDiagram
    participant Client
    participant AuthMiddleware
    participant RequestTraceCtx as request_trace ContextVar
    participant SpanExporter as NAT SpanExporter (patched)
    participant DaskWorker as Dask Worker (run_agent_job)

    Note over AuthMiddleware: install_request_trace_span_injection() on init

    Client->>AuthMiddleware: HTTP / WebSocket request
    AuthMiddleware->>AuthMiddleware: build_request_trace_tags(headers, scope, user)
    AuthMiddleware->>RequestTraceCtx: request_trace_tag_context(tags).__enter__
    AuthMiddleware->>SpanExporter: workflow emits spans via _process_start_event
    SpanExporter->>RequestTraceCtx: get_request_trace_tags()
    SpanExporter->>SpanExporter: set_attribute(nat.aiq.*, value) on span

    Note over DaskWorker: Async job path
    AuthMiddleware->>DaskWorker: submit job_args with *_get_parent_trace_context() (includes tags)
    DaskWorker->>DaskWorker: install_request_trace_span_injection()
    DaskWorker->>RequestTraceCtx: request_trace_tag_context(request_trace_tags).__enter__
    DaskWorker->>SpanExporter: workflow spans tagged with caller context

Loading

_{Reviews (2): Last reviewed commit: "generalize request trace access channels" | Re-trigger Greptile}

[AjayThorve]

Addressed the review blockers in the latest branch update:

• Added and tested the build_request_trace_tags API helper used by WebSocket auth, so websocket_reconnect.py no longer has the import-time failure.
• Moved request trace injection into the API auth package and install it from middleware instead of an agent module.
• Added debug logging for span lookup misses during request tag injection.
• Added a WebSocket regression test covering request tag context binding.
• Kept public access-channel examples generic (ui, api, headless, internal, anonymous, unknown).

Validation: uv run pytest tests/ -q passes locally with 881 passed, 2 skipped.

[cdgamarose-nv]

Looks good!