feat(request-trace): native S3 sink for request-trace records

[YiqiuLiu]

Summary

Adds a native S3 RequestTraceSink that writes existing dynamo.request.trace.v1 records directly to a bucket -- parallel to the OTLP sink, no schema change, no partitioning. Smallest reviewable unit for the S3 destination work in #11768.

• New RequestTraceSinkKind::S3 variant with S3RequestTraceSink (lib/llm/src/request_trace/s3_sink.rs), gated behind the new request-trace-s3 cargo feature so the AWS SDK is opt-in.
• Records are batched in-process as gzipped JSONL and each finished batch is uploaded via a single PutObject. Object keys use a simple time-based layout for this PR ({prefix}/{yyyy}/{mm}/{dd}/{host}-{HHMMSS}-{seq}.jsonl.gz).
• Credentials come from the AWS SDK default provider chain -- env vars, IMDS, IRSA, Pod Identity, shared profiles. How the frontend pod is credentialed is a deployment concern and is not part of this PR.
• New env vars: DYN_REQUEST_TRACE_S3_{BUCKET,REGION,PREFIX}; opt in via DYN_REQUEST_TRACE_SINKS=...,s3.

Scope split (per #11768)

This is PR 1 of 3:

1. This PR -- functional S3 sink, same schema, no layout changes.
2. Format/layout -- Hive-style model=/date=/hour= keys, tunable roll thresholds, Prometheus metrics.
3. Deploy docs -- how to provide AWS credentials to the frontend pod (IRSA / Pod Identity / static). No operator or CRD code, per feat(request-trace): native S3 sink for request-trace records #11768 explicit non-goals.

Supersedes #10903, which was pinned to the pre-ai-dynamo/dynamo#11180 audit module.

Test plan

• cargo fmt --check -p dynamo-llm
• cargo-deny -L error --all-features check licenses bans (bans ok, licenses ok)
• cargo metadata --locked --format-version 1
• cargo doc --no-deps -- 0 warnings from new files
• cargo test --locked --doc
• cargo clippy --no-deps --all-targets -- -D warnings (default features)
• cargo clippy --no-deps --all-targets --features request-trace-s3 -- -D warnings
• cargo machete -- no unused deps
• cargo test -p dynamo-llm --lib --features request-trace-s3 request_trace -- 66/66 pass, including 4 new s3_sink tests
• MinIO integration test -- deferred; will add as a follow-up commit if reviewers want it in this PR

Non-goals (deferred to PRs 2/3 per #11768)

• Rich object-key partitioning (model/day/hour).
• Sampling.
• New record schema fields.
• Any operator/CRD/IAM changes.

---

Summary by CodeRabbit

• New Features
  • Added optional S3-backed request tracing.
  • Request traces are uploaded as compressed, batched JSONL files with organized time-based paths.
  • Added configuration for S3 bucket, region, and object prefix.
  • Added safeguards for buffering, shutdown, and upload failures.
• Documentation
  • Documented the new S3 request-tracing configuration settings.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019fb3e6-5549-72fb-8223-be660d433d68.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 1 potential issue.

[coderabbitai]

Walkthrough

Adds an optional S3 request-trace sink with environment-based configuration, feature-gated AWS dependencies, asynchronous JSONL batching, gzip compression, S3 uploads, shutdown handling, and key-formatting tests.

Changes

S3 request-trace sink

Layer / File(s)	Summary
S3 feature and configuration contracts lib/llm/Cargo.toml, lib/llm/src/request_trace/config.rs, lib/runtime/src/config/environment_names.rs	Adds the optional S3 feature and AWS dependencies, S3 policy fields, environment variables, and s3 sink parsing.
S3 sink worker and object uploads lib/llm/src/request_trace/s3_sink.rs	Adds bounded asynchronous delivery, JSONL batching, gzip compression, shutdown draining, UTC-partitioned S3 keys, uploads, and unit tests.
Feature-gated sink construction lib/llm/src/request_trace/mod.rs, lib/llm/src/request_trace/sink.rs	Loads the S3 sink module conditionally and constructs the sink when the feature is enabled, otherwise returning an error.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it misses the required template sections for reviewer start and related issues.	Add the required Overview, Details, Where should the reviewer start?, and Related Issues sections, including either a linked issue or the no-issue confirmation.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and clearly describes the main change: adding a native S3 request-trace sink.

---

_{Comment @coderabbitai help to get the list of available commands.}

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 39.89% (-9.13%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fedc2da | Docs | Datadog PR Page | Give us feedback!}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/llm/src/request_trace/s3_sink.rs`:
- Around line 6-9: Update the module documentation describing S3 object keys to
match the implemented format `{host}-{HHMMSS}-{run_id}-{seq}.jsonl.gz`, while
preserving the existing prefix and date-partition layout.
- Around line 155-168: In the shutdown branch of the request-trace worker, close
the receiver before draining pending records, then consume records with recv()
until it returns None instead of using try_recv(). Preserve the existing
serialization warning and final upload_ready_batch behavior, ensuring emits
racing with shutdown cannot enqueue records after draining begins.

In `@lib/runtime/src/config/environment_names.rs`:
- Around line 549-559: Update the documentation for DYN_REQUEST_TRACE_SINKS to
include s3 in its supported sink values and mapping description, while
preserving the existing documented values and behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2b3ab66d-b65a-4749-8383-7aae911a3709

📥 Commits

Reviewing files that changed from the base of the PR and between 92ec76b and 03d2e23.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (6)

• lib/llm/Cargo.toml
• lib/llm/src/request_trace/config.rs
• lib/llm/src/request_trace/mod.rs
• lib/llm/src/request_trace/s3_sink.rs
• lib/llm/src/request_trace/sink.rs
• lib/runtime/src/config/environment_names.rs

[tmonty12]

/ok to test f5738a0

[nnshah1]

/ok to test c3a50dc

[dagil-nvidia]

/ok to test 2daa3f1

[dagil-nvidia]

/ok to test 380f0c0

[nv-anants]

/ok to test ef0241e

[nv-anants]

/ok to test 9b417aa

[nv-anants]

/ok to test 34c5379

[dynamo-ops]

/ok to test 34c5379

[nealvaidya]

/ok to test d4a7a9f

[nv-anants]

/ok to test fedc2da

[dynamo-ops]

/ok to test fedc2da