fix(bench): gate demo capture on frontend-proxy readiness + slice steady state (PR-N3.2)

[jensholdgaard]

What

Makes the OTel Demo capture produce a healthy corpus. Follow-up to the first successful capture (run 26715857483), which ran but yielded a skewed corpus.

The problem with the first corpus

6,004 records / ~485 templates / 12 services — real diversity, but:

• ~34% was the load generator's ERR_NAME_NOT_RESOLVED against frontend-proxy, just two templates repeated ~2,030×.
• The actual business services barely emitted: checkout 14, cart 11, payment 1, currency 1.

Root cause (from the run logs + the demo compose): the load-generator only depends_on: frontend (service_started, not healthy), frontend-proxy (Envoy — the host the load generator actually targets via LOCUST_HOST) has no healthcheck, and our readiness gate waited only for otel-collector. The load-generator even started 0.4s before frontend-proxy. So its Playwright browser tasks hammered Envoy before it was resolvable/serving and flooded the corpus with one error, while real traffic never ramped.

Fix

• Gate on the real traffic path: after the collector is up, poll Envoy's published admin /ready endpoint (localhost:${ENVOY_ADMIN_PORT}) until it returns 200 — the true "frontend-proxy is serving" signal. On timeout, dump compose ps + frontend-proxy logs.
• Restart the load-generator once Envoy is ready, so its tasks run fresh against a resolvable proxy instead of replaying the startup-race failures.
• Steady-state slice: the collector now writes logs.raw.jsonl; after a post-restart settle the workflow marks the raw line offset and slices only the post-settle window into the final logs.jsonl, deterministically discarding all startup/pre-restart noise. Raw file is uploaded too for debugging.

All ~20 demo containers came up fine on the runner last time, so this is purely a readiness/ordering fix — not a resource problem.

Verification

• Both YAML files validate; step flow confirmed (bring up → wait traffic path → capture window → slice + verify → tear down → manifest → upload).
• No Rust changes. The real test is the next dispatch after merge — I'll report the new corpus's diversity (and whether the business services now dominate over infra noise).

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• Chores
  • Updated OpenTelemetry Collector configuration to capture raw logs to file.
  • Improved demo startup reliability with explicit readiness checks for the proxy.
  • Enhanced log capture workflow to record both raw and processed log artifacts.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ccfc6625-4d55-4bb3-8f08-4559ea75c0c0

📥 Commits

Reviewing files that changed from the base of the PR and between b08b177 and bcb6622.

📒 Files selected for processing (2)

• .github/otel-demo-capture-extras.yml
• .github/workflows/capture-otel-demo-corpus.yml

---

📝 Walkthrough

Walkthrough

This PR enhances the OTel Demo corpus capture workflow by adding raw log export via a new file-based exporter, implementing Envoy readiness probing to avoid startup races, and refactoring the capture logic to use offset-based slicing of steady-state logs with validation.

Changes

OTel Demo Corpus Capture Workflow

Layer / File(s)	Summary
Collector raw log exporter configuration .github/otel-demo-capture-extras.yml	Adds file/corpus exporter configured to write raw logs to /capture/logs.raw.jsonl in JSON format with 1-second flush interval; updates logs pipeline to include this exporter alongside demo defaults.
Workflow environment and readiness gates .github/workflows/capture-otel-demo-corpus.yml	Pins Envoy admin port via ENVOY_ADMIN_PORT environment variable; extends startup sequence with wait_for helper, otel-collector readiness check, frontend-proxy admin /ready polling, and load-generator restart to ensure service dependencies are met before capture begins.
Capture offset recording and log slicing .github/workflows/capture-otel-demo-corpus.yml	Records initial file offset from logs.raw.jsonl after warmup, captures for configured duration, stops otel-collector, validates raw capture exists and is non-empty, slices logs written after offset into logs.jsonl with validation; updates artifact uploads to include both logs.jsonl and logs.raw.jsonl.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• jensholdgaard/ourios#73: Both PRs modify the same OTel Demo collector overlay (.github/otel-demo-capture-extras.yml) to add file/corpus and replace the logs pipeline exporters, and they both target the same capture workflow (.github/workflows/capture-otel-demo-corpus.yml)—with the main PR extending/adjusting the capture behavior and artifacts from the earlier version.
• jensholdgaard/ourios#74: Both PRs modify the OTel Demo capture configuration to write the file/corpus exporter output into a mounted /capture directory (and adjust the capture workflow accordingly), so the main PR's new logs.raw.jsonl capture is directly connected to #74's bind-mount + exporter-path changes.

Poem

🐰 A rabbit hops through logs so raw,
Offset slicing—nature's law!
With readiness probes standing guard,
Corpus captured, not too hard.
Raw and cooked, both safe and sound! 📋✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description comprehensively explains the problem, root cause, fixes, and verification, but lacks the standard checklist section (cargo fmt, clippy, tests, docs, RFC).	Add the standard checklist section with items for cargo fmt, clippy, tests, docs/CHANGELOG updates, and RFC links to match the repository template.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main changes: gating demo capture on frontend-proxy readiness and slicing steady-state logs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/otel-demo-health-readiness

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Fixes the OTel Demo corpus capture workflow so the captured corpus reflects real service diversity rather than load-generator startup-race errors. After the collector is up, the workflow now actively probes the frontend-proxy (Envoy) admin /ready endpoint, restarts the load generator once Envoy is serving, and then slices a steady-state window out of a raw capture file by line offset for the final logs.jsonl.

Changes:

• Add an ENVOY_ADMIN_PORT env and a new "Wait for the traffic path to be ready" step that polls Envoy /ready and then restarts load-generator.
• Switch the file exporter to write logs.raw.jsonl, mark a line offset after warmup, and slice the post-settle window into logs.jsonl in a dedicated step (also splits teardown into an if: always() step and uploads the raw file as artifact).
• Update collector extras to write to logs.raw.jsonl with an explanatory comment.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
.github/workflows/capture-otel-demo-corpus.yml	Adds Envoy readiness gating, load-generator restart, steady-state slice + verify, separate teardown, and raw-file artifact upload.
.github/otel-demo-capture-extras.yml	Renames the file exporter output to logs.raw.jsonl so the workflow can slice the steady-state window from it.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.