Skip to content

fix(bench): bypass frontend-proxy, drive frontend directly (PR-N3.3) - #76

Merged
jensholdgaard merged 1 commit into
mainfrom
fix/otel-demo-bypass-envoy
May 31, 2026
Merged

fix(bench): bypass frontend-proxy, drive frontend directly (PR-N3.3)#76
jensholdgaard merged 1 commit into
mainfrom
fix/otel-demo-bypass-envoy

Conversation

@jensholdgaard

@jensholdgaard jensholdgaard commented May 31, 2026

Copy link
Copy Markdown
Owner

What

Removes frontend-proxy (Envoy) from the OTel Demo capture path. Follow-up to run 26716710757, which failed the readiness gate.

Root cause (and a re-read of the first capture)

Envoy fails its bootstrap proto validation in CI:

Clusters[9] ... SocketAddress.Address: value length must be at least 1 characters

The published frontend-proxy image (demo 2.2.0) carries a baked Envoy config with a 10th cluster whose address env var the demo doesn't set — so Envoy crash-loops. The source tag's envoy.tmpl.yaml has only 9 clusters, all with vars set in .env, and the image is distroless (no shell to inspect the baked template), so the empty var can't be pinned from outside.

That crash-loop reframes the first capture: its ~34% ERR_NAME_NOT_RESOLVED flood wasn't a startup race — it was intermittent DNS for a restarting frontend-proxy. PR-N3.2's readiness probe correctly caught it instead of silently capturing a broken corpus.

Fix — bypass Envoy

Per maintainer call, drive the frontend HTTP API directly instead of debugging a distroless image's baked config:

  • Compose override sets, on load-generator:

    • LOCUST_BROWSER_TRAFFIC_ENABLED=false
    • LOCUST_HOST=http://frontend:${FRONTEND_PORT}

    (environment merges by key, overriding the base's hardcoded =true and the .env LOCUST_HOST.)

  • The frontend still fans out to cart / checkout / currency / product-catalog / payment / …, so business-service log diversity still flows — the corpus just becomes API-driven rather than browser-driven.

  • Readiness now gates on otel-collector + frontend running. Frontend has no healthcheck and only an ephemeral host port, so "running" + the post-restart settle + the existing steady-state slice (which discards early ramp errors) is the readiness story.

  • The crash-looping frontend-proxy is stopped so it doesn't burn runner CPU (nothing depends on it — it fronts the frontend, not the reverse).

Verification

  • All three YAML files validate; step flow unchanged.
  • No Rust changes. Real test is the next dispatch — I'll report whether the business services now dominate the corpus.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Updated OpenTelemetry demo corpus capture configuration to route load-generator traffic directly to the frontend service
    • Disabled browser traffic simulation in load generation
    • Simplified workflow readiness checks and service startup orchestration

The published frontend-proxy image (demo 2.2.0) carries a baked
Envoy bootstrap with a 10th cluster whose address env var the
demo doesn't set, so Envoy fails proto validation and
crash-loops in CI. That flapping container — not a startup race
— is what gave the first capture its ERR_NAME_NOT_RESOLVED flood
(intermittent DNS for a restarting container). The source tag's
envoy.tmpl.yaml has only 9 clusters all with set vars, and the
image is distroless (no shell to inspect the baked template), so
the empty var can't be identified from outside.

Rather than reverse-engineer the image, remove Envoy from the
path: the compose override sets LOCUST_BROWSER_TRAFFIC_ENABLED=
false + LOCUST_HOST=http://frontend: so the load
generator drives the frontend's HTTP API directly. The frontend
still fans out to cart / checkout / currency / product-catalog /
…, so business-service log diversity still flows. Readiness now
gates on otel-collector + frontend running (frontend has no
healthcheck and only an ephemeral host port, so the post-restart
settle + steady-state slice absorb early ramp errors), and the
crash-looping frontend-proxy is stopped to free runner CPU.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jensholdgaard
jensholdgaard requested a review from Copilot May 31, 2026 15:54
@coderabbitai

coderabbitai Bot commented May 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 41d7b528-e207-440e-9cdb-26ac48035099

📥 Commits

Reviewing files that changed from the base of the PR and between e3fa3ad and 8de2f3b.

📒 Files selected for processing (2)
  • .github/otel-demo-capture-compose-override.yml
  • .github/workflows/capture-otel-demo-corpus.yml

📝 Walkthrough

Walkthrough

Updated the OTel demo corpus capture by configuring load-generator to bypass the frontend-proxy/Envoy path and send traffic directly to the frontend service. The Docker Compose override now sets load-generator environment variables, and the workflow startup script no longer waits on Envoy's readiness endpoint; it instead polls service status and manages service startup order directly.

Changes

OTel Demo Corpus Capture - Bypass Envoy Path

Layer / File(s) Summary
Compose override - load-generator direct routing configuration
.github/otel-demo-capture-compose-override.yml
Documentation expanded to explain why load-generator bypasses frontend-proxy/Envoy. Added load-generator service environment overrides: disables LOCUST_BROWSER_TRAFFIC_ENABLED and sets LOCUST_HOST to route traffic directly to the frontend service using ${FRONTEND_PORT}.
Workflow startup orchestration - skip Envoy and reorder services
.github/workflows/capture-otel-demo-corpus.yml
Removed ENVOY_ADMIN_PORT from workflow environment. Refactored the readiness script to use docker compose ps polling for otel-collector and frontend running status instead of curl-based Envoy /ready endpoint checks. Script now stops frontend-proxy to prevent crash-looping and restarts load-generator after frontend reaches running state.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • jensholdgaard/ourios#74: Updates the same OTel demo capture files with bind-mounted capture directory coordination for otel-collector output.
  • jensholdgaard/ourios#73: Modifies the same capture workflow by changing how compose services are waited on and orchestrated during the capture run.
  • jensholdgaard/ourios#75: Adds frontend-proxy readiness gating and restart logic to the workflow—directly conflicts with this PR's Envoy bypass and load-generator routing changes.

🐰 In captures old and new,
We skip the proxy's view,
Load runs to frontend, true and bright,
*No Envoy gates—just traffic's flight! 🚀

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description provides a comprehensive explanation of the 'What' and 'Why' but is missing most of the template's checklist items (cargo fmt/clippy, tests, docs/CHANGELOG updates, RFC links). Complete the checklist by confirming or documenting cargo fmt/clippy status, whether tests were added, and CHANGELOG.md updates.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and specifically describes the main change: bypassing the frontend-proxy and driving the frontend directly, which is the central objective of this PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/otel-demo-bypass-envoy

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the OTel Demo corpus capture workflow to bypass the crash-looping frontend-proxy/Envoy path and drive the demo frontend directly, preserving business-service log generation for benchmark corpus capture.

Changes:

  • Removes the Envoy admin readiness probe and instead gates on otel-collector and frontend reaching running.
  • Stops frontend-proxy after startup to avoid CI runner churn.
  • Overrides load-generator settings to disable browser traffic and target frontend directly.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.github/workflows/capture-otel-demo-corpus.yml Updates readiness and load-generator restart flow for direct frontend capture.
.github/otel-demo-capture-compose-override.yml Adds load-generator environment overrides to bypass frontend-proxy.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@jensholdgaard
jensholdgaard merged commit 22ac807 into main May 31, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants