fix(bench): filter collector + load-generator noise from demo corpus (PR-N3.4)

[jensholdgaard]

What

Drops the two non-application noise sources from the demo corpus via a dedicated, filtered capture pipeline. Branch-validated before this PR (see results below).

Why

The first healthy capture (post-Envoy-bypass) was ~53% non-application noise: otelcol-contrib self-telemetry (the collector's debug-exporter "Logs"/"Metrics"/"Traces" lines, ~32%) + load-generator locust operational logs (~22%). That dilutes the business-service signal and would skew A1 toward a handful of repetitive infra templates.

How

A dedicated logs/corpus pipeline in the collector overlay, sharing the base otlp receiver, with a filter processor that drops records whose service.name is otelcol-contrib or load-generator, feeding only file/corpus. The demo's own logs pipeline is untouched.

Branch validation (run 26718134959, 900s window)

Filter confirmed working — otelcol-contrib = 0, load-generator = 0:

	pre-filter	post-filter (this PR)
records	7,420	3,395
distinct templates	320	241
infra noise	53%	0%
bytes	5.76 MB	2.87 MB

Now 100% application/service logs: product-reviews 1455, kafka 661, shipping 472, cart 371, ad 168, recommendation 150, currency 73, checkout 45. Top templates are genuine parameterized app logs — GetCartAsync called with userId={userId}, AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}, Convert conversion successful, connection-retry backoff sequences, kafka segment management — exactly what the template miner is built to collapse.

Notes

• No Rust changes; extras YAML validates. Workflow itself unchanged.
• Corpus is still modest (3,395 records / 2.87 MB); a longer window scales it. But it's now clean signal.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added a dedicated logs capture pipeline with filtering to exclude infrastructure noise while preserving application service logs for analysis.

• Changes

  • Existing demo logging configuration remains unaffected by these updates.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aa5d85c7-b4b0-45a4-b903-eaa15d5d1c38

📥 Commits

Reviewing files that changed from the base of the PR and between 22ac807 and bcaad09.

📒 Files selected for processing (1)

• .github/otel-demo-capture-extras.yml

---

📝 Walkthrough

Walkthrough

This PR modifies the OpenTelemetry Collector overlay configuration to add a dedicated logs/corpus pipeline with filtering. It separates corpus log capture into its own pipeline, introduces a filter processor to drop infrastructure noise, and preserves the demo's original logs pipeline behavior.

Changes

Logs Corpus Capture Pipeline

Layer / File(s)	Summary
Documentation and pipeline intent .github/otel-demo-capture-extras.yml	Comments updated to describe the new dedicated logs/corpus pipeline approach instead of modifying the demo's existing logs pipeline.
Filter processor and file exporter configuration .github/otel-demo-capture-extras.yml	filter/corpus processor defined to drop logs from otelcol-contrib and load-generator services; file/corpus exporter configured for JSONL output to /capture/logs.raw.jsonl with 1-second flush interval.
Pipeline wiring and receivers .github/otel-demo-capture-extras.yml	service.pipelines["logs/corpus"] wired to receive from shared otlp receiver, apply memory_limiter and filter/corpus processors, and export to file/corpus.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• jensholdgaard/ourios#73: Updates .github/otel-demo-capture-extras.yml to modify the OpenTelemetry Demo collector's logs capture via an adjusted logs pipeline with file/corpus exporter wiring.
• jensholdgaard/ourios#75: Directly connected modifications to .github/otel-demo-capture-extras.yml introducing exporters.file/corpus and adjusting logs pipeline wiring for the same capture flow.
• jensholdgaard/ourios#58: Adds corpus loader support for ingesting OTLP File Exporter JSONL output, matching the format produced by this PR's file/corpus exporter (/capture/logs.raw.jsonl).

Poem

🐰 A corpus pipeline takes its shape,
Filters out the infra noise—escape!
Logs flow clean through JSONL streams,
While the demo's peace remains in dreams.
Hop along, fresh capture gleams! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The description covers key sections (Summary, Related, branch validation) but is missing the explicit Checklist section with completion status for required tasks like cargo fmt/clippy, tests, and documentation updates.	Add a completed Checklist section documenting whether cargo fmt/clippy were run, tests were added, and CHANGELOG.md was updated as per the template.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main change: adding a filter to remove infrastructure noise (collector and load-generator logs) from the demo corpus.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/otel-demo-trim-infra-noise

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR refines the OTel Demo corpus capture overlay so the benchmark corpus excludes collector/load-generator operational noise while leaving the demo’s normal logs pipeline untouched.

Changes:

• Adds a filter/corpus processor to drop otelcol-contrib and load-generator log records by service.name.
• Introduces a dedicated logs/corpus pipeline that shares the OTLP receiver and writes only filtered records to file/corpus.
• Updates comments to document the capture architecture and filtering rationale.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.