Skip to content

ci: end-to-end deploy smoke test on an ephemeral kind cluster - #318

Merged
jensholdgaard merged 2 commits into
mainfrom
ci/deploy-smoke-test-kind
Jun 30, 2026
Merged

ci: end-to-end deploy smoke test on an ephemeral kind cluster#318
jensholdgaard merged 2 commits into
mainfrom
ci/deploy-smoke-test-kind

Conversation

@jensholdgaard

@jensholdgaard jensholdgaard commented Jun 30, 2026

Copy link
Copy Markdown
Owner

What

A nightly / on-demand end-to-end deploy smoke test: spin up an ephemeral kind cluster, deploy the Ourios Helm chart against an in-cluster LocalStack (the chart's S3 production path), then exercise the real path — push an OTLP/JSON log batch at the receiver and read it back through the querier.

This is the "see the chart actually run" validation. The old Talos cluster is gone and there's no persistent cluster, so CI is the deploy target for now; managed K8s (AKS/GKE) + bare metal are a later infra workstream.

Flow

  1. Build the ourios-server image from the checkout, kind load it (no registry pull).
  2. Deploy LocalStack (S3) in-cluster + create the bucket.
  3. helm install with backend=s3, endpoint→LocalStack, creds Secret, image.pullPolicy=Never.
  4. Wait for receiver / querier / compactor rollout.
  5. Ingest an OTLP/JSON log (service.name=ourios-smoke) at :4318/v1/logs.
  6. Force flush: rollout restart the receiver — records buffer in the sink until a size/age (300 s) flush, and graceful shutdown drains the sink to S3.
  7. Query :4319/v1/query (service == "ourios-smoke", X-Ourios-Tenant: ourios-smoke) and assert rows >= 1.

Notes

  • On-demand + nightly, not per-PR (kind boot + image build + deploy is too heavy to gate every PR; matches the bench/fuzz posture).
  • Tenant is derived from the service.name resource attribute, so the ingested service name and the query tenant header must match.
  • Asserts on row count, not body text — body reconstruction is gated on querier returns empty body for ingested clean rows — receiver doesn't persist miner template audit events #302 (clean rows return empty bodies until the receiver persists template_created audit events).
  • Actions SHA-pinned; permissions: contents: read; failure step dumps pod/describe/logs.
  • Validated locally with actionlint + helm lint + helm template (renders, names match). The real run can only be exercised in CI — validating via workflow_dispatch on this branch before merge.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Added a deploy smoke test GitHub Actions workflow that runs nightly and supports manual runs.
    • Validates Helm linting, builds the Docker image, deploys an ephemeral in-cluster setup, ingests a sample log batch, and confirms log retrieval returns results.
  • Chores
    • Improved failure diagnostics by collecting key cluster and application logs/status (pods and LocalStack) when the smoke test fails.

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@jensholdgaard, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 47 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a6b24fc3-b77d-41e6-8a6a-b915d4a37806

📥 Commits

Reviewing files that changed from the base of the PR and between 9c52fc5 and 2865683.

📒 Files selected for processing (1)
  • .github/workflows/deploy-test.yml
📝 Walkthrough

Walkthrough

Adds .github/workflows/deploy-test.yml, a new GitHub Actions workflow that runs on a nightly cron and manual dispatch. It provisions a kind cluster, deploys LocalStack S3 and the Ourios Helm chart, ingests an OTLP log batch, restarts the receiver to flush buffers, queries logs back, and fails if no rows are returned. Failure diagnostics collect pod and component logs.

Changes

Deploy Smoke Test Workflow

Layer / File(s) Summary
Workflow triggers, env vars, and cluster setup
.github/workflows/deploy-test.yml
Defines nightly/manual triggers, permissions, and env vars; adds steps for Helm lint, Docker image build, kind cluster provisioning, LocalStack S3 deployment, S3 credentials Secret creation, and Ourios Helm install with timed wait.
Ingest, flush, query validation and failure diagnostics
.github/workflows/deploy-test.yml
Waits for receiver/querier/compactor rollouts, ingests OTLP/JSON logs via port-forward (fails on non-200), restarts receiver StatefulSet to force buffer flush, queries logs back with tenant header retrying until a row is returned, and collects pod descriptions and component logs on workflow failure.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐇 Hoppity-hop through the pipeline we go,
A kind little cluster all set up to grow,
LocalStack wakes, logs flow like a stream,
Query returns rows — it's more than a dream!
The rabbit checks rows, then leaps with delight,
The smoke test is green, all systems are right! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is detailed, but it does not follow the required template and omits the Summary, Related, and Checklist sections. Rewrite the PR description using the repository template, including Summary, Related, and a completed Checklist.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the new ephemeral kind-based deploy smoke test.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/deploy-smoke-test-kind

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a scheduled/on-demand GitHub Actions workflow that performs an end-to-end “deploy + ingest + query” smoke test by standing up an ephemeral kind cluster, deploying the Ourios Helm chart against an in-cluster LocalStack S3 backend, and validating the real OTLP ingest/query path.

Changes:

  • Introduces a new deploy-test.yml workflow that builds and kind-loads an ourios-server image, deploys LocalStack, and installs the Helm chart configured for S3.
  • Exercises the running deployment by ingesting OTLP/JSON logs via the receiver and querying them back via the querier, with failure diagnostics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/deploy-test.yml
Comment thread .github/workflows/deploy-test.yml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/workflows/deploy-test.yml (1)

134-136: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Receiver label selector matches two Services; pin the target instead.

app.kubernetes.io/component=receiver matches both the client-facing ourios-receiver and the headless ourios-receiver-headless Service (see deploy/helm/ourios/templates/services.yaml). items[0] then depends on kubectl's name ordering. Both happen to expose otlp-http:4318 so it works today, but selecting the headless Service for a port-forward is fragile. Since the release name is fixed, target the client Service by name.

♻️ Pin the receiver Service
-          recv="$(kubectl get svc -l app.kubernetes.io/component=receiver -o jsonpath='{.items[0].metadata.name}')"
-          kubectl port-forward "svc/${recv}" 4318:4318 >/tmp/pf-recv.log 2>&1 &
+          kubectl port-forward svc/ourios-receiver 4318:4318 >/tmp/pf-recv.log 2>&1 &
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/deploy-test.yml around lines 134 - 136, The receiver
Service selection in the deploy-test workflow is ambiguous because the label
selector can return both the client-facing and headless Services. Update the
port-forward setup in the workflow step that assigns recv and runs kubectl
port-forward to target the client Service by its fixed name instead of relying
on app.kubernetes.io/component=receiver plus items[0]. Keep the rest of the trap
and port-forward logic unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/deploy-test.yml:
- Around line 161-172: The retry loop in the deploy-test workflow is being
short-circuited by bash errexit when curl or jq fails inside command
substitutions. Update the query polling block to make the `out` and `rows`
assignments tolerant of transient failures so the loop continues through all
attempts instead of exiting on the first bad response. Use the existing retry
logic around the curl/jq calls in the smoke-test step to keep failures falling
through until the final check.

---

Nitpick comments:
In @.github/workflows/deploy-test.yml:
- Around line 134-136: The receiver Service selection in the deploy-test
workflow is ambiguous because the label selector can return both the
client-facing and headless Services. Update the port-forward setup in the
workflow step that assigns recv and runs kubectl port-forward to target the
client Service by its fixed name instead of relying on
app.kubernetes.io/component=receiver plus items[0]. Keep the rest of the trap
and port-forward logic unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6198f0ef-de7c-4fb8-bbd8-023819b87643

📥 Commits

Reviewing files that changed from the base of the PR and between 3b815e4 and 9cbaaf7.

📒 Files selected for processing (1)
  • .github/workflows/deploy-test.yml

Comment thread .github/workflows/deploy-test.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/deploy-test.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/deploy-test.yml Outdated
Deploy the Helm chart to a kind cluster against an in-cluster LocalStack
(the chart's S3 production path), then exercise the real path: push an
OTLP/JSON log batch at the receiver and read it back through the querier.

No persistent cluster exists, so CI is the deploy target (managed K8s +
bare metal come later). On-demand + nightly (not per-PR): a kind boot +
image build + deploy is too heavy to gate every PR.

Notes: tenant == the service.name resource attribute, so ingest and the
query's X-Ourios-Tenant match; ingested records buffer in the sink until a
size/age (300s) flush, so the test restarts the receiver to force the
graceful-shutdown drain to S3 before querying; the image is built from the
checkout and kind-loaded (no registry pull). Port-forward targets the
explicit ClusterIP services (the component label also matches the headless
receiver service). The curl loops tolerate transient failures (bash -e) so
retries work. Actions + the LocalStack image are pinned; least-privilege.

A temporary path-scoped pull_request trigger validates this on its own PR
(workflow_dispatch can't target a non-default branch); removed before merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/deploy-test.yml Outdated
Validated end-to-end on this PR (kind deploy + OTLP ingest → query, rows=1)
with the explicit ClusterIP service targets, errexit-tolerant retry loops,
and pinned LocalStack. Revert to on-demand + nightly only — not a per-PR gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@jensholdgaard
jensholdgaard merged commit 05d42b0 into main Jun 30, 2026
21 checks passed
@jensholdgaard
jensholdgaard deleted the ci/deploy-smoke-test-kind branch June 30, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants