Skip to content

feat(intake): add intake clickhouse to nemo-platform helm - #998

Merged
BrianNewsom merged 6 commits into
mainfrom
brnewsom/ase-243-helm-chart-for-intake-and-clickhouse-remote-deployment-in
Jul 30, 2026
Merged

feat(intake): add intake clickhouse to nemo-platform helm#998
BrianNewsom merged 6 commits into
mainfrom
brnewsom/ase-243-helm-chart-for-intake-and-clickhouse-remote-deployment-in

Conversation

@BrianNewsom

@BrianNewsom BrianNewsom commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features
    • Added embedded single-node ClickHouse for Intake with configurable persistence, ports, and Kubernetes service/service account setup.
    • Added support for externally managed ClickHouse (optionally secure/TLS) using an existing credentials Secret.
    • Enhanced Helm notes and API connectivity with ClickHouse URL/credentials/database details.
  • Bug Fixes
    • Updated embedded ClickHouse password default so an unset value generates a random password; tightened validation for missing external settings.
  • Documentation
    • Expanded “Intake and ClickHouse” with embedded limitations, external Secret/keys setup, and ClickHouse sizing guidance.
  • Tests
    • Added Helm template negative assertions for embedded/external ClickHouse credential configuration expectations.

@BrianNewsom
BrianNewsom requested review from a team as code owners July 30, 2026 18:57
@github-actions github-actions Bot added the feat label Jul 30, 2026
Signed-off-by: Brian Newsom <brnewsom@nvidia.com>
@BrianNewsom
BrianNewsom force-pushed the brnewsom/ase-243-helm-chart-for-intake-and-clickhouse-remote-deployment-in branch from 7d2c927 to f459811 Compare July 30, 2026 19:02
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The Helm chart now supports Intake with embedded or externally managed ClickHouse, including configuration values, Kubernetes resources, API connection wiring, documentation, installation notes, CI examples, and validation checks.

Changes

Intake ClickHouse integration

Layer / File(s) Summary
ClickHouse configuration contracts
k8s/helm/values.yaml, k8s/helm/README.md, k8s/helm/helm-docs-template/..., k8s/helm/ci/*
Defines embedded and external ClickHouse settings, credential requirements, sizing guidance, and CI configurations.
Connection resolution and Intake wiring
k8s/helm/templates/_helpers.tpl, k8s/helm/templates/api/api-deployment.yaml, k8s/helm/templates/NOTES.txt
Resolves endpoints and credentials, injects ClickHouse connection variables into the API, and emits mode-specific installation notes.
Embedded ClickHouse resources
k8s/helm/templates/clickhouse/*
Renders conditional authentication, ServiceAccount, Service, and StatefulSet resources with generated passwords and optional persistent storage.
External configuration validation
tools/lint/lint-helm.sh
Validates required external connection fields and generated embedded credential references.

Sequence Diagram(s)

sequenceDiagram
  participant HelmChart
  participant IntakeAPI
  participant EmbeddedClickHouse
  participant ExternalClickHouse
  HelmChart->>IntakeAPI: inject ClickHouse URL and credentials
  IntakeAPI->>EmbeddedClickHouse: connect when embedded mode is enabled
  IntakeAPI->>ExternalClickHouse: connect when embedded mode is disabled
Loading

Suggested reviewers: ironcommit, benmccown, crookedstorm

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding ClickHouse support for Intake in the Helm chart.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch brnewsom/ase-243-helm-chart-for-intake-and-clickhouse-remote-deployment-in

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl`:
- Around line 41-58: Document the external ClickHouse Secret prerequisite before
the values block in k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl,
including a concrete kubectl create secret generic command that creates
clickhouse-credentials with the configured password key; then regenerate
k8s/helm/README.md so the same prerequisite and command appear there.

In `@k8s/helm/templates/_helpers.tpl`:
- Around line 421-425: Update the clickhouse.passwordKey helper and generated
Secret handling so embedded ClickHouse without an existing Secret always uses
the same key that clickhouse-secret.yaml creates: return the literal password
key, or make the generated Secret use the helper’s configured key. Preserve
existing-secret and external ClickHouse behavior, and keep StatefulSet/API
references aligned with the generated Secret.

In `@k8s/helm/values.yaml`:
- Around line 260-270: The default auth.password in the ClickHouse auth
configuration must not contain the known “nemo” credential. Replace the shipped
default with an empty or unset value and configure validation/templates so
deployment requires either auth.existingSecret or an explicitly supplied
password before creating the ClickHouse credentials.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4147a697-5a03-421b-ae95-ba78cf196a36

📥 Commits

Reviewing files that changed from the base of the PR and between 7d03ce4 and 7d2c927.

📒 Files selected for processing (13)
  • k8s/helm/README.md
  • k8s/helm/ci/23-embedded-clickhouse.yaml
  • k8s/helm/ci/24-external-clickhouse.yaml
  • k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl
  • k8s/helm/templates/NOTES.txt
  • k8s/helm/templates/_helpers.tpl
  • k8s/helm/templates/api/api-deployment.yaml
  • k8s/helm/templates/clickhouse/clickhouse-secret.yaml
  • k8s/helm/templates/clickhouse/clickhouse-service.yaml
  • k8s/helm/templates/clickhouse/clickhouse-serviceaccount.yaml
  • k8s/helm/templates/clickhouse/clickhouse-statefulset.yaml
  • k8s/helm/values.yaml
  • tools/lint/lint-helm.sh

Comment thread k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl
Comment thread k8s/helm/templates/_helpers.tpl
Comment thread k8s/helm/values.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/lint/lint-helm.sh`:
- Around line 48-55: Add a third Helm validation case in the lint script after
the missing-Secret check that sets externalClickhouse.existingSecret while
omitting externalClickhouse.existingSecretPasswordKey, captures the helm
template output, fails if rendering succeeds, and asserts the expected
missing-password-key error from the helper validation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a11cec04-3ef2-4edd-ab32-64ed26c42b98

📥 Commits

Reviewing files that changed from the base of the PR and between 7d2c927 and f459811.

📒 Files selected for processing (13)
  • k8s/helm/README.md
  • k8s/helm/ci/23-embedded-clickhouse.yaml
  • k8s/helm/ci/24-external-clickhouse.yaml
  • k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl
  • k8s/helm/templates/NOTES.txt
  • k8s/helm/templates/_helpers.tpl
  • k8s/helm/templates/api/api-deployment.yaml
  • k8s/helm/templates/clickhouse/clickhouse-secret.yaml
  • k8s/helm/templates/clickhouse/clickhouse-service.yaml
  • k8s/helm/templates/clickhouse/clickhouse-serviceaccount.yaml
  • k8s/helm/templates/clickhouse/clickhouse-statefulset.yaml
  • k8s/helm/values.yaml
  • tools/lint/lint-helm.sh
🚧 Files skipped from review as they are similar to previous changes (2)
  • k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl
  • k8s/helm/README.md

Comment thread tools/lint/lint-helm.sh
Signed-off-by: Brian Newsom <brnewsom@nvidia.com>
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 28406/36247 78.4% 62.8%
Integration Tests 16991/34965 48.6% 21.2%

Signed-off-by: Brian Newsom <brnewsom@nvidia.com>
Comment thread k8s/helm/values.yaml Fixed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tools/lint/lint-helm.sh (1)

67-79: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Scope the credential assertions to the generated Secret and password references.

These global greps cannot distinguish the encoded password from a legitimate username or unrelated manifest field, so bmVtbw== may cause a false failure while its absence does not prove that data.password is generated. Select the embedded ClickHouse Secret and assert its password value and consumers’ key: password references directly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/lint/lint-helm.sh` around lines 67 - 79, Update the assertions around
generated_clickhouse_output to isolate the embedded ClickHouse Secret and
inspect its data.password value directly, rather than globally searching for
bmVtbw==. Also scope the consumer validation to ClickHouse password references
and require key: password, ensuring the generated Secret and all consumers use
the generated password key without matching unrelated manifest fields.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tools/lint/lint-helm.sh`:
- Around line 67-79: Update the assertions around generated_clickhouse_output to
isolate the embedded ClickHouse Secret and inspect its data.password value
directly, rather than globally searching for bmVtbw==. Also scope the consumer
validation to ClickHouse password references and require key: password, ensuring
the generated Secret and all consumers use the generated password key without
matching unrelated manifest fields.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c727b9cd-826c-44b7-8b5a-79b86cf5fd17

📥 Commits

Reviewing files that changed from the base of the PR and between 3ec7b56 and fc70535.

📒 Files selected for processing (1)
  • tools/lint/lint-helm.sh

Signed-off-by: Brian Newsom <brnewsom@nvidia.com>
@svvarom
svvarom self-requested a review July 30, 2026 20:15
Comment thread k8s/helm/templates/clickhouse/clickhouse-statefulset.yaml
Comment thread k8s/helm/templates/clickhouse/clickhouse-statefulset.yaml Outdated
Comment thread k8s/helm/templates/clickhouse/clickhouse-statefulset.yaml
Signed-off-by: Brian Newsom <brnewsom@nvidia.com>
Signed-off-by: Brian Newsom <brnewsom@nvidia.com>

@svvarom svvarom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BrianNewsom
BrianNewsom added this pull request to the merge queue Jul 30, 2026
Merged via the queue into main with commit cac16db Jul 30, 2026
58 checks passed
@BrianNewsom
BrianNewsom deleted the brnewsom/ase-243-helm-chart-for-intake-and-clickhouse-remote-deployment-in branch July 30, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants