Skip to content

feat(self-managed): add profile-based observability - #509

Merged
kristinapathak merged 13 commits into
mainfrom
feat/observability-profiles
Jul 29, 2026
Merged

feat(self-managed): add profile-based observability#509
kristinapathak merged 13 commits into
mainfrom
feat/observability-profiles

Conversation

@kristinapathak

@kristinapathak kristinapathak commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

TL;DR

  • Replaces the observability boolean matrix with one disabled, control, compute, or all profile.
  • Installs shared observability infrastructure once per cluster with complete, overridable monitor defaults.

Additional Details

  • Defaults control monitors on for control and all; defaults NVCA, DCGM, and worker monitors on for compute and all.
  • Enables the NVCA self-managed collector and BYOObservability for compute and all, while preserving explicit NVCA overrides.
  • Uses one generic ServiceMonitor template and one generic PodMonitor template.
  • Keeps component ownership overrides with install, existing, and disabled.
  • Deploys the function autoscaler for control and all, pins image 1.18.3, and uses chart 0.1.0.
  • Maps the resolved PromQL and authentication settings into the autoscaler chart's existing environment ConfigMap; no separate profile contract is published.

For the Reviewer

  • Review profile resolution and backend validation in 01-observability.yaml.gotmpl.
  • Review data-driven monitor rendering and the autoscaler values mapping in global.yaml.gotmpl.
  • Review compute-profile NVCA defaults and override handling in deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl.

For QA

  • make test in deploy/stacks/observability
  • make test-local in deploy/stacks/nvcf-compute-plane
  • python3 .cursor/hooks/validate-skill-fanout.py
  • git diff --check

Live-cluster QA remains for autoscaler readiness and scaling.

Issues

Relates to #15

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date.

Summary by CodeRabbit

  • New Features
    • Introduced profile-based observability (disabled, control, compute, all) with component ownership modes and profile-driven defaults.
    • Expanded default monitoring for compute-plane and worker (including DCGM and NVCA-related discovery/label filtering).
    • Updated function autoscaler behavior to follow the selected observability and metrics backend configuration.
  • Bug Fixes
    • Strengthened validation for incompatible profile/component/metrics backend combinations and required endpoint/auth settings.
  • Documentation
    • Reworked observability guidance, overrides, monitor ownership, and deployment/validation steps.
  • Tests
    • Added/expanded rendering and behavior tests, including negative cases for invalid settings.
  • Chores
    • Bumped observability-related chart versions and added an observability stack make test target.

Signed-off-by: Kristina Pathak <kpathak@nvidia.com>
@kristinapathak
kristinapathak requested review from a team as code owners July 28, 2026 20:15
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The observability stack moves from mode-based configuration to profile-based ownership, adds profile-derived monitor and metrics backend resolution, integrates the function autoscaler for selected self-managed profiles, and adds Helmfile, chart, and integration validation.

Changes

Observability profile migration

Layer / File(s) Summary
Profile contracts and defaults
ai-tooling/dev/skills/..., deploy/stacks/observability/{AGENTS.md,README.md}, deploy/stacks/observability/environments/{local,base}.yaml, deploy/stacks/nvcf-compute-plane/{README.md,environments/base.yaml}
Documents disabled, control, compute, and all profiles, component modes, metrics contracts, monitor ownership, and profile-based environment defaults.
Monitor chart and environment defaults
deploy/stacks/observability/charts/..., deploy/stacks/observability/environments/base.yaml
Adds grouped ServiceMonitor and generic PodMonitor rendering, compute and worker monitor definitions, collector settings, and chart metadata updates.
Profile resolution and release wiring
deploy/stacks/observability/helmfile.d/..., deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
Validates profiles and component dependencies, resolves metrics endpoints and collector settings, and wires profile-derived collector and monitor releases.
Self-managed autoscaler deployment
deploy/stacks/self-managed/{environments,global.yaml.gotmpl,helmfile.d/03-observability.yaml.gotmpl
Adds profile-gated function-autoscaler deployment with runtime, metrics, database, image, namespace, and scheduling configuration.
Compute-plane observability wiring
deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
Derives NVCA collector enablement and the BYOObservability feature gate from the selected observability profile and explicit overrides.
Profile and autoscaler validation
deploy/stacks/{observability,self-managed,nvcf-compute-plane}/{Makefile,tests}/*
Adds lint, profile, monitor, autoscaler, golden comparison, backend validation, and invalid-profile tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Values as observability.profile
  participant Helmfiles as observability Helmfiles
  participant Monitors as default-monitors
  participant Collector as otel-collector
  participant Autoscaler as function-autoscaler
  participant NVCA as nvca-operator
  Values->>Helmfiles: select profile and component ownership
  Helmfiles->>Monitors: render profile-derived monitor groups
  Helmfiles->>Collector: pass resolved collector and metrics settings
  Helmfiles->>Autoscaler: render release for control or all profiles
  Values->>NVCA: derive collector and BYOObservability settings
Loading

Possibly related issues

Possibly related PRs

  • NVIDIA/nvcf#258: Updates related observability guidance and imports.yaml documentation.
  • NVIDIA/nvcf#480: Provides earlier self-managed observability Helmfile wiring that this profile-based configuration updates.
  • NVIDIA/nvcf#512: Overlaps in function-autoscaler Helmfile integration and observability configuration.

Suggested reviewers: sbaum1994

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title follows Conventional Commits and accurately summarizes the main self-managed observability profile change.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/observability-profiles

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Kristina Pathak <kpathak@nvidia.com>
Signed-off-by: Kristina Pathak <kpathak@nvidia.com>
Signed-off-by: Kristina Pathak <kpathak@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
deploy/stacks/observability/Makefile (1)

25-34: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Split test target to satisfy checkmake's max body length.

Static analysis flags the test target body (8 lines) exceeding checkmake's 5-line limit. Consider splitting into a lint prerequisite target plus a slimmer test target that just runs the two scripts.

♻️ Example split
-test:
-	`@helm` lint charts/nvcf-default-monitors
-	`@helm` lint charts/nvcf-observability-contract
-	`@helm` lint charts/nvcf-otel-collector
-	`@helm` lint ../../helm/function-autoscaler \
-		--set functionautoscaler.image.registry=nvcr.io \
-		--set functionautoscaler.image.repository=test/nvcf-function-autoscaler
-	`@tests/profile-defaults.sh`
-	@../self-managed/tests/observability-autoscaler.sh
+lint:
+	`@helm` lint charts/nvcf-default-monitors
+	`@helm` lint charts/nvcf-observability-contract
+	`@helm` lint charts/nvcf-otel-collector
+	`@helm` lint ../../helm/function-autoscaler \
+		--set functionautoscaler.image.registry=nvcr.io \
+		--set functionautoscaler.image.repository=test/nvcf-function-autoscaler
+
+test: lint
+	`@tests/profile-defaults.sh`
+	@../self-managed/tests/observability-autoscaler.sh
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/stacks/observability/Makefile` around lines 25 - 34, Split the
Makefile’s test target into a lint prerequisite containing the four helm lint
commands and a shorter test target that depends on lint and runs only
tests/profile-defaults.sh and ../self-managed/tests/observability-autoscaler.sh.
Preserve the existing command order and behavior while keeping each target body
within checkmake’s five-line limit.

Source: Linters/SAST tools

deploy/stacks/self-managed/environments/base.yaml (1)

136-155: 🧹 Nitpick | 🔵 Trivial

New default installs the full shared observability stack for every self-managed control-plane deployment.

Switching the self-managed default to observability.profile: control means Prometheus Operator CRDs, the OTel Operator, otel-collector, VictoriaMetrics, and default-monitors are now installed unconditionally out of the box, even for minimal/local test clusters. Worth confirming this is the intended default footprint for smaller self-managed environments (e.g. local.yaml) and documenting the added resource requirements in the stack README/AGENTS.md if not already covered elsewhere.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/stacks/self-managed/environments/base.yaml` around lines 136 - 155,
The self-managed base configuration now enables the full observability stack by
default via observability.profile: control. Confirm whether this footprint is
intended for minimal environments such as local.yaml; if not, change the base
default to preserve lightweight deployments and explicitly opt control-plane
environments into the stack. If the default is intentional, retain it and
document the added resource requirements in the relevant stack documentation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/observability/AGENTS.md`:
- Around line 25-32: Add development-instruction sections to AGENTS.md covering
build commands, test commands, code style, and subtree-specific conventions.
Document the required commands make template HELMFILE_ENV=local and make test,
while preserving the existing scope and Key Files content.

In `@deploy/stacks/self-managed/global.yaml.gotmpl`:
- Around line 592-610: The functionautoscaler checksum currently hashes
unresolved top-level values instead of the rendered observability contract.
Update the `$observabilityContract` construction used by
`checksum/observability-contract` to consume the resolved metricsBackend and
autoscalerIntegration values forwarded by the observability infrastructure
render, or otherwise thread those resolved values into this template, so the
annotation changes whenever the ConfigMap-backed contract changes.

---

Nitpick comments:
In `@deploy/stacks/observability/Makefile`:
- Around line 25-34: Split the Makefile’s test target into a lint prerequisite
containing the four helm lint commands and a shorter test target that depends on
lint and runs only tests/profile-defaults.sh and
../self-managed/tests/observability-autoscaler.sh. Preserve the existing command
order and behavior while keeping each target body within checkmake’s five-line
limit.

In `@deploy/stacks/self-managed/environments/base.yaml`:
- Around line 136-155: The self-managed base configuration now enables the full
observability stack by default via observability.profile: control. Confirm
whether this footprint is intended for minimal environments such as local.yaml;
if not, change the base default to preserve lightweight deployments and
explicitly opt control-plane environments into the stack. If the default is
intentional, retain it and document the added resource requirements in the
relevant stack documentation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 28159a2b-aa8b-4742-a251-241dee71eebc

📥 Commits

Reviewing files that changed from the base of the PR and between afca4ec and 67b8f0b.

📒 Files selected for processing (27)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/helm/function-autoscaler/Chart.yaml
  • deploy/helm/function-autoscaler/templates/deployment.yaml
  • deploy/helm/function-autoscaler/values.yaml
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/nvca-servicemonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/worker-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/autoscaler-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/profile-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/values.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/environments/local.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/global.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh

Comment thread deploy/stacks/observability/AGENTS.md
Comment thread deploy/stacks/self-managed/global.yaml.gotmpl Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md`:
- Around line 73-77: Revise the “Which subtree do I edit to change X” guidance
to remove the claim that all stack sources are monorepo-native. Limit editable
paths to checked-in sources and direct users to the external repository or
Helmfile chart reference for charts not present in the monorepo.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4e63fc26-c0ff-489d-af26-4976a5fe4916

📥 Commits

Reviewing files that changed from the base of the PR and between 67b8f0b and 6666163.

📒 Files selected for processing (6)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
🚧 Files skipped from review as they are similar to previous changes (4)
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/observability/README.md

Comment thread ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
Signed-off-by: Kristina Pathak <kpathak@nvidia.com>
Signed-off-by: Kristina Pathak <kpathak@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
deploy/stacks/observability/Makefile (1)

25-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Split the test recipe to satisfy Makefile lint.

checkmake reports maxbodylength here: the target has eight recipe lines versus the configured limit of five. Extract chart linting and shell validations into helper targets or a dedicated script, then make test depend on them.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/stacks/observability/Makefile` around lines 25 - 34, Reduce the recipe
length of the test target to satisfy checkmake’s maxbodylength limit. Extract
the Helm lint commands and shell validation commands into helper targets or a
dedicated script, then make test depend on those helpers while preserving all
existing validations.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@deploy/stacks/observability/charts/nvcf-default-monitors/templates/nvca-servicemonitor.yaml`:
- Around line 18-39: Remove the compute-plane monitor wiring from both
deploy/stacks/observability/charts/nvcf-default-monitors/templates/nvca-servicemonitor.yaml
lines 18-39 and
deploy/stacks/observability/charts/nvcf-default-monitors/templates/worker-podmonitor.yaml
lines 18-38, or relocate those resources to the owning compute-plane layer.
Ensure the shared observability release renders neither the NVCA ServiceMonitor
nor the worker PodMonitor, and does not add control-plane or compute-plane
targets.

In `@deploy/stacks/observability/tests/profile-defaults.sh`:
- Around line 186-204: Add mTLS contract coverage alongside the existing token
scenario in profile-defaults.sh. Render the observability-contract with mTLS
authentication and assert the generated manifest contains
TIMESERIES_DB__AUTH_MODE plus the expected certificate-path and private-key-path
fields, using the established test helpers and failure pattern.

---

Nitpick comments:
In `@deploy/stacks/observability/Makefile`:
- Around line 25-34: Reduce the recipe length of the test target to satisfy
checkmake’s maxbodylength limit. Extract the Helm lint commands and shell
validation commands into helper targets or a dedicated script, then make test
depend on those helpers while preserving all existing validations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 226f04c1-e784-412f-8936-e290f17981f4

📥 Commits

Reviewing files that changed from the base of the PR and between afca4ec and 6666163.

📒 Files selected for processing (26)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/helm/function-autoscaler/templates/deployment.yaml
  • deploy/helm/function-autoscaler/values.yaml
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/nvca-servicemonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/worker-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/autoscaler-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/profile-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/values.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/environments/local.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/global.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
🚧 Files skipped from review as they are similar to previous changes (18)
  • deploy/stacks/observability/environments/local.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/values.yaml
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/observability/charts/nvcf-observability-contract/Chart.yaml
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
  • deploy/stacks/observability/charts/nvcf-otel-collector/values.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/helm/function-autoscaler/values.yaml
  • deploy/helm/function-autoscaler/templates/deployment.yaml
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md

Comment thread deploy/stacks/observability/tests/profile-defaults.sh Outdated
Signed-off-by: Kristina Pathak <kpathak@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml`:
- Around line 76-79: Replace the unsupported namespaceSelector.matchExpressions
in the default PodMonitor configuration with namespaceSelector.matchNames
containing the explicit worker namespace names. Add a profile-defaults.sh
assertion that verifies the rendered PodMonitor uses the expected matchNames
structure and values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 12581773-3e5f-4b4d-b254-bc2d5cf1b3d4

📥 Commits

Reviewing files that changed from the base of the PR and between 6666163 and 1b52225.

📒 Files selected for processing (8)
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/servicemonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/worker-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/tests/profile-defaults.sh
🚧 Files skipped from review as they are similar to previous changes (4)
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/self-managed/global.yaml.gotmpl`:
- Around line 596-622: Update the functionautoscaler namespace field to derive
from autoscalerIntegration.namespace instead of hard-coding nvcf, keeping the
pod and contract ConfigMap co-located. Add a render test covering a non-default
autoscalerIntegration namespace and verifying both resources use it.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 42de8336-4c13-4e34-9c5b-aa0db4696737

📥 Commits

Reviewing files that changed from the base of the PR and between afca4ec and 1b52225.

📒 Files selected for processing (26)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/helm/function-autoscaler/templates/deployment.yaml
  • deploy/helm/function-autoscaler/values.yaml
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/servicemonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/worker-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/autoscaler-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/templates/profile-configmap.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/values.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/environments/local.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/global.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
🚧 Files skipped from review as they are similar to previous changes (18)
  • deploy/stacks/observability/charts/nvcf-otel-collector/values.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/Chart.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/dcgm-podmonitor.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/values.yaml
  • deploy/stacks/observability/charts/nvcf-otel-collector/Chart.yaml
  • deploy/stacks/observability/environments/local.yaml
  • deploy/stacks/observability/charts/nvcf-observability-contract/Chart.yaml
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/helm/function-autoscaler/values.yaml
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
  • deploy/helm/function-autoscaler/templates/deployment.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/tests/profile-defaults.sh
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md

Comment thread deploy/stacks/self-managed/global.yaml.gotmpl Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml`:
- Around line 33-37: Update the generic PodMonitor labels template to merge
defaultMonitors.commonLabels with each monitor’s labels, ensuring common labels
are included while per-monitor labels take precedence on conflicts. Preserve the
existing chart labels and apply this behavior to the PodMonitors rendered by the
surrounding template.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 731e9d11-997a-4576-9182-28563b500c5c

📥 Commits

Reviewing files that changed from the base of the PR and between 1b52225 and 06842ea.

📒 Files selected for processing (13)
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/self-managed/global.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
💤 Files with no reviewable changes (3)
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl
  • deploy/stacks/observability/AGENTS.md
🚧 Files skipped from review as they are similar to previous changes (5)
  • deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl
  • deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl
  • deploy/stacks/self-managed/environments/base.yaml
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/observability/README.md`:
- Around line 20-23: Update the enabled-profiles description near the reusable
and self-managed stack defaults to qualify that these components are installed
by default, unless an existing ownership override prevents this stack from
installing them. Keep the listed components and profile defaults unchanged, and
align the wording with the documented override behavior below.
- Around line 97-102: Update the bundled VictoriaMetrics endpoint documentation
near the remote write and PromQL URLs to state that the monitoring namespace is
configuration-dependent. Explain that users overriding the Helmfile-derived
namespace must replace “monitoring” in the service hostname with their
configured namespace.
- Around line 129-133: Update the Validate section in the observability README
to include git diff --check alongside the existing template and test commands,
ensuring the documented validation steps satisfy the repository requirement
before committing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0947be5c-08e2-424b-adda-c2b028504b8e

📥 Commits

Reviewing files that changed from the base of the PR and between 06842ea and 669d2af.

📒 Files selected for processing (8)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
🚧 Files skipped from review as they are similar to previous changes (7)
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/tests/profile-defaults.sh
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md

Comment thread deploy/stacks/observability/README.md
Comment thread deploy/stacks/observability/README.md Outdated
Comment thread deploy/stacks/observability/README.md
@kristinapathak
kristinapathak requested a review from a team as a code owner July 28, 2026 21:44
@kristinapathak

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh`:
- Around line 75-90: Extend the observability profile test around render_values
to provide explicit otelCollector imageRepository and imageTag values, then
assert both are preserved in the rendered compute-overrides.yaml. Use the
existing YAML inspection/assertion patterns and collector-related symbols in
observability-profile.sh, while keeping the enabled and feature-gate checks
unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a89be95f-b9b7-46c8-8401-1c3cb87df2a8

📥 Commits

Reviewing files that changed from the base of the PR and between 06842ea and a38018d.

⛔ Files ignored due to path filters (1)
  • deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-31ea6bf9-nvca-operator/helm-nvca-operator/templates/self-managed-nvcfbackend-cm.yaml is excluded by !**/testdata/**
📒 Files selected for processing (15)
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/stacks/nvcf-compute-plane/Makefile
  • deploy/stacks/nvcf-compute-plane/README.md
  • deploy/stacks/nvcf-compute-plane/environments/base.yaml
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • deploy/stacks/nvcf-compute-plane/tests/compare-golden.sh
  • deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
🚧 Files skipped from review as they are similar to previous changes (9)
  • deploy/stacks/observability/AGENTS.md
  • deploy/stacks/observability/Makefile
  • deploy/stacks/observability/charts/nvcf-default-monitors/templates/podmonitors.yaml
  • ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
  • deploy/stacks/observability/README.md
  • deploy/stacks/observability/tests/profile-defaults.sh
  • deploy/stacks/self-managed/tests/observability-autoscaler.sh
  • deploy/stacks/observability/charts/nvcf-default-monitors/values.yaml
  • deploy/stacks/observability/environments/base.yaml

Comment thread deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
@kristinapathak

Copy link
Copy Markdown
Collaborator Author

Local k3d smoke test on d728e6390:

  • observability.profile: all deployed the five expected shared releases: Prometheus Operator CRDs, OpenTelemetry Operator, VictoriaMetrics, one collector/Target Allocator, and default monitors. All workloads are ready.
  • The Target Allocator loaded the five ServiceMonitors and two PodMonitors and assigned a matching state-metrics endpoint to the collector.
  • A synthetic metric traversed ServiceMonitor -> Target Allocator -> OTel collector -> Prometheus remote write -> VictoriaMetrics. The collector reported 60 accepted and 60 exported points with 0 failed points; a VictoriaMetrics PromQL query returned the test series.
  • The live run exposed missing Target Allocator Secret-watch RBAC. Commit d728e6390 adds read-only get/list/watch access and a render assertion; after resync, the Secret informer synchronized with no authorization errors.
  • The self-managed Helmfile selected exactly one function autoscaler for all, rendered chart 0.1.0, image pin 1.18.3, and the bundled VictoriaMetrics PromQL endpoint. The Deployment reached 1/1 and the chart Helm test passed using a local probe-compatible stand-in image.

The autoscaler result validates profile selection, rendered configuration, packaging, probes, Service/DNS connectivity, and chart lifecycle. It does not validate the production Rust autoscaler scaling loop, Cassandra integration, or NVCF API calls.

@kristinapathak
kristinapathak added this pull request to the merge queue Jul 29, 2026
Merged via the queue into main with commit ef6f96c Jul 29, 2026
15 checks passed
@kristinapathak
kristinapathak deleted the feat/observability-profiles branch July 29, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants