Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 13 additions & 19 deletions ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@ description: >-
namespaces, and `needs:` dependency chains. Reads
deploy/stacks/self-managed/helmfile.d/*.yaml.gotmpl and
deploy/stacks/nvcf-compute-plane/helmfile.d/*.yaml.gotmpl as the source of
truth for ordering and versions, with imports.yaml for upstream provenance.
truth for ordering and versions.
Use when a user or developer asks "what deploys X", "what does X depend
on", "what hooks run for X", "walk me through deployment order", "which
subtree do I edit to change X", "what namespaces does the stack use", or
mentions stack topology, helmfile stages, deployment DAG, or dependency map.
license: Apache-2.0
compatibility: Requires a local checkout of the NVCF monorepo with deploy/stacks/self-managed/ and deploy/stacks/nvcf-compute-plane/ present
author: "nvcf-core-eng <nvcf-core-eng@exchange.nvidia.com>"
version: "1.0.0"
version: "1.1.0"
tags: [nvcf, self-managed, self-hosted, helmfile, deployment, stack-topology]
tools: [Read, Grep, Glob]
metadata:
internal: false
author: "nvcf-core-eng <nvcf-core-eng@exchange.nvidia.com>"
version: "1.0.0"
version: "1.1.0"
tags: [nvcf, self-managed, self-hosted, helmfile, deployment, stack-topology, dependencies, hooks]
languages: [yaml, markdown]
frameworks: [helmfile, helm, kubernetes]
Expand All @@ -39,48 +39,42 @@ Use this skill long enough to answer the question, then hand off to the right ex

## Required inputs

Read these from the monorepo root (the directory containing `imports.yaml`):
Read these from the monorepo root.

Authoritative (always read first when answering):

- `deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl`
- `deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl`
- `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl`
- `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`

Provenance (when asked which subtree is monorepo-native vs. upstream-owned):

- `imports.yaml`

Chart-level (when the chart is checked into the monorepo):

- `deploy/helm/<chart>/Chart.yaml`
- `deploy/helm/<chart>/values.yaml`

If workspace routing metadata is available and disagrees with the helmfile, the
helmfile wins. Update stale routing metadata in the same change rather than
guessing.

## Common questions

What deploys X
: Look up release `X` in the helmfile stage files. Return chart name, version, namespace, and which gotmpl file declares it. If the chart is checked in, also point at `deploy/helm/<chart>/`.

What does X depend on
: Return the `needs:` chain for that release plus the stage gate it sits behind (control-plane stages 1 -> 2 -> 3, then compute-plane). Include any `condition:` that gates whether X deploys at all.
: Return the `needs:` chain for that release plus the stage gate it sits behind (control-plane stages 0 -> 1 -> 2 -> 3, then compute-plane). Include any profile, `condition:`, or component mode that gates whether X deploys at all.

What hooks run for X
: Read the checked-in chart under `deploy/helm/<chart>/` when available. Search its `templates/` directory for Helm hook annotations, weights, hook events (pre-install / post-install), images used, and purpose. Cite the chart-relative template file path. If the chart is consumed from OCI only and the local workspace has routing metadata, use the workspace metadata and cite it.
: Read the checked-in chart under `deploy/helm/<chart>/` when available. Search its `templates/` directory for Helm hook annotations, weights, hook events (pre-install / post-install), images used, and purpose. Cite the chart-relative template file path. If the chart is not checked in, cite its Helmfile chart reference and state that local templates are unavailable.

Walk me through the full deployment order
: Summarize control-plane stages 0 through 3 from the self-managed gotmpl file headers. Then summarize the compute-plane stage from `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl` and `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`. Call out which releases run in parallel inside a stage and which are serialized by `needs:`.
: Summarize control-plane stages 0 through 3 from the self-managed gotmpl files. Stage 0 delegates to the shared observability Helmfile when `observability.profile` is enabled. Stage 3 installs State Metrics and the function autoscaler for `control` and `all`. Then summarize the compute-plane stage from `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl` and `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`. Call out which releases run in parallel inside a stage and which are serialized by `needs:`.

Which subtree do I edit to change X
: Two answers, both are important. For chart wiring (Helm hooks, manifests, values, hook weights) point at `deploy/helm/<chart>/` if the chart is checked in, or note `oci-only` (the chart is consumed from the OCI registry and does not live in the monorepo). For runtime application logic, use the chart image repository, imports.yaml, and any available workspace routing metadata. `authoritative_source: native` means edits land here. `upstream` means edits also flow back to the upstream repo through the internal source-sync workflow.
: Point at the Helmfile path for orchestration, `deploy/helm/<chart>/` for checked-in chart wiring, and the chart image repository under `src/`, `infra/`, or `migrations/` for runtime behavior. All stack sources are monorepo-native, so edits land in this repository.

What namespaces does the stack use
: Return the list from the helmfile (`namespace:` per release). If workspace routing metadata includes a destroy namespace list, include it as supplemental context and cite that source.
: Return the list from the helmfile (`namespace:` per release).
Comment thread
kristinapathak marked this conversation as resolved.
Resolved

## Subtree mapping

Expand All @@ -89,8 +83,8 @@ The stack lives in three layers across the monorepo:
| Concern | Lives at |
|---------|----------|
| Helmfile orchestration (stage ordering, env wiring, secrets flow) | `deploy/stacks/self-managed/` |
| Chart manifests, helm hooks, values | `deploy/helm/<chart>/` (when vendored) or OCI registry only |
| Runtime application code, migrations | `src/`, `infra/`, `migrations/` (per `imports.yaml`) |
| Chart manifests, helm hooks, values | `deploy/helm/<chart>/` when checked in; otherwise use the Helmfile chart reference |
| Runtime application code, migrations | `src/`, `infra/`, `migrations/` |

When a question crosses layers, answer by layer and tell the user the order to edit (chart wiring first if the deploy contract changes, image source if behavior changes).

Expand Down
7 changes: 6 additions & 1 deletion deploy/helm/function-autoscaler/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,16 @@ spec:
{{- toYaml .Values.functionautoscaler.readinessProbe | nindent 10 }}
resources:
{{- toYaml .Values.functionautoscaler.resources | nindent 10 }}
{{- if .Values.functionautoscaler.env }}
{{- if or .Values.functionautoscaler.env .Values.functionautoscaler.envFrom }}
envFrom:
{{- if .Values.functionautoscaler.env }}
- configMapRef:
name: {{ include "function-autoscaler.fullname" . }}-env
{{- end }}
{{- with .Values.functionautoscaler.envFrom }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- if or .Values.functionautoscaler.volumeMounts .Values.functionautoscaler.vault.enabled }}
volumeMounts:
{{- with .Values.functionautoscaler.volumeMounts }}
Expand Down
7 changes: 7 additions & 0 deletions deploy/helm/function-autoscaler/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,10 @@ functionautoscaler:
# Read the secrets file rendered by the Vault Agent. Override this value
# when mounting secrets manually.
SECRETS_PATH: "/vault/secrets/secrets.json"

# Additional ConfigMaps or Secrets to load after the chart-owned env
# ConfigMap. This lets a stack provide resolved infrastructure contracts
# without duplicating them in this chart's values.
envFrom: []
# - configMapRef:
# name: nvcf-observability-autoscaler
38 changes: 19 additions & 19 deletions deploy/stacks/observability/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

## Purpose

This is the shared NVCF observability Helmfile stack scaffold. It owns optional
cluster-level observability infrastructure intended for self-managed
control-plane and compute-plane deployments once consuming stack wiring lands.
This is the shared NVCF observability Helmfile stack. It owns cluster-level
observability infrastructure for self-managed control-plane and compute-plane
deployments.

## Scope

Expand All @@ -14,30 +14,30 @@ control-plane and compute-plane deployments once consuming stack wiring lands.
- Keep application metrics endpoints, labels, port names, and paths in the
application charts that expose those endpoints.
- Do not add control-plane services or compute-plane services here.
- Keep base runtime gates off until a consuming stack owns chart distribution,
mirrored images, and environment overlays.
- Consumer overlays for self-managed control-plane deployments may enable the
control-plane collector, OTel Operator, Prometheus Operator CRDs, and bundled
TSDB contract together.
- Keep BYOO and NVCA collectors default off unless a consumer explicitly
enables them through values.
- Keep the reusable base profile `disabled`.
- Derive control-plane and compute-plane behavior from
`observability.profile`; do not expose parallel plane booleans.
- Enabled profiles install shared components once by default. Fine-grained
component modes are `install`, `existing`, and `disabled`.
- Derive NVCA BYOO support from compute and all profiles. Keep the NVCA
collector disabled by default.

## Key Files

- `helmfile.d/01-observability.yaml.gotmpl`: shared observability releases
- `environments/base.yaml`: scaffold defaults and scrape contracts
- `charts/nvcf-otel-collector`: OpenTelemetryCollector resource and Target Allocator RBAC
- `charts/nvcf-default-monitors`: centrally owned concrete monitor resources
- `charts/nvcf-observability-contract`: resolved settings for consumers such as the autoscaler
- `values/victoria-metrics.yaml.gotmpl`: values bridge for the VictoriaMetrics chart
Comment thread
kristinapathak marked this conversation as resolved.
Resolved

## Modes
## Profiles

- `install`: this stack installs and owns enabled shared infrastructure.
- `existing`: this stack may render default monitors, but shared infrastructure
is owned by another stack instance.
- `disabled`: this stack renders no observability runtime resources. This is
not a supported self-managed control-plane autoscaler mode, but is valid for
compute-plane deployments that do not need NVCF-managed scrape coverage.
- `disabled`: render no observability runtime resources.
- `control`: install control-plane metrics and autoscaler backend defaults.
- `compute`: install NVCA, DCGM, and worker metric defaults.
- `all`: install the union once.

Do not use Helm `lookup` to silently change modes based on live cluster state.
The consuming stack must choose the mode explicitly.
Do not use Helm `lookup` to silently change profiles or component ownership
based on live cluster state. The consuming stack must choose the profile
explicitly.
13 changes: 12 additions & 1 deletion deploy/stacks/observability/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ KUBECONFIG_FLAG = $(if $(KUBECONFIG_FILE),--kubeconfig $(KUBECONFIG_FILE))
HELMFILE_SELECTOR ?=
HELMFILE_EXTRA_ARGS ?= --allow-no-matching-release

.PHONY: template install sync apply destroy clean help
.PHONY: template test install sync apply destroy clean help
.DEFAULT_GOAL := help

template: clean
Expand All @@ -22,6 +22,16 @@ template: clean
$(if $(HELMFILE_SELECTOR),--selector $(HELMFILE_SELECTOR)) \
template --output-dir "$(OUTPUT_DIR)"

test:
@helm lint charts/nvcf-default-monitors
@helm lint charts/nvcf-observability-contract
@helm lint charts/nvcf-otel-collector
@helm lint ../../helm/function-autoscaler \
--set functionautoscaler.image.registry=nvcr.io \
--set functionautoscaler.image.repository=test/nvcf-function-autoscaler
@tests/profile-defaults.sh
@../self-managed/tests/observability-autoscaler.sh

install:
@echo ">>> Installing observability stack"
HELMFILE_ENV="$(HELMFILE_ENV)" \
Expand Down Expand Up @@ -55,6 +65,7 @@ help:
@echo ""
@echo "Targets:"
@echo " template Generate Kubernetes manifests without applying"
@echo " test Validate profile release and monitor defaults"
@echo " install Install the stack with helmfile sync"
@echo " apply Apply changes with helmfile apply"
@echo " destroy Remove stack releases"
Expand Down
Loading
Loading