Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 13 additions & 19 deletions ai-tooling/dev/skills/nvcf-explore-stack/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@ description: >-
namespaces, and `needs:` dependency chains. Reads
deploy/stacks/self-managed/helmfile.d/*.yaml.gotmpl and
deploy/stacks/nvcf-compute-plane/helmfile.d/*.yaml.gotmpl as the source of
truth for ordering and versions, with imports.yaml for upstream provenance.
truth for ordering and versions.
Use when a user or developer asks "what deploys X", "what does X depend
on", "what hooks run for X", "walk me through deployment order", "which
subtree do I edit to change X", "what namespaces does the stack use", or
mentions stack topology, helmfile stages, deployment DAG, or dependency map.
license: Apache-2.0
compatibility: Requires a local checkout of the NVCF monorepo with deploy/stacks/self-managed/ and deploy/stacks/nvcf-compute-plane/ present
author: "nvcf-core-eng <nvcf-core-eng@exchange.nvidia.com>"
version: "1.0.0"
version: "1.1.0"
tags: [nvcf, self-managed, self-hosted, helmfile, deployment, stack-topology]
tools: [Read, Grep, Glob]
metadata:
internal: false
author: "nvcf-core-eng <nvcf-core-eng@exchange.nvidia.com>"
version: "1.0.0"
version: "1.1.0"
tags: [nvcf, self-managed, self-hosted, helmfile, deployment, stack-topology, dependencies, hooks]
languages: [yaml, markdown]
frameworks: [helmfile, helm, kubernetes]
Expand All @@ -39,48 +39,42 @@ Use this skill long enough to answer the question, then hand off to the right ex

## Required inputs

Read these from the monorepo root (the directory containing `imports.yaml`):
Read these from the monorepo root.

Authoritative (always read first when answering):

- `deploy/stacks/self-managed/helmfile.d/00-observability-infrastructure.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl`
- `deploy/stacks/self-managed/helmfile.d/03-observability.yaml.gotmpl`
- `deploy/stacks/observability/helmfile.d/01-observability.yaml.gotmpl`
- `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl`
- `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`

Provenance (when asked which subtree is monorepo-native vs. upstream-owned):

- `imports.yaml`

Chart-level (when the chart is checked into the monorepo):

- `deploy/helm/<chart>/Chart.yaml`
- `deploy/helm/<chart>/values.yaml`

If workspace routing metadata is available and disagrees with the helmfile, the
helmfile wins. Update stale routing metadata in the same change rather than
guessing.

## Common questions

What deploys X
: Look up release `X` in the helmfile stage files. Return chart name, version, namespace, and which gotmpl file declares it. If the chart is checked in, also point at `deploy/helm/<chart>/`.

What does X depend on
: Return the `needs:` chain for that release plus the stage gate it sits behind (control-plane stages 1 -> 2 -> 3, then compute-plane). Include any `condition:` that gates whether X deploys at all.
: Return the `needs:` chain for that release plus the stage gate it sits behind (control-plane stages 0 -> 1 -> 2 -> 3, then compute-plane). Include any profile, `condition:`, or component mode that gates whether X deploys at all.

What hooks run for X
: Read the checked-in chart under `deploy/helm/<chart>/` when available. Search its `templates/` directory for Helm hook annotations, weights, hook events (pre-install / post-install), images used, and purpose. Cite the chart-relative template file path. If the chart is consumed from OCI only and the local workspace has routing metadata, use the workspace metadata and cite it.
: Read the checked-in chart under `deploy/helm/<chart>/` when available. Search its `templates/` directory for Helm hook annotations, weights, hook events (pre-install / post-install), images used, and purpose. Cite the chart-relative template file path. If the chart is not checked in, cite its Helmfile chart reference and state that local templates are unavailable.

Walk me through the full deployment order
: Summarize control-plane stages 0 through 3 from the self-managed gotmpl file headers. Then summarize the compute-plane stage from `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl` and `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`. Call out which releases run in parallel inside a stage and which are serialized by `needs:`.
: Summarize control-plane stages 0 through 3 from the self-managed gotmpl files. Stage 0 delegates to the shared observability Helmfile when `observability.profile` is enabled. Stage 3 installs State Metrics and the function autoscaler for `control` and `all`. Then summarize the compute-plane stage from `deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl` and `deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl`. Call out which releases run in parallel inside a stage and which are serialized by `needs:`.

Which subtree do I edit to change X
: Two answers, both are important. For chart wiring (Helm hooks, manifests, values, hook weights) point at `deploy/helm/<chart>/` if the chart is checked in, or note `oci-only` (the chart is consumed from the OCI registry and does not live in the monorepo). For runtime application logic, use the chart image repository, imports.yaml, and any available workspace routing metadata. `authoritative_source: native` means edits land here. `upstream` means edits also flow back to the upstream repo through the internal source-sync workflow.
: Point at the Helmfile path for orchestration, `deploy/helm/<chart>/` for checked-in chart wiring, and the image source under `src/`, `infra/`, or `migrations/` for runtime behavior. If a referenced chart is not checked in, report its Helmfile chart reference and do not claim a local source path.

What namespaces does the stack use
: Return the list from the helmfile (`namespace:` per release). If workspace routing metadata includes a destroy namespace list, include it as supplemental context and cite that source.
: Return the list from the helmfile (`namespace:` per release).
Comment thread
kristinapathak marked this conversation as resolved.

## Subtree mapping

Expand All @@ -89,8 +83,8 @@ The stack lives in three layers across the monorepo:
| Concern | Lives at |
|---------|----------|
| Helmfile orchestration (stage ordering, env wiring, secrets flow) | `deploy/stacks/self-managed/` |
| Chart manifests, helm hooks, values | `deploy/helm/<chart>/` (when vendored) or OCI registry only |
| Runtime application code, migrations | `src/`, `infra/`, `migrations/` (per `imports.yaml`) |
| Chart manifests, helm hooks, values | `deploy/helm/<chart>/` when checked in; otherwise use the Helmfile chart reference |
| Runtime application code, migrations | `src/`, `infra/`, `migrations/` |

When a question crosses layers, answer by layer and tell the user the order to edit (chart wiring first if the deploy contract changes, image source if behavior changes).

Expand Down
10 changes: 7 additions & 3 deletions deploy/stacks/nvcf-compute-plane/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ include Makefile.dist
-include helmfile-docker.mk

# --- Development-Only Targets ---
.PHONY: dist clean-dist ensure-helm ensure-helmfile ensure-binaries render-local test-local generate-golden
.PHONY: dist clean-dist ensure-helm ensure-helmfile ensure-binaries render-local test-observability-profile test-local generate-golden

# --- Binary Management (Development Only) ---
ensure-helm:
Expand Down Expand Up @@ -159,9 +159,12 @@ render-local: dist
@cp -r testdata/registration $(DIST_DIR)/
@cd $(DIST_DIR) && CLUSTER_NAME=ncp-local NCA_ID=ncp-local HELMFILE_ENV=local make template

test-local: render-local
test-observability-profile:
@tests/observability-profile.sh

test-local: test-observability-profile render-local
@echo ">>> Comparing $(DIST_DIR)/out against golden $(GOLDEN_LOCAL_DIR)..."
@if ! diff -ruN "$(GOLDEN_LOCAL_DIR)" "$(DIST_DIR)/out"; then \
@if ! tests/compare-golden.sh "$(GOLDEN_LOCAL_DIR)" "$(DIST_DIR)/out"; then \
echo ""; \
echo "ERROR: rendered output in $(DIST_DIR)/out differs from golden $(GOLDEN_LOCAL_DIR)."; \
echo " Review the diff above. If the change is intentional, refresh the golden files:"; \
Expand Down Expand Up @@ -208,6 +211,7 @@ help:
@echo " clean-dist Remove the distribution directory"
@echo ""
@echo "Testing Targets:"
@echo " test-observability-profile Verify NVCA observability profile defaults"
@echo " test-local Render the local env and diff against golden testdata"
@echo " generate-golden Render the local env and update golden testdata"
@echo ""
Expand Down
25 changes: 25 additions & 0 deletions deploy/stacks/nvcf-compute-plane/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,31 @@ Helmfile.
an environment file for your registry and service endpoints, then pass its name
without the `.yaml` suffix.

## Observability

The stack defaults `observability.profile` to `compute`. The `compute` and
`all` profiles enable the NVCA collector and `BYOObservability` feature gate.
The `control` and `disabled` profiles leave both off.

One value selects the normal behavior:

```yaml
observability:
profile: compute
```

Explicit NVCA values override the profile defaults:

```yaml
global:
nvcaOperator:
selfManaged:
otelCollector:
enabled: false
featureGateValues:
- "-BYOObservability"
```

## Chart and Image Sources

The stack pins the NVCA operator chart in
Expand Down
4 changes: 4 additions & 0 deletions deploy/stacks/nvcf-compute-plane/environments/base.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ global:
# value: worker
# effect: NoSchedule

# One profile controls all compute observability defaults.
observability:
profile: compute

# =============================================================================
# Optional Add-on Operators
# =============================================================================
Expand Down
24 changes: 22 additions & 2 deletions deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ environments:
the raw list; output is the normalized list as YAML.
*/}}
{{- define "nvca-operator.featureGateValues" -}}
{{- $featureGateValues := . | default list -}}
{{- $featureGateValues := dig "values" list . | default list -}}
{{- $defaultFeatureGates := list
"DynamicGPUDiscovery"
"InfraResourceOverhead"
Expand All @@ -32,6 +32,9 @@ environments:
"EnforceHelmTaskResourceLimits"
"EnforceContainerTaskResourceLimits"
-}}
{{- if dig "byooEnabled" false . -}}
{{- $defaultFeatureGates = append $defaultFeatureGates "BYOObservability" -}}
{{- end -}}
{{- range $gate := $defaultFeatureGates -}}
{{- if not (or (has $gate $featureGateValues) (has (printf "-%s" $gate) $featureGateValues)) -}}
{{- $featureGateValues = append $featureGateValues $gate -}}
Expand Down Expand Up @@ -59,6 +62,11 @@ helmDefaults:
{{- $registrationValues := readFile $registrationValuesPath | fromYaml | default dict }}
{{- $registrationAgentMergeConfig := dig "agentConfig" "mergeConfig" "" $registrationValues }}
{{- $environmentAgentMergeConfig := dig "agentConfig" "mergeConfig" "" .Values }}
{{- $observabilityProfile := dig "observability" "profile" "compute" .Values }}
{{- if not (has $observabilityProfile (list "disabled" "control" "compute" "all")) }}
{{- fail (printf "observability.profile must be disabled, control, compute, or all, got %q" $observabilityProfile) }}
{{- end }}
{{- $computeObservabilityEnabled := or (eq $observabilityProfile "compute") (eq $observabilityProfile "all") }}
{{- $agentMergeConfig := dict }}
{{- if $registrationAgentMergeConfig }}
{{- $agentMergeConfig = mergeOverwrite $agentMergeConfig ($registrationAgentMergeConfig | fromYaml | default dict) }}
Expand Down Expand Up @@ -104,6 +112,8 @@ releases:
clusterName: {{ requiredEnv "CLUSTER_NAME" }}
ncaId: {{ requiredEnv "NCA_ID" }}
{{- $selfManaged := dig "nvcaOperator" "selfManaged" dict .Values.global }}
{{- $otelCollector := dig "otelCollector" dict $selfManaged }}
{{- $otelCollectorEnabled := dig "enabled" $computeObservabilityEnabled $otelCollector }}
selfManaged:
identitySource: psat
{{- with dig "icmsServiceURL" "" $selfManaged }}
Expand All @@ -125,7 +135,17 @@ releases:
natsHostOverride: {{ . | quote }}
{{- end }}
featureGateValues:
{{- include "nvca-operator.featureGateValues" (dig "featureGateValues" list $selfManaged) | nindent 10 }}
{{- include "nvca-operator.featureGateValues" (dict
"values" (dig "featureGateValues" list $selfManaged)
"byooEnabled" $computeObservabilityEnabled) | nindent 10 }}
otelCollector:
enabled: {{ $otelCollectorEnabled }}
{{- with dig "imageRepository" "" $otelCollector }}
imageRepository: {{ . | quote }}
{{- end }}
{{- with dig "imageTag" "" $otelCollector }}
imageTag: {{ . | quote }}
{{- end }}
imageCredHelper:
imageRepository: {{ .Values.global.image.registry }}/{{ .Values.global.image.repository }}/nvcf-image-credential-helper
{{- with dig "imageCredHelper" "imageTag" "" $selfManaged }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ data:
cloudProvider: "NCP"
region: "us-west-1"
attributes: []
capabilities: ["DynamicGPUDiscovery","InfraResourceOverhead","EnforceHelmFunctionResourceLimits","EnforceContainerFunctionResourceLimits","EnforceHelmTaskResourceLimits","EnforceContainerTaskResourceLimits"]
capabilities: ["DynamicGPUDiscovery","InfraResourceOverhead","EnforceHelmFunctionResourceLimits","EnforceContainerFunctionResourceLimits","EnforceHelmTaskResourceLimits","EnforceContainerTaskResourceLimits","BYOObservability"]
gpusB64: ""
cacheCSIVolumeMountOptionsConfig:
disabled: false
Expand All @@ -67,7 +67,7 @@ data:
imageRepository: "nvcr.io/0651155215864979/ncp-dev/samba"
imageTag: "1.0.5"
otelCollector:
enabled: false
enabled: true
imageConfig:
repository: "nvcr.io/nvidia/nvcf-byoc/nvcf-otel-collector"
tag: "0.143.2"
Expand Down
27 changes: 27 additions & 0 deletions deploy/stacks/nvcf-compute-plane/tests/compare-golden.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail

expected_dir="${1:?expected golden directory is required}"
actual_dir="${2:?actual manifest directory is required}"
work_dir="$(mktemp -d)"
trap 'rm -rf "$work_dir"' EXIT

index_tree() {
local root_dir="$1"
local output_file="$2"

find "$root_dir" -type f -print |
while IFS= read -r file; do
relative_path="${file#"$root_dir"/}"
normalized_path="$(
printf '%s\n' "$relative_path" |
sed -E 's#^([^/]+)-[0-9a-f]{8}-#\1-HASH-#'
)"
printf '%s %s\n' "$(git hash-object "$file")" "$normalized_path"
done |
sort >"$output_file"
}

index_tree "$expected_dir" "$work_dir/expected"
index_tree "$actual_dir" "$work_dir/actual"
diff -u "$work_dir/expected" "$work_dir/actual"
105 changes: 105 additions & 0 deletions deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
set -euo pipefail

stack_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
work_dir="$(mktemp -d)"
cluster_name="observability-profile-test-$$"
registration_values="$stack_dir/out/$cluster_name-register-values.yaml"
trap 'rm -rf "$work_dir"; rm -f "$registration_values"' EXIT

fail() {
echo "observability-profile: $*" >&2
exit 1
}

mkdir -p "$stack_dir/out"
cp "$stack_dir/testdata/registration/ncp-local-register-values.yaml" \
"$registration_values"

render_values() {
local profile="$1"
local output_file="$2"
local profile_args=()
shift 2

if [[ "$profile" != "default" ]]; then
profile_args=(--state-values-set "observability.profile=$profile")
fi

HELMFILE_ENV=base \
CLUSTER_NAME="$cluster_name" \
NCA_ID=nvcf-default \
helmfile \
--file "$stack_dir/helmfile.d/02-nvca.yaml.gotmpl" \
--environment default \
"${profile_args[@]}" \
"$@" \
--selector name=nvca-operator \
write-values \
--output-file-template "$output_file" >/dev/null
}

collector_enabled() {
awk '
/^selfManaged:$/ { self_managed = 1; next }
self_managed && /^ otelCollector:$/ { collector = 1; next }
collector && /^ enabled:/ { print $2; exit }
' "$1"
}

has_byoo_gate() {
grep -q '^[[:space:]]*- BYOObservability$' "$1"
}

for profile in default disabled control compute all; do
values="$work_dir/$profile.yaml"
render_values "$profile" "$values"

case "$profile" in
default|compute|all)
test "$(collector_enabled "$values")" = "true" ||
fail "$profile profile did not enable the NVCA collector"
has_byoo_gate "$values" ||
fail "$profile profile did not enable BYOObservability"
;;
disabled|control)
test "$(collector_enabled "$values")" = "false" ||
fail "$profile profile enabled the NVCA collector"
if has_byoo_gate "$values"; then
fail "$profile profile enabled BYOObservability"
fi
;;
esac
done

render_values compute "$work_dir/compute-overrides.yaml" \
--state-values-set global.nvcaOperator.selfManaged.otelCollector.enabled=false \
--state-values-set-string 'global.nvcaOperator.selfManaged.featureGateValues[0]=-BYOObservability'

test "$(collector_enabled "$work_dir/compute-overrides.yaml")" = "false" ||
fail "explicit collector disable did not override the compute profile"
if has_byoo_gate "$work_dir/compute-overrides.yaml"; then
fail "explicit BYOObservability disable did not override the compute profile"
fi
grep -q '^[[:space:]]*- -BYOObservability$' \
"$work_dir/compute-overrides.yaml" ||
fail "explicit BYOObservability disable was not preserved"

render_values control "$work_dir/control-overrides.yaml" \
--state-values-set global.nvcaOperator.selfManaged.otelCollector.enabled=true \
--state-values-set-string 'global.nvcaOperator.selfManaged.featureGateValues[0]=BYOObservability'
Comment thread
coderabbitai[bot] marked this conversation as resolved.

test "$(collector_enabled "$work_dir/control-overrides.yaml")" = "true" ||
fail "explicit collector enable did not override the control profile"
has_byoo_gate "$work_dir/control-overrides.yaml" ||
fail "explicit BYOObservability enable did not override the control profile"

if render_values invalid "$work_dir/invalid.yaml" \
>"$work_dir/invalid.log" 2>&1; then
fail "invalid observability profile was accepted"
fi
grep -q 'observability.profile must be disabled, control, compute, or all' \
"$work_dir/invalid.log" ||
fail "invalid profile did not return the expected error"

echo "observability-profile: all checks passed"
Loading
Loading