Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ class OverridesSpec(BaseModel):

profilingJob: Optional[Dict[str, Any]] = Field(
default=None,
description="ProfilingJob allows overriding the profiling Job specification. Fields set here are merged into the controller-generated Job spec.",
description="ProfilingJob allows overriding the profiling Job specification. Fields set here are merged into the controller-generated Job spec. Security: creating a DGDR is workload-creation authority in its namespace — these overrides carry the same blast radius as creating a Job or Pod directly there, by design. Pod security is enforced centrally by Kubernetes Pod Security Admission on the resulting Pods once the namespace is labeled (see pod-security.kubernetes.io/enforce): it applies the full Pod Security Standards — covering privileged, host namespaces, and hostPath, not only securityContext — not this API. ServiceAccount identity is a separate layer: these overrides can set serviceAccountName and automountServiceAccountToken, which are bounded by RBAC and namespace membership rather than PSA — the same authority any Pod author in the namespace already holds. Grant create/update on DGDRs only to principals trusted to create Pods in the namespace. The profiling Job always runs in the DGDR's own namespace and overrides cannot change that — but namespace containment is not node or cross-tenant isolation. Dynamo's workloads, including this Job, satisfy the baseline standard, so enforce baseline (non-exempt) on every resulting Pod to close the privileged, host-namespace, host-device, and hostPath paths.",
)
trustRemoteCode: bool = Field(
default=False,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,22 @@ type ModelCacheSpec struct {
type OverridesSpec struct {
// ProfilingJob allows overriding the profiling Job specification.
// Fields set here are merged into the controller-generated Job spec.
//
// Security: creating a DGDR is workload-creation authority in its namespace —
Comment thread
dmitry-tokarev-nv marked this conversation as resolved.
// these overrides carry the same blast radius as creating a Job or Pod directly
// there, by design. Pod security is enforced centrally by Kubernetes Pod Security
// Admission on the resulting Pods once the namespace is labeled (see
// pod-security.kubernetes.io/enforce): it applies the full Pod Security Standards —
// covering privileged, host namespaces, and hostPath, not only securityContext —
// not this API. ServiceAccount identity is a separate layer: these overrides can
// set serviceAccountName and automountServiceAccountToken, which are bounded by
// RBAC and namespace membership rather than PSA — the same authority any Pod author
// in the namespace already holds. Grant create/update on DGDRs only to principals
// trusted to create Pods in the namespace. The profiling Job always runs in the
// DGDR's own namespace and overrides cannot change that — but namespace containment
// is not node or cross-tenant isolation. Dynamo's workloads, including this Job,
// satisfy the baseline standard, so enforce baseline (non-exempt) on every resulting
// Pod to close the privileged, host-namespace, host-device, and hostPath paths.
// +optional
ProfilingJob *batchv1.JobSpec `json:"profilingJob,omitempty"`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -776,6 +776,22 @@ spec:
description: |-
ProfilingJob allows overriding the profiling Job specification.
Fields set here are merged into the controller-generated Job spec.

Security: creating a DGDR is workload-creation authority in its namespace —
these overrides carry the same blast radius as creating a Job or Pod directly
there, by design. Pod security is enforced centrally by Kubernetes Pod Security
Admission on the resulting Pods once the namespace is labeled (see
pod-security.kubernetes.io/enforce): it applies the full Pod Security Standards —
covering privileged, host namespaces, and hostPath, not only securityContext —
not this API. ServiceAccount identity is a separate layer: these overrides can
set serviceAccountName and automountServiceAccountToken, which are bounded by
RBAC and namespace membership rather than PSA — the same authority any Pod author
in the namespace already holds. Grant create/update on DGDRs only to principals
trusted to create Pods in the namespace. The profiling Job always runs in the
DGDR's own namespace and overrides cannot change that — but namespace containment
is not node or cross-tenant isolation. Dynamo's workloads, including this Job,
satisfy the baseline standard, so enforce baseline (non-exempt) on every resulting
Pod to close the privileged, host-namespace, host-device, and hostPath paths.
properties:
activeDeadlineSeconds:
description: |-
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,49 @@ replacement behavior and requires the complete desired argument list.
For the complete merge, metadata, and validation rules, see
[DGDR Reference — Generated DGD overrides](../../reference/kubernetes-api/dynamo-graph-deployment-request.mdx#generated-dgd-overrides).

### Profiling job overrides and the trust boundary

`spec.overrides.profilingJob` accepts a partial Kubernetes `JobSpec` that the operator merges
into the profiling Job it launches (for example, to add tolerations or adjust resources). Because
the operator creates that Job on your behalf, treat this the way Kubernetes treats any
workload-creation API:

> [!IMPORTANT]
> Any principal that can create workloads in a namespace where Dynamo runs — a `Pod` directly, or
> any resource that creates Pods (`Job`, `Deployment`, `DynamoGraphDeploymentRequest`, …) — is
> inside that namespace's trust boundary: it can run code with the Secrets and ServiceAccount
> tokens mounted in that namespace. Creating a DGDR is one such path and is no more privileged than
> creating a `Job` or `Pod` there — including through `overrides.profilingJob`. This is by design
> and matches how Kubernetes treats every Pod-spawning resource.
>
> Enforce Pod security **centrally on the resulting Pods** with
> [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/)
> (and any admission webhooks), which applies the full
> [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) —
> not only `securityContext` fields but also `privileged`, host namespaces, `hostPath` volumes, and
> host ports. PSA applies no policy until you label the namespace — set
> `pod-security.kubernetes.io/enforce: <level>` (plus the matching `audit`/`warn` labels) on every
> namespace where Dynamo runs, exactly as you would for any workload. The operator does not
> re-implement those checks.
>
> Dynamo's operator-generated workloads — including the DGDR profiling Job — satisfy the **`baseline`**
> standard. Enforce **`baseline`** to block the privileged-container, host-namespace, host-device, and
> `hostPath` escalation paths while keeping Dynamo running.
Comment on lines +480 to +482

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The profiling Job does satisfy baseline, but "Dynamo's operator-generated workloads" does not: the inter-pod GMS layout mounts a hostPath at /run/gms on the weight-server pod and the engine pods (deploy/operator/internal/dynamo/failover.go:212-229, reached from graph.go:2823 and graph.go:2843 when Experimental.GPUMemoryService.Mode is interpod). Baseline forbids hostPath outright, and the comment at failover.go:243-246 already expects baseline to reject that init container. So an operator who reads "enforce baseline while keeping Dynamo running" and does it non-exempt gets GMS pods rejected at admission.

Suggested change
> Dynamo's operator-generated workloads — including the DGDR profiling Job — satisfy the **`baseline`**
> standard. Enforce **`baseline`** to block the privileged-container, host-namespace, host-device, and
> `hostPath` escalation paths while keeping Dynamo running.
> Dynamo's default operator-generated workloads, including the DGDR profiling Job, satisfy the **`baseline`**
> standard. Enforce **`baseline`** to block the privileged-container, host-namespace, host-device, and
> `hostPath` escalation paths. The experimental inter-pod GMS layout is the exception: it mounts a
> `/run/gms` hostPath volume, so namespaces running it need a more permissive level or a scoped exemption.

>
> PSA governs a Pod's security *posture*, not its *identity*: `overrides.profilingJob` can set the
> Job's `serviceAccountName` and `automountServiceAccountToken`, and those are bounded by RBAC and
> namespace membership, not by PSA — the same authority any Pod author in the namespace already has.
> So grant `create`/`update` on DGDRs — and on workload resources generally — only to principals you
> would trust to create Pods in that namespace, and use namespaces as the tenancy boundary — see
> [Kubernetes RBAC good practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/#workload-creation).

The profiling Job object always remains in the DGDR's own namespace and overrides cannot relocate
it — but namespace containment is not node or cross-tenant isolation. If admission permits
privileged containers, host namespaces, host devices, or `hostPath` mounts, a DGDR creator can
obtain those capabilities through the operator, exactly as any Pod author in the namespace could.
Comment on lines +492 to +494

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one points the other way and overstates the surface by one item. applyPodSpecOverrides (profiling_job_overrides.go:325-382) is an allowlist and never copies HostNetwork, HostPID, or HostIPC, so host namespaces are silently dropped. Privileged and hostPath are reachable, through the wholesale container SecurityContext replace at :423-425 and the Volumes and InitContainers merge at :369-370. Dropping "host namespaces" from the list keeps the rest of the sentence true.

Suggested change
it — but namespace containment is not node or cross-tenant isolation. If admission permits
privileged containers, host namespaces, host devices, or `hostPath` mounts, a DGDR creator can
obtain those capabilities through the operator, exactly as any Pod author in the namespace could.
it — but namespace containment is not node or cross-tenant isolation. If admission permits
privileged containers, host devices, or `hostPath` mounts, a DGDR creator can obtain those
capabilities through the operator, exactly as any Pod author in the namespace could.

Enforcing `baseline` non-exempt on every resulting Pod closes those paths; use namespaces as the
tenancy boundary for anything stronger.

## Next steps

| Goal | Guide |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
> **⚠️ Important**: This documentation is automatically generated from source code.
> Do not edit this file directly.

# API Reference

Check warning on line 10 in docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md

View workflow job for this annotation

GitHub Actions / Docs Lint

docs-lint FRONTMATTER

body `# H1` duplicates the Fern nav-generated title — start the body at `##`

## Packages
- [nvidia.com/v1alpha1](#nvidiacomv1alpha1)
Expand Down Expand Up @@ -2578,7 +2578,7 @@

| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `profilingJob` _[JobSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#jobspec-v1-batch)_ | ProfilingJob allows overriding the profiling Job specification.<br />Fields set here are merged into the controller-generated Job spec. | | Optional: \{\} <br /> |
| `profilingJob` _[JobSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#jobspec-v1-batch)_ | ProfilingJob allows overriding the profiling Job specification.<br />Fields set here are merged into the controller-generated Job spec.<br />Security: creating a DGDR is workload-creation authority in its namespace —<br />these overrides carry the same blast radius as creating a Job or Pod directly<br />there, by design. Pod security is enforced centrally by Kubernetes Pod Security<br />Admission on the resulting Pods once the namespace is labeled (see<br />pod-security.kubernetes.io/enforce): it applies the full Pod Security Standards —<br />covering privileged, host namespaces, and hostPath, not only securityContext —<br />not this API. ServiceAccount identity is a separate layer: these overrides can<br />set serviceAccountName and automountServiceAccountToken, which are bounded by<br />RBAC and namespace membership rather than PSA — the same authority any Pod author<br />in the namespace already holds. Grant create/update on DGDRs only to principals<br />trusted to create Pods in the namespace. The profiling Job always runs in the<br />DGDR's own namespace and overrides cannot change that — but namespace containment<br />is not node or cross-tenant isolation. Dynamo's workloads, including this Job,<br />satisfy the baseline standard, so enforce baseline (non-exempt) on every resulting<br />Pod to close the privileged, host-namespace, host-device, and hostPath paths. | | Optional: \{\} <br /> |
| `trustRemoteCode` _boolean_ | TrustRemoteCode explicitly permits generated vLLM and SGLang workers to<br />execute custom code from the configured model repository. When enabled,<br />the profiler adds --trust-remote-code to every generated worker component<br />after the deployment topology has been generated. Enable this setting only<br />for model repositories you trust. | false | Optional: \{\} <br /> |
| `dgd` _[RawExtension](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg)_ | DGD provides a partial, versioned DynamoGraphDeployment override for the<br />profiler-generated deployment. Set apiVersion to nvidia.com/v1alpha1 or<br />nvidia.com/v1beta1 and kind to DynamoGraphDeployment.<br />The profiler merges the override using the schema for its declared version.<br />If the generated DGD uses another supported version, the complete DGD is<br />converted before the merge and converted back afterward. The final DGD<br />selected or created by a DGDR is nvidia.com/v1beta1.<br />The override can update DGD fields, but topology entries are limited to<br />services or components already present in the generated DGD. Metadata labels<br />and annotations are merged, metadata.name selects the final DGD name, and<br />other identity or runtime metadata is ignored.<br />V1alpha1 worker argument lists retain legacy append behavior. V1beta1 follows<br />structural schema merge behavior, including map-list merging and atomic-list<br />replacement.<br />The raw embedded resource preserves either supported schema. The API server<br />validates that it has apiVersion and kind; override processing validates the<br />DGD kind, supported version, and field schema. | | EmbeddedResource: \{\} <br />Optional: \{\} <br /> |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3689,7 +3689,7 @@ OverridesSpec allows customizing the profiling job and the generated DynamoGraph
**Appears in:** [DynamoGraphDeploymentRequestSpec](#v1beta1-dynamographdeploymentrequestspec)

<ParamField path="profilingJob" type="JobSpec">
ProfilingJob allows overriding the profiling Job specification. Fields set here are merged into the controller-generated Job spec.
ProfilingJob allows overriding the profiling Job specification. Fields set here are merged into the controller-generated Job spec. Security: creating a DGDR is workload-creation authority in its namespace — these overrides carry the same blast radius as creating a Job or Pod directly there, by design. Pod security is enforced centrally by Kubernetes Pod Security Admission on the resulting Pods once the namespace is labeled (see pod-security.kubernetes.io/enforce): it applies the full Pod Security Standards — covering privileged, host namespaces, and hostPath, not only securityContext — not this API. ServiceAccount identity is a separate layer: these overrides can set serviceAccountName and automountServiceAccountToken, which are bounded by RBAC and namespace membership rather than PSA — the same authority any Pod author in the namespace already holds. Grant create/update on DGDRs only to principals trusted to create Pods in the namespace. The profiling Job always runs in the DGDR's own namespace and overrides cannot change that — but namespace containment is not node or cross-tenant isolation. Dynamo's workloads, including this Job, satisfy the baseline standard, so enforce baseline (non-exempt) on every resulting Pod to close the privileged, host-namespace, host-device, and hostPath paths.
See [JobSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#jobspec-v1-batch).
**Validation:** Optional: \&#123;\&#125;
</ParamField>
Expand Down
Loading