From e693cc71693de23a746702554b67316bd48e0dfe Mon Sep 17 00:00:00 2001 From: Aaron Gonzales Date: Mon, 3 Aug 2026 22:16:32 +0000 Subject: [PATCH 1/5] docs: add container deployment guides Signed-off-by: Aaron Gonzales --- docs/user-guide/docker.md | 388 ++++++--------------- docs/user-guide/getting-started.md | 24 +- docs/user-guide/kubernetes.md | 200 +++++++++++ docs/user-guide/private-workload-images.md | 99 ++++++ mkdocs.yml | 2 + 5 files changed, 423 insertions(+), 290 deletions(-) create mode 100644 docs/user-guide/kubernetes.md create mode 100644 docs/user-guide/private-workload-images.md diff --git a/docs/user-guide/docker.md b/docs/user-guide/docker.md index 3c83c092a..d2e670be0 100644 --- a/docs/user-guide/docker.md +++ b/docs/user-guide/docker.md @@ -3,336 +3,162 @@ # Running in Docker -Run the full Safe Synthesizer pipeline in a container with GPU access. -No local Python install required -- the container ships everything needed -for training, generation, and evaluation. +Run the Safe Synthesizer CLI from the published GPU runtime image. You do not +need a source checkout or local Python installation. The image contains the +installed runtime; it does **not** contain input data or a workload +configuration. Supply those at runtime. ---- +## Select an image tag -## Prerequisites - -- Docker 20.10+ (BuildKit enabled by default in 23.0+) -- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed and configured -- NVIDIA driver compatible with the CUDA libraries installed by the image - variant (`cu129` today) -- NVIDIA GPU (A100 or better recommended) - -Verify GPU access works: +The public image is: -```bash -docker run --rm --gpus all nvidia/cuda:12.9.1-base-ubuntu22.04 nvidia-smi +```text +ghcr.io/nvidia-nemo/safe-synthesizer ``` ---- - -## Quick Start - -The container wraps the `safe-synthesizer` CLI. Mount your data and -Hugging Face cache, then pass CLI arguments after the image name: +Use `latest-cu129` to evaluate the current CUDA 12.9 release: ```bash -docker run --gpus all --shm-size=1g \ - -v /path/to/your/data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - nss-gpu:latest \ - run --config /workspace/data/config.yaml --data-source /workspace/data/input.csv +docker pull ghcr.io/nvidia-nemo/safe-synthesizer:latest-cu129 ``` -The entrypoint prints helpful warnings if it detects common mistakes -(empty `/workspace`, missing `HF_HOME`, no GPU access). - -More examples: +For a reproducible workload, replace that tag with an approved versioned +`-cu129` tag or, preferably, pin the resolved manifest digest: -```bash -# Train only -docker run --gpus all --shm-size=1g \ - -v /path/to/data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - nss-gpu:latest run train --data-source /workspace/data/input.csv - -# Generate from a trained adapter -docker run --gpus all --shm-size=1g \ - -v /path/to/data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - nss-gpu:latest run generate --data-source /workspace/data/input.csv --auto-discover-adapter - -# Validate a config file (no GPU needed) -docker run \ - -v /path/to/data:/workspace/data \ - nss-gpu:latest config validate --config /workspace/data/config.yaml +```text +ghcr.io/nvidia-nemo/safe-synthesizer:-cu129 +ghcr.io/nvidia-nemo/safe-synthesizer@sha256: ``` ---- - -## Mounting Your Data +Release tags can be easier to audit; a digest identifies immutable image +content. Keep the `cu129` suffix when selecting a version tag because it names +the CUDA dependency variant. -The container starts with an empty `/workspace`. You bring your own data -by bind-mounting host directories with `-v`: - -```bash -docker run --gpus all --shm-size=1g \ - -v /home/user/project:/workspace/data \ - ... - nss-gpu:latest run --data-source /workspace/data/input.csv -``` - -Docker requires absolute paths for bind mounts. Relative paths like -`-v data:/workspace/data` are silently interpreted as named volumes -- -Docker won't error, but you'll get an empty mount instead of your host -directory. Use `$(pwd)` to expand relative paths: +## Prerequisites -```bash --v $(pwd)/my_data:/workspace/data # correct --v my_data:/workspace/data # wrong -- Docker treats this as a named volume -``` +- Docker with GPU support +- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) + installed and configured +- An NVIDIA driver compatible with the image's CUDA 12.9 libraries +- An NVIDIA GPU (A100 or larger recommended) -You can mount multiple directories at different paths: +Verify that Docker can expose the GPU before running the workload: ```bash -docker run --gpus all \ - -v /data/inputs:/workspace/inputs \ - -v /data/configs:/workspace/configs \ - -v /data/output:/workspace/output \ - -e NSS_ARTIFACTS_PATH=/workspace/output \ - ... - nss-gpu:latest run --config /workspace/configs/my_config.yaml --data-source /workspace/inputs/data.csv +docker run --rm --gpus all nvidia/cuda:12.9.1-base-ubuntu22.04 nvidia-smi ``` -Artifacts are written to `/workspace/safe-synthesizer-artifacts/` by default -(override with `NSS_ARTIFACTS_PATH`). Make sure to mount a host directory -there if you want to retrieve results after the container exits. - ---- - -## Secrets and API Keys +## Quick start -Pass secrets as environment variables at runtime -- never bake them into the -image. The most common ones: +Create host directories for artifacts and the Hugging Face cache, ensure the +user running Docker can write to them, and use absolute mount paths: ```bash -docker run --gpus all --shm-size=1g \ - -v /path/to/data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - -e HF_TOKEN="hf_..." \ - nss-gpu:latest run --data-source /workspace/data/input.csv -``` - -| Variable | Required | Purpose | -|----------|----------|---------| -| `HF_TOKEN` | For gated models | Hugging Face token for downloading gated models (Llama, Mistral, etc.). Get one at [hf.co/settings/tokens](https://huggingface.co/settings/tokens) | -| `NSS_INFERENCE_KEY` | For PII classification | API key for `NSS_INFERENCE_ENDPOINT`. Set when using the CLI/SDK for column classification | -| `NSS_INFERENCE_ENDPOINT` | For PII classification | NIM/OpenAI-compatible endpoint URL (default: `https://integrate.api.nvidia.com/v1`). Override for a custom endpoint | -| `WANDB_API_KEY` | For experiment tracking | WandB API key. Only needed when `--wandb-mode online` is used | - -If `HF_TOKEN` is already stored in your HF cache (`~/.cache/huggingface/token`), -mounting the cache directory is sufficient -- the Hub library reads the token -file automatically. - -See [Environment Variables](environment.md) for the full reference. - ---- +mkdir -p /path/to/artifacts /path/to/hf-cache -## Hugging Face Model Cache - -Safe Synthesizer downloads models from Hugging Face Hub on first use. -Mount a host directory to persist downloads across container runs: - -```bash -docker run --gpus all \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - ... +docker run --rm --gpus all --shm-size=1g \ + --user "$(id -u):$(id -g)" \ + -v /path/to/input:/workspace/input:ro \ + -v /path/to/config:/workspace/config:ro \ + -v /path/to/artifacts:/workspace/artifacts \ + -v /path/to/hf-cache:/workspace/.hf_cache \ + --env HF_TOKEN \ + ghcr.io/nvidia-nemo/safe-synthesizer:latest-cu129 \ + run --config /workspace/config/config.yaml \ + --data-source /workspace/input/input.csv \ + --artifact-path /workspace/artifacts ``` -| Host path | Container path | Env var | Purpose | -|-----------|---------------|---------|---------| -| `~/.cache/huggingface` | `/workspace/.hf_cache` | `HF_HOME` | Model weights, tokenizers, configs | - -The image sets `HF_HOME=/workspace/.hf_cache`, so mounting the cache there is -enough -- passing `-e HF_HOME=/workspace/.hf_cache` is redundant but harmless. -Override the variable only to point at a different container path. - -Without this mount, models are downloaded into the container's ephemeral -filesystem and lost when it exits. +Replace the paths and filenames with your own. Omit `--env HF_TOKEN` when the +selected models do not require it or when an approved token already exists in +the mounted Hugging Face cache. The inherited entrypoint passes everything +after the image reference to `safe-synthesizer` and warns about common mount, +cache, GPU, token, and shared-memory problems. -For shared environments (team servers, CI), point at a shared cache: +See [Running Safe Synthesizer](running.md) for other stages and CLI options and +[Configuration](configuration.md) for the YAML schema and override precedence. -```bash --v /shared/hf_cache:/workspace/.hf_cache -e HF_HOME=/workspace/.hf_cache -``` +## Runtime mounts and persistence ---- +Docker bind mounts preserve host ownership. The image normally runs as +`appuser` with uid and gid 1000; the quick start uses `--user` to match the +host owner of writable mounts. In managed environments, you can instead +provision artifact and cache directories writable by uid/gid 1000. -## GPU Access +| Content | Container path | Access | Lifecycle | +|---------|----------------|--------|-----------| +| Input data | `/workspace/input` | Read-only | Supplied by the user; never shipped in the image or repository | +| YAML configuration | `/workspace/config` | Read-only | Supplied by the user | +| Run artifacts | `/workspace/artifacts` | Read-write | Persist to retain adapters, generated data, reports, and logs | +| Hugging Face cache | `/workspace/.hf_cache` | Read-write | Persist to reuse downloaded models | -The image declares `NVIDIA_VISIBLE_DEVICES=all` and -`NVIDIA_DRIVER_CAPABILITIES=compute,utility`, so the NVIDIA Container Toolkit -knows it needs GPU access. You still need `--gpus` to tell Docker to inject -the GPU devices: +The image sets `HF_HOME=/workspace/.hf_cache`. It also starts in `/workspace` +and defaults artifacts to a relative `safe-synthesizer-artifacts` directory, +so pass the explicit `/workspace/artifacts` path whenever you mount a dedicated +artifact volume. See [Running -- Artifacts and Output](running.md#artifacts-and-output) +for the output tree and [Environment Variables](environment.md) for cache, +offline, logging, endpoint, and artifact settings. -```bash -# All GPUs -docker run --gpus all ... +Docker treats a relative source such as `-v data:/workspace/input` as a named +volume. Use an absolute host path or expand one with `$(pwd)`. -# Specific GPUs -docker run --gpus '"device=0,1"' ... -``` +## Secrets -To restrict which GPUs are visible inside the container, override the -environment variable: +Inject credentials only at runtime. For example, export an approved value in +the calling shell and pass its name without putting the value in shell history: ```bash -docker run --gpus all -e NVIDIA_VISIBLE_DEVICES=0,1 ... +export HF_TOKEN="" +docker run --rm --gpus all --env HF_TOKEN ... ``` ---- - -## Shared Memory (`--shm-size`) - -PyTorch uses `/dev/shm` for inter-process communication during training -(multi-worker data loading). Docker defaults to 64 MB, which causes -"Bus error" crashes. Always pass `--shm-size=1g` (or `--ipc=host`) when -running training workloads: - -```bash -docker run --gpus all --shm-size=1g ... -``` +Other workflows can require `NSS_INFERENCE_KEY` or `WANDB_API_KEY`. Do not bake +credentials into an image. The complete variables and their purposes are in +[Environment Variables](environment.md). -The entrypoint script warns if `/dev/shm` is below 256 MB. -Generation-only runs are typically fine without it. - ---- - -## File Permissions - -The container runs as `appuser` (uid 1000). When bind-mounting host -directories, Docker preserves host ownership. If your host user has a -different uid, writes to the mounted directory (artifacts, outputs) will -fail with "Permission denied". A mounted Hugging Face cache is affected the -same way: already-cached models still load, but downloading a new one fails. - -Fix by matching the container user to your host uid: - -```bash -docker run --gpus all --user "$(id -u):$(id -g)" \ - -v /path/to/data:/workspace/data \ - ... -``` +## GPU Access -This overrides `appuser` with your host identity. The `--user` flag also -works with the dev image and interactive shells. +The image declares NVIDIA runtime visibility and compute capabilities, but +Docker still needs `--gpus all` (or an explicit device selection). Training +uses `/dev/shm` for worker communication; use `--shm-size=1g` as a starting +point and size it for your workload. The entrypoint warns below 256 MiB. ---- +GPU, CPU, memory, and shared-memory requirements vary with the model, dataset, +and configuration. See [Program Runtime](troubleshooting.md) for GPU, OOM, +permissions, cache, and offline failures. ## Offline and Air-Gapped Environments -Pre-cache models by running the pipeline once with internet access, then -reuse the populated cache in the target environment: - -```bash -# Step 1: populate cache (internet required) -docker run --gpus all --shm-size=1g \ - -v /path/to/data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - nss-gpu:latest run --config /workspace/data/config.yaml --data-source /workspace/data/input.csv - -# Step 2: use in offline environment -docker run --gpus all --shm-size=1g \ - -v /path/to/data:/workspace/data \ - -v /shared/hf_cache:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - -e HF_HUB_OFFLINE=1 \ - nss-gpu:latest run --config /workspace/data/config.yaml --data-source /workspace/data/input.csv -``` - -See [Environment Variables -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline) -for details on `HF_HOME`, `HF_HUB_OFFLINE`, and `VLLM_CACHE_ROOT`. - ---- - -## Building from Source - -If pulling a pre-built image is not available, build locally: +Populate a persistent model cache in an approved connected environment, move +or attach it according to organizational policy, and mount it at +`/workspace/.hf_cache`. Then add `--env HF_HUB_OFFLINE=1`. Required models must +already exist in the cache. See +[Environment -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline) +for the complete offline contract. -```bash -mise run container:build:gpu # runtime image -mise run container:build:gpu-dev # dev image with test tooling -``` +## Building the project image from source -Override build arguments for a different package extra, image variant, or -Python slim base version: +Consuming the public image above is the normal user path. Building the project +image is a separate developer workflow that requires a source checkout and +produces local tags rather than pulling the published runtime: ```bash -docker build -f containers/Dockerfile.cuda \ - --build-arg CONTAINER_EXTRA=cu129 \ - --build-arg CONTAINER_VARIANT=cu129 \ - --build-arg PYTHON_VERSION=3.12 \ - --target runtime -t nss-gpu:custom . +mise run container:build:gpu +mise run container:build:gpu-dev ``` -See [Developer Guide -- Docker](../developer-guide/docker.md) for -build stages, ARG reference, and customization details. - ---- - -## Interactive Shell - -To explore the container or debug issues, override the entrypoint to get -a bash shell. Mount your data the same way as a normal run: - -```bash -docker run -it --gpus all --shm-size=1g \ - -v $(pwd)/my_data:/workspace/data \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - --entrypoint /bin/bash \ - nss-gpu:latest -``` - -Inside the container you can run `safe-synthesizer` commands directly: - -```bash -appuser@container:/workspace$ safe-synthesizer run --data-source /workspace/data/input.csv -appuser@container:/workspace$ safe-synthesizer config validate --config /workspace/data/config.yaml -``` - ---- - -## Mise Container Tasks - -For developers with the repo checked out, mise provides convenience tasks -that handle GPU flags, HF cache mounts, and workspace bind mounts: - -| Command | What it does | -|---------|-------------| -| `mise run container:build:gpu` | Build the runtime image | -| `CMD="run --config ..." mise run container:run:gpu` | Run a pipeline command | -| `mise run container:build:gpu-dev` | Build the dev image | -| `CMD="mise run test" mise run container:run:gpu-dev` | Run a command in the dev container | - -Override variables as needed: - -```bash -CONTAINER_HF_CACHE=/shared/hf_cache CMD="run --data-source /workspace/data.csv" mise run container:run:gpu -``` - -Mount data from outside the repo tree with `CONTAINER_EXTRA_MOUNTS`: - -```bash -CONTAINER_EXTRA_MOUNTS="-v /data/sensitive:/workspace/data" \ - CMD="run --data-source /workspace/data/customers.csv" \ - mise run container:run:gpu -``` +See [Developer Guide -- Docker](../developer-guide/docker.md) for build stages, +arguments, and developer-image behavior. Those internals do not change the +public-image consumption contract on this page. ---- +## Other deployment paths -## What to Read Next +- [Kubernetes Job](kubernetes.md) translates this workflow to a portable + `batch/v1` Job. +- [Private Workload Images](private-workload-images.md) explains how to derive + a governed image from an immutable public base while keeping sensitive data, + artifacts, and caches external by default. -- [Running Safe Synthesizer](running.md) -- pipeline execution, CLI commands -- [Configuration Reference](configuration.md) -- parameter tables -- [Environment Variables](environment.md) -- `HF_HOME`, `NSS_ARTIFACTS_PATH`, logging -- [Troubleshooting](troubleshooting.md) -- OOM fixes, offline errors +For the shared runtime contract, continue with +[Running Safe Synthesizer](running.md), [Configuration](configuration.md), +[Environment Variables](environment.md), or [Program Runtime](troubleshooting.md). diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md index 7fc4aea49..a90fd6eb5 100644 --- a/docs/user-guide/getting-started.md +++ b/docs/user-guide/getting-started.md @@ -100,17 +100,23 @@ indexes outside PyPI. You must pass the extra index URLs shown below. === "Docker (Linux with NVIDIA GPU)" ```bash - mise run container:build:gpu - - docker run --gpus all --shm-size=1g \ - -v $(pwd):/workspace \ - -v ~/.cache/huggingface:/workspace/.hf_cache \ - -e HF_HOME=/workspace/.hf_cache \ - nss-gpu:latest run --config /workspace/config.yaml --data-source /workspace/data.csv + docker run --rm --gpus all --shm-size=1g \ + --user "$(id -u):$(id -g)" \ + -v /path/to/input:/workspace/input:ro \ + -v /path/to/config:/workspace/config:ro \ + -v /path/to/artifacts:/workspace/artifacts \ + -v /path/to/hf-cache:/workspace/.hf_cache \ + ghcr.io/nvidia-nemo/safe-synthesizer:latest-cu129 \ + run --config /workspace/config/config.yaml \ + --data-source /workspace/input/input.csv \ + --artifact-path /workspace/artifacts ``` - No local Python install needed. See [Docker](docker.md) for full - setup, volume mounts, and offline usage. + The public image contains the runtime, not input data or configuration. + `latest-cu129` is suitable for evaluation; select an approved versioned + `cu129` tag or digest for reproducible workloads. No local Python install + or source build is needed. See [Docker](docker.md) for tag selection, + directory preparation, secrets, mounts, and offline usage. === "Bare package for config definitions" diff --git a/docs/user-guide/kubernetes.md b/docs/user-guide/kubernetes.md new file mode 100644 index 000000000..dfedfd1e6 --- /dev/null +++ b/docs/user-guide/kubernetes.md @@ -0,0 +1,200 @@ + + + +# Running as a Kubernetes Job + +The published Safe Synthesizer runtime can run the same CLI workflow as a +Kubernetes `batch/v1` Job. The project does not ship input data, workload +configuration, Kubernetes resources, a Helm chart, or an operator. The +template below is a portable starting point to adapt to your platform. + +Start with [Docker -- Select an image tag](docker.md#select-an-image-tag). Use +an approved versioned `cu129` tag or, preferably, an immutable digest for a +repeatable Job. + +## Platform prerequisites + +Your platform team owns: + +- NVIDIA GPU Operator or device-plugin installation and compatible GPU drivers +- StorageClass and PVC provisioning, access modes, retention policy, and input + data upload +- Node selection, taints/tolerations, admission policy, and any RuntimeClass +- Registry authentication and image-pull Secrets when required +- CPU, GPU, memory, ephemeral storage, and `/dev/shm` sizing +- Any Helm packaging or higher-level workload orchestration + +Safe Synthesizer does not claim a supported Kubernetes chart or operator. GPU, +storage, security-context, and scheduling behavior varies by cluster; validate +the template with the platform owner. + +Before creating the Job, provision these names in its namespace: + +- `safe-synthesizer-input`: PVC containing the input file +- `safe-synthesizer-artifacts`: writable PVC for all run outputs +- `safe-synthesizer-hf-cache`: writable PVC for Hugging Face downloads +- `safe-synthesizer-config`: ConfigMap with a `config.yaml` key +- `safe-synthesizer-runtime`: Secret with the runtime credential keys your + configuration needs + +The ConfigMap is for non-secret YAML only. Do not put tokens in it. The +[Configuration](configuration.md) and [Environment Variables](environment.md) +pages define configuration and credential requirements without binding them to +a particular secret-management system. + +## Portable Job template + +Replace the image placeholder with an approved versioned or digest-pinned +`cu129` reference, and replace the input filename. This manifest deliberately +references resources that already exist; it does not provision storage or +include data/configuration content. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: safe-synthesizer +spec: + backoffLimit: 1 + template: + metadata: + labels: + app.kubernetes.io/name: safe-synthesizer + spec: + restartPolicy: Never + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: safe-synthesizer + image: ghcr.io/nvidia-nemo/safe-synthesizer@sha256: + imagePullPolicy: IfNotPresent + args: + - run + - --config + - /workspace/config/config.yaml + - --data-source + - /workspace/input/input.csv + - --artifact-path + - /workspace/artifacts + env: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: safe-synthesizer-runtime + key: HF_TOKEN + optional: true + - name: NSS_INFERENCE_KEY + valueFrom: + secretKeyRef: + name: safe-synthesizer-runtime + key: NSS_INFERENCE_KEY + optional: true + resources: + limits: + nvidia.com/gpu: 1 + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + volumeMounts: + - name: input + mountPath: /workspace/input + readOnly: true + - name: config + mountPath: /workspace/config + readOnly: true + - name: artifacts + mountPath: /workspace/artifacts + - name: hf-cache + mountPath: /workspace/.hf_cache + - name: dshm + mountPath: /dev/shm + volumes: + - name: input + persistentVolumeClaim: + claimName: safe-synthesizer-input + readOnly: true + - name: config + configMap: + name: safe-synthesizer-config + defaultMode: 0444 + - name: artifacts + persistentVolumeClaim: + claimName: safe-synthesizer-artifacts + - name: hf-cache + persistentVolumeClaim: + claimName: safe-synthesizer-hf-cache + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 1Gi +``` + +The image already supplies its non-root CLI entrypoint, so the Job sets +`args`, not `command`. Pod- and container-level security contexts establish a +uid/gid/fsGroup 1000 baseline that matches the image. A storage driver or +cluster policy can interpret ownership and read-only requests differently; +confirm that the two writable PVCs permit writes by uid/gid 1000. + +The `nvidia.com/gpu: 1` limit requests one GPU when the cluster advertises that +resource. It does not install or configure GPU support. Likewise, the +memory-backed `emptyDir` provides `/dev/shm`, but its capacity and accounting +are cluster-dependent. Add CPU and memory requests/limits based on measured +workload needs; do not treat this example's omission as sizing guidance. + +## Submit and observe + +Save your adapted manifest outside this repository, submit it with your normal +deployment process, and inspect status and logs: + +```bash +kubectl apply -f safe-synthesizer-job.yaml +kubectl get job safe-synthesizer +kubectl get pods -l job-name=safe-synthesizer +kubectl logs --follow job/safe-synthesizer +kubectl describe job safe-synthesizer +``` + +For logs intended for aggregation, set `NSS_LOG_FORMAT=json` through your +platform's environment policy. See [Running -- Logging and Experiment Tracking](running.md#logging-and-experiment-tracking) +for log behavior and [Program Runtime](troubleshooting.md) for failures. Job and +Pod conditions reflect Kubernetes scheduling/process state; Safe Synthesizer +results remain in the artifact PVC. + +## Persistence and follow-on generation + +The artifact PVC retains the resolved configuration, training adapter, +generated data, evaluation report, metrics, and run logs described in +[Running -- Artifacts and Output](running.md#artifacts-and-output). The cache +PVC avoids downloading Hugging Face models for every Job. Whether PVC data +survives Job or namespace deletion depends on platform retention policy. + +For a follow-on generation Job, create a new Job name, mount the same artifact +and cache PVCs, and replace `args` with an exact persisted run path: + +```yaml +args: + - run + - generate + - --run-path + - /workspace/artifacts/---/ +``` + +An exact run path avoids ambiguity when a PVC holds multiple trained runs. See +[Running Safe Synthesizer](running.md) for generation options and +[Configuration](configuration.md) for workload tuning. + +## Related deployment paths + +- [Docker](docker.md) is the canonical public-image runtime contract. +- [Private Workload Images](private-workload-images.md) describes governed + derived images that can replace the public image reference in this Job. +- [Environment Variables](environment.md) covers cache, offline, endpoint, + logging, and artifact settings. +- [Program Runtime](troubleshooting.md) covers runtime and resource failures. diff --git a/docs/user-guide/private-workload-images.md b/docs/user-guide/private-workload-images.md new file mode 100644 index 000000000..66c84cee9 --- /dev/null +++ b/docs/user-guide/private-workload-images.md @@ -0,0 +1,99 @@ + + + +# Private Workload Images + +Runtime mounts are the default and recommended way to supply sensitive input +data and configuration. In a governed environment, an organization can instead +derive a private workload image containing an intentionally approved, +non-secret snapshot. The derived image uses the same Docker and Kubernetes +runtime contract as the published image. + +## Pin the published base + +Start `FROM` the published runtime at an approved immutable digest. Do not use +`latest-cu129` as the base of a reproducible derived image. Resolve and review +the digest for the selected release through your registry tooling, then record +it directly in the Dockerfile: + +```dockerfile +# syntax=docker/dockerfile:1 + +FROM ghcr.io/nvidia-nemo/safe-synthesizer@sha256: + +COPY --chown=1000:1000 --chmod=0440 governed/config.yaml /workspace/workload/config.yaml +COPY --chown=1000:1000 --chmod=0440 governed/input.csv /workspace/workload/input.csv +``` + +This Dockerfile intentionally has no `USER`, `ENTRYPOINT`, or `CMD` +instruction. It therefore preserves the published runtime's non-root +`appuser` (uid/gid 1000), `tini`-wrapped CLI entrypoint, and default command. +`--chown=1000:1000` makes the governed files readable by that inherited user. +The filenames are schematic: the repository does not provide a dataset or +workload configuration. + +Build and push the derived image only to an approved private registry, then +capture its digest for deployment: + +```bash +docker build -t registry.example.com/team/safe-synthesizer-workload: . +docker push registry.example.com/team/safe-synthesizer-workload: +``` + +Registry login, access controls, scanning, signing/attestation, retention, and +promotion are organization-owned controls. + +## Understand layer exposure + +`COPY` makes the snapshot part of the image content. Anyone who can pull the +image may be able to extract it. The content can also persist in local build +caches, registry layers and replicas, vulnerability-scanning systems, backups, +and downstream images for as long as their separate retention policies allow. +Deleting the file in a later layer does not remove it from earlier layers. + +Use this pattern only when governance explicitly approves that distribution +and retention boundary. For sensitive or frequently changing data and +configuration, keep using read-only runtime mounts as shown in +[Docker](docker.md) and [Kubernetes Job](kubernetes.md). + +Never embed: + +- credentials, private keys, Hugging Face tokens, inference API keys, or other + secrets +- Hugging Face, vLLM, Triton, or other model/runtime caches +- artifacts, generated data, evaluation reports, logs, or trained adapters + +Inject credentials through runtime secret mechanisms. Keep artifact and model +caches on external writable volumes with an organization-defined lifecycle. +See [Environment Variables](environment.md) for their paths and variables. + +## Run the derived image + +For Docker, replace the public image reference in the canonical command and +keep writable artifact and cache mounts external: + +```bash +docker run --rm --gpus all --shm-size=1g \ + --user "$(id -u):$(id -g)" \ + -v /path/to/artifacts:/workspace/artifacts \ + -v /path/to/hf-cache:/workspace/.hf_cache \ + registry.example.com/team/safe-synthesizer-workload@sha256: \ + run --config /workspace/workload/config.yaml \ + --data-source /workspace/workload/input.csv \ + --artifact-path /workspace/artifacts +``` + +For Kubernetes, replace `image` in the [Job template](kubernetes.md#portable-job-template) +with the derived digest. If the approved snapshot supplies both files, remove +only the input/config volume mounts and volumes; retain writable artifact and +HF-cache PVCs, runtime Secrets, GPU limit, `/dev/shm`, and security contexts. + +The CLI workflow and outputs do not change. Use +[Running Safe Synthesizer](running.md) for execution and artifacts, +[Configuration](configuration.md) for parameters, +[Environment Variables](environment.md) for infrastructure settings, and +[Program Runtime](troubleshooting.md) for failure handling. + +For the Safe Synthesizer project's own multi-stage Dockerfile and publication +mechanics—not for workload-image policy—see +[Developer Guide -- Docker](../developer-guide/docker.md). diff --git a/mkdocs.yml b/mkdocs.yml index 8b7864102..7f5ef5d7a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -181,6 +181,8 @@ nav: - Getting Started: user-guide/getting-started.md - Running Safe Synthesizer: user-guide/running.md - Docker: user-guide/docker.md + - Kubernetes Job: user-guide/kubernetes.md + - Private Workload Images: user-guide/private-workload-images.md - Troubleshooting: - Synthetic Data Quality: user-guide/evaluating-data.md - Program Runtime: user-guide/troubleshooting.md From d4ee40ba3ca76e4440d903c36c41b3b8eacc3a7e Mon Sep 17 00:00:00 2001 From: Aaron Gonzales Date: Mon, 3 Aug 2026 22:27:42 +0000 Subject: [PATCH 2/5] docs: restore Docker debugging shell Signed-off-by: Aaron Gonzales --- docs/user-guide/docker.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/user-guide/docker.md b/docs/user-guide/docker.md index d2e670be0..63e00e3fd 100644 --- a/docs/user-guide/docker.md +++ b/docs/user-guide/docker.md @@ -127,6 +127,34 @@ GPU, CPU, memory, and shared-memory requirements vary with the model, dataset, and configuration. See [Program Runtime](troubleshooting.md) for GPU, OOM, permissions, cache, and offline failures. +## Debug the runtime image + +For an interactive inspection session, override the entrypoint and start a +shell in the published runtime image. Mount the same input, configuration, +artifact, and cache directories you use for a normal run: + +```bash +docker run --rm -it --gpus all --shm-size=1g \ + --user "$(id -u):$(id -g)" \ + --mount type=bind,src="$(pwd)/input",dst=/workspace/input,readonly \ + --mount type=bind,src="$(pwd)/config",dst=/workspace/config,readonly \ + --mount type=bind,src="$(pwd)/artifacts",dst=/workspace/artifacts \ + --mount type=bind,src="$HOME/.cache/huggingface",dst=/workspace/.hf_cache \ + --entrypoint /bin/bash \ + ghcr.io/nvidia-nemo/safe-synthesizer:latest-cu129 +``` + +Inside the shell, inspect GPU visibility or validate a configuration without +starting a synthesis run: + +```bash +nvidia-smi +safe-synthesizer config validate --config /workspace/config/config.yaml +``` + +Overriding the entrypoint bypasses its startup diagnostics. Use the normal +`docker run` command for actual pipeline runs. + ## Offline and Air-Gapped Environments Populate a persistent model cache in an approved connected environment, move From b5830ef6ebe3ce9363e357c271354cc6e1ca8a3b Mon Sep 17 00:00:00 2001 From: Aaron Gonzales Date: Mon, 3 Aug 2026 22:36:22 +0000 Subject: [PATCH 3/5] docs: clarify published container image Signed-off-by: Aaron Gonzales --- README.md | 5 +++++ containers/README.md | 4 +++- docs/developer-guide/docker.md | 6 ++++-- docs/user-guide/docker.md | 2 +- 4 files changed, 13 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d49f60f53..0b3ddffee 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,11 @@ make setup # installs pinned mise, pinned tools from mise.lock, and .venv mise run bootstrap-nss cuda ``` +### Run the public container + +The published GPU runtime is available from [GitHub Container Registry](https://github.com/NVIDIA-NeMo/Safe-Synthesizer/pkgs/container/safe-synthesizer) at `ghcr.io/nvidia-nemo/safe-synthesizer`. +It runs the CLI without a local Python installation; see the [Docker guide](https://nvidia-nemo.github.io/Safe-Synthesizer/user-guide/docker/) for image tags, mounts, GPU access, and configuration. + Development tools (`dprint`, `ruff`, `ty`, `yq`, `gh`, etc.) are managed via [mise](https://mise.jdx.dev/). Tool versions are declared in `.mise.toml` and locked in `mise.lock` (committed). mise also manages environment variables -- place project-local secrets or overrides in `.env` or `.env.local` (both git-ignored, auto-loaded by mise). For IDEs to discover mise-managed tools, [add mise shims to the `PATH` in your default shell profile](https://mise.jdx.dev/ide-integration.html#adding-shims-to-path-in-your-default-shell). diff --git a/containers/README.md b/containers/README.md index 262c9035a..b1f5381a2 100644 --- a/containers/README.md +++ b/containers/README.md @@ -113,7 +113,9 @@ CONTAINER_GPU_PLATFORM=linux/arm64 mise run container:build:gpu Multi-platform manifests must be pushed to a registry: ```bash -CONTAINER_GPU_REGISTRY=ghcr.io/nvidia-nemo mise run container:build:gpu-multiarch +CONTAINER_GPU_REGISTRY=registry.example.com/team \ +CONTAINER_GPU_IMAGE=safe-synthesizer:custom-cu129 \ + mise run container:build:gpu-multiarch ``` This builds and pushes `$(CONTAINER_GPU_REGISTRY)/$(CONTAINER_GPU_IMAGE)`. diff --git a/docs/developer-guide/docker.md b/docs/developer-guide/docker.md index 5621b68fb..0de4c1e0f 100644 --- a/docs/developer-guide/docker.md +++ b/docs/developer-guide/docker.md @@ -277,7 +277,9 @@ single tag. Clients pull the correct variant automatically. Because directly to a registry: ```bash -CONTAINER_GPU_REGISTRY=ghcr.io/nvidia-nemo mise run container:build:gpu-multiarch +CONTAINER_GPU_REGISTRY=registry.example.com/team \ +CONTAINER_GPU_IMAGE=safe-synthesizer:custom-cu129 \ + mise run container:build:gpu-multiarch ``` This runs: @@ -285,7 +287,7 @@ This runs: ```bash docker buildx build \ --platform linux/amd64,linux/arm64 \ - --tag ghcr.io/nvidia-nemo/nss-gpu:latest \ + --tag registry.example.com/team/safe-synthesizer:custom-cu129 \ --target runtime --push \ -f containers/Dockerfile.cuda . ``` diff --git a/docs/user-guide/docker.md b/docs/user-guide/docker.md index 63e00e3fd..f480eaacf 100644 --- a/docs/user-guide/docker.md +++ b/docs/user-guide/docker.md @@ -10,7 +10,7 @@ configuration. Supply those at runtime. ## Select an image tag -The public image is: +The public image is available from [GitHub Container Registry (GHCR)](https://github.com/NVIDIA-NeMo/Safe-Synthesizer/pkgs/container/safe-synthesizer): ```text ghcr.io/nvidia-nemo/safe-synthesizer From f47ef7ebbb178595c46a9d86c439c21bf74b5028 Mon Sep 17 00:00:00 2001 From: Aaron Gonzales Date: Tue, 4 Aug 2026 02:01:55 +0000 Subject: [PATCH 4/5] docs: clarify runtime credential handling Signed-off-by: Aaron Gonzales --- docs/user-guide/docker.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/docker.md b/docs/user-guide/docker.md index f480eaacf..17a2375e2 100644 --- a/docs/user-guide/docker.md +++ b/docs/user-guide/docker.md @@ -104,12 +104,19 @@ volume. Use an absolute host path or expand one with `$(pwd)`. ## Secrets -Inject credentials only at runtime. For example, export an approved value in -the calling shell and pass its name without putting the value in shell history: +Inject credentials only at runtime. Prefer your organization's approved +credential handler or secret manager when one is available. Follow its existing +standards for secret storage, access, rotation, audit, and runtime injection. + +If an approved local workflow requires an environment variable, read it without +echoing it or placing the value in shell history: ```bash -export HF_TOKEN="" +read -r -s -p "Hugging Face token: " HF_TOKEN +printf '\n' +export HF_TOKEN docker run --rm --gpus all --env HF_TOKEN ... +unset HF_TOKEN ``` Other workflows can require `NSS_INFERENCE_KEY` or `WANDB_API_KEY`. Do not bake From b8c06411625430031d812e0d25073369ea760537 Mon Sep 17 00:00:00 2001 From: Aaron Gonzales Date: Tue, 4 Aug 2026 22:14:12 +0000 Subject: [PATCH 5/5] docs: fix derived workload runtime paths Signed-off-by: Aaron Gonzales --- docs/user-guide/private-workload-images.md | 23 +++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/private-workload-images.md b/docs/user-guide/private-workload-images.md index 66c84cee9..86e23ebc7 100644 --- a/docs/user-guide/private-workload-images.md +++ b/docs/user-guide/private-workload-images.md @@ -74,7 +74,6 @@ keep writable artifact and cache mounts external: ```bash docker run --rm --gpus all --shm-size=1g \ - --user "$(id -u):$(id -g)" \ -v /path/to/artifacts:/workspace/artifacts \ -v /path/to/hf-cache:/workspace/.hf_cache \ registry.example.com/team/safe-synthesizer-workload@sha256: \ @@ -83,10 +82,28 @@ docker run --rm --gpus all --shm-size=1g \ --artifact-path /workspace/artifacts ``` +Do not add the canonical command's `--user` override here: the embedded files +are readable by the inherited `appuser` (uid/gid 1000), not arbitrary host +identities. Provision the artifact and cache directories so uid/gid 1000 can +write to them. + For Kubernetes, replace `image` in the [Job template](kubernetes.md#portable-job-template) with the derived digest. If the approved snapshot supplies both files, remove -only the input/config volume mounts and volumes; retain writable artifact and -HF-cache PVCs, runtime Secrets, GPU limit, `/dev/shm`, and security contexts. +only the input/config volume mounts and volumes, then replace `args` with: + +```yaml +args: + - run + - --config + - /workspace/workload/config.yaml + - --data-source + - /workspace/workload/input.csv + - --artifact-path + - /workspace/artifacts +``` + +Retain writable artifact and HF-cache PVCs, runtime Secrets, GPU limit, +`/dev/shm`, and security contexts. The CLI workflow and outputs do not change. Use [Running Safe Synthesizer](running.md) for execution and artifacts,