nvsnap-server has no authentication and defaults to type: LoadBalancer

[balajinvda]

Found auditing NVCA's agent calls for #734.

The audit result first

NVCA is not affected by the agent auth work in #555. It never calls the agent API. pkg/nvca/nvsnap/client.go targets nvsnap-server only:

const DefaultBaseURL = "http://nvsnap-server.nvsnap-system.svc.cluster.local:8080"

Every method (CreateCheckpoint, GetCheckpoint, LookupCheckpoints, GetPVCPromoteState, DeleteCheckpoint) goes to /api/v1/checkpoints* on :8080. Checked every Go file the #208 branch adds or changes: none references :8081 or nvsnap-agent. There is no nvsnap code in nvca on main at all, so the whole surface is #208, and no change is needed there for --auth-mode=required.

The gap that audit exposed

nvsnap-server itself has no authentication, and it is the component NVCA actually depends on.

• No auth of any kind. The only middleware on the router is CORS (internal/server/server.go:135). No token check, no middleware chain, nothing equivalent to the agent's tokenGuard.
• The chart defaults the Service to type: LoadBalancer (values.yaml, server service.type), i.e. a public IP by default rather than opt-in.
• The API is not read-only: 14 POST, 5 DELETE, 1 PUT routes, including DELETE /api/v1/checkpoints/{id} (deletes a checkpoint, cascading to L1 and the blobstore) and POST /api/v1/demo/deploy (creates pods).
• GET /api/v1/checkpoints/{id}/file serves checkpoint file contents. Checkpoint dumps are process memory images -- a single vllm-small dump on this cluster is 33G, of which pages-11.img is 32G of HBM. Anything resident in a checkpointed process is readable through that route.

The chart ships one NetworkPolicy (network-policy-restore-pods.yaml) and it grants restore pods egress to the server; nothing restricts ingress to it.

Exposure note, stated precisely

On nvcf-dgxc-k8s-aws-usw2-dev2 the Service is LoadBalancer but has no external address, because provisioning is failing for an unrelated reason:

SyncLoadBalancerFailed: Multiple untagged security groups found for instance i-...;
ensure the k8s security group is tagged

So there is no live external exposure on that cluster today. That is an accident of a misconfigured cluster, not a property of the chart. On a correctly tagged cluster the default assigns a public IP to an unauthenticated API that can read checkpoint memory and delete checkpoints.

Why this is worth more than the agent case

#486/#555 secured the agent, whose reachability is a hostPort on the node network. nvsnap-server is a cluster Service that defaults to a cloud load balancer, so its default exposure is strictly wider while its API is at least as powerful.

Suggested direction

1. Default server.service.type to ClusterIP and make LoadBalancer opt-in. This is a one-line change and removes the default public IP independently of the rest.
2. Extend the shared-token scheme from feat(nvsnap): shared-token authentication for the agent API #555 to nvsnap-server, keeping /health and /metrics unauthenticated for probes and scraping. The agent's AuthMode (disabled/permissive/required) and unauthenticatedPaths are directly reusable.
3. NVCA's client needs to carry the token once (2) lands -- pkg/nvca/nvsnap/client.go already has a WithHTTPClient option, so an authTransport like the agent's cascade_fetch.go wrapper drops in without touching call sites.
4. Consider whether the /demo/* routes belong in a build that can be exposed at all.

Ordering matters: (2) before (3) breaks NVCA the same way required broke the scripts in #734, so land the client change first or default the server to permissive.

[balajinvda]

Closing: these are internal services, so application-level auth is not required. Network isolation is the control, and adding a token scheme to nvsnap-server is not worth the coupling it would create with NVCA and the init containers.

Recording the one factual detail so it is not lost if this is revisited: the chart currently defaults server.service.type to LoadBalancer, which requests a public IP rather than keeping the service internal. On nvcf-dgxc-k8s-aws-usw2-dev2 no external address was assigned, but only because LB provisioning fails there for an unrelated reason:

SyncLoadBalancerFailed: Multiple untagged security groups found for instance i-...

That default appears intentional, so the UI and demo endpoints get a reachable address. If so, the internal-only assumption holds for ClusterIP deployments and not for LoadBalancer ones. Anyone exposing this service publicly should know the API includes DELETE /api/v1/checkpoints/{id} and GET /api/v1/checkpoints/{id}/file, which serves checkpoint contents.

No action planned. The agent-side auth in #555 is unaffected and stays as built, since it is already implemented and passing e2e.