BrianNewsom · Aug 4, 2026 · Aug 4, 2026 · Aug 4, 2026
@@ -30,6 +30,7 @@ clusters, use the NeMo Platform Helm chart on Kubernetes.
 | Kubernetes | minikube, kind, EKS, AKS, GKE, OKE, OpenShift, or on-prem Kubernetes | Use a cluster with storage, networking, and image-pull access configured before installing the chart. |
 | Helm | Helm 3 | Required to install and upgrade the NeMo Platform chart. |
 | Storage | ReadWriteMany-capable persistent storage | Required for shared files and job storage in multi-pod deployments. |
+| ClickHouse for Intake | External production service, or the chart's embedded development/testing instance | Embedded ClickHouse creates a separate RWO PVC. Use a separately managed ClickHouse service for production. See [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake). |
 | Image access | NGC credentials and Kubernetes image pull secrets | Required to pull NeMo Platform container images. |
 
 ## GPU Requirements

@@ -2,11 +2,12 @@
 title: "Backup and Restore"
 description: ""
 ---
-NeMo Platform use several storage systems that require backup consideration:
+NeMo Platform uses several storage systems that require backup consideration:
 
 1. **PostgreSQL database**: stores entity metadata and relationships
-2. **Object storage**: stores model files, datasets, and other artifacts
-3. **Persistent Volume Claims (PVC)**: stores Kubernetes persistent data
+2. **ClickHouse database**: stores Intake spans, traces, annotations, and evaluator results
+3. **Object storage**: stores model files, datasets, and other artifacts
+4. **Persistent Volume Claims (PVC)**: stores Kubernetes persistent data
 
 ## PostgreSQL Backup
 
@@ -15,6 +16,13 @@ For PostgreSQL backup and restore procedures, refer to the official PostgreSQL d
 - [PostgreSQL Backup and Restore](https://www.postgresql.org/docs/current/backup.html)
 - [Continuous Archiving and Point-in-Time Recovery (PITR)](https://www.postgresql.org/docs/current/continuous-archiving.html)
 
+## ClickHouse Backup
+
+For ClickHouse backup and restore procedures, refer to the official ClickHouse documentation:
+
+- [Backup and restore in ClickHouse](https://clickhouse.com/docs/concepts/features/backup-restore/overview)
+- [Review and restore ClickHouse Cloud backups](https://clickhouse.com/docs/products/cloud/guides/backups/review-and-restore-backups)
+
 ## Object Storage Backup
 
 Backup procedures for object storage depend on your chosen storage solution:

@@ -80,3 +80,44 @@ Use individual `host`, `port`, `user`, and `database` values and a secret that c
  existingSecret: my-db-password-secret
  existingSecretPasswordKey: "password" # key in the secret that holds the password
  ```
+
+## ClickHouse for Intake
+
+The NeMo Platform uses ClickHouse to store Intake data such as spans, traces, annotations, and evaluator results.
+
+By default, the NeMo Platform Helm chart deploys an embedded ClickHouse instance (a single-node StatefulSet using the official ClickHouse image) to enable quick installation. This should not be used for production use, and an external ClickHouse instance is recommended.
+
+For production, use a managed service such as [ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse separately, such as with the official [ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview).
+
+### External ClickHouse Database
+
+To use an external ClickHouse database in the Helm chart:
+
+1. Set `clickhouse.enabled: false` and configure `externalClickhouse`.
+2. Create a Secret containing the ClickHouse password in the same namespace as the NeMo Platform release:
+
+   ```sh
+   NMP_NAMESPACE="your-release-namespace"
+   CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password"
+   kubectl create secret generic clickhouse-credentials \
+     --namespace "$NMP_NAMESPACE" \
+     --from-file="password=$CLICKHOUSE_PASSWORD_FILE"
+   ```
+
+3. Configure the external HTTP endpoint in `values.yaml`:
+
+   ```yaml
+   clickhouse:
+     enabled: false
+
+   externalClickhouse:
+     host: clickhouse.example.internal
+     port: 8443
+     secure: true
+     user: nemo
+     database: intake
+     existingSecret: clickhouse-credentials
+     existingSecretPasswordKey: password
+   ```
+
+   Set `host` to a hostname only, without a URL scheme or path. Set `secure: true` for HTTPS; the common HTTPS port is `8443`.
@@ -26,7 +26,7 @@ Install the NeMo Platform using the Helm chart.
 </Card>
 <Card title="Database Setup" href="/documentation/self-managed-deployment/setup/helm/database-setup">
 
-Set up an external database for the NeMo Platform.
+Configure external PostgreSQL and ClickHouse databases.
 
 <small><span class="md-tag">cluster-admin</span> <span class="md-tag">on-prem</span> <span class="md-tag">cloud</span></small>
 

@@ -172,6 +172,11 @@ postgresql:
   persistence:
     storageClass: gp3
     size: 20Gi
+
+clickhouse:
+  persistence:
+    storageClass: gp3
+    size: 20Gi
 ```
 
 If you store Files service data in S3, configure the file storage backend and make AWS credentials available through IRSA or another boto3-supported credential source:
@@ -188,6 +193,8 @@ platformConfig:
 
 For multi-node GPU jobs on EKS, also configure EFA networking as described in [Multinode Networking](/documentation/self-managed-deployment/setup/helm/multinode-networking).
 
+For production Intake deployments, use a separately managed ClickHouse service. See [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake).
+
 ## Install Volcano
 
 Install the Volcano scheduler before installing the chart. This is required for customization jobs that leverage multiple nodes.

@@ -69,6 +69,19 @@ Replace `"nfs"` with your StorageClass name (e.g. `oci-nfs`, `gp3`). For NIM sca
 
 Refer to the [platform configuration documentation](/documentation/self-managed-deployment/config-reference) for the full config reference.
 
+## Embedded ClickHouse Storage
+
+The embedded ClickHouse StatefulSet creates a separate ReadWriteOnce volume. Configure it with a low-latency block StorageClass:
+
+```yaml
+clickhouse:
+  persistence:
+    storageClass: fast-rwo
+    size: 20Gi
+```
+
+The embedded deployment is not recommended for production. For production, use an external ClickHouse instance as described in [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake).
+
 ## Persistent volume options
 
 Use a ReadWriteMany-capable filesystem such as NFS or CephFS so the volume can be mounted read-write across nodes and pods.
@@ -100,6 +113,8 @@ On AKS, Azure Disk (`managed-csi`) only supports ReadWriteOnce (RWO). Use **Azur
 1. Enable the [Azure Files CSI driver](https://learn.microsoft.com/en-us/azure/aks/azure-files-csi) if not already installed in your cluster.
 2. Set `core.storage.storageClass` to `azurefile` (or `azurefile-csi` on AKS 1.29+) and `postgresql.persistence.storageClass` to `managed-csi`.
 
+For embedded ClickHouse, also set `clickhouse.persistence.storageClass` to `managed-csi`.
+
 ## Oracle persistent volumes
 
 On OKE with OCI File Storage, see [Setting Up Storage for Kubernetes Clusters](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingpersistentvolumeclaim.htm) to make NFS-backed persistent volumes available, then set `core.storage.storageClass` (and NIM config if needed) accordingly.
@@ -53,6 +53,8 @@ Alternatively, you can configure each microservice to use an external database d
 | 5432/TCP | NeMo Data Store Database  |
 | 5432/TCP | NeMo Evaluator Database |
 | 5432/TCP | NeMo Entity Store Database |
+| 8123/TCP | Embedded ClickHouse HTTP interface used by Intake |
+| 9000/TCP | Embedded ClickHouse native TCP protocol |
 | 9091/TCP | Milvus metrics |
 | 19530/TCP | Milvus API |
 

@@ -31,22 +31,29 @@ secrets will not decrypt with a new key.
 
 ## Intake and ClickHouse
 
-Intake is included in the platform API service group. By default, the chart
-deploys a single-node embedded ClickHouse 26.3 LTS service, and Intake creates
-and migrates its own `intake` database on first use.
+NeMo Platform uses ClickHouse to store Intake data such as spans, traces,
+annotations, and evaluator results.
 
-The embedded ClickHouse is intended for development, evaluation, and
-non-critical single-node installations. It does not provide replication or
-automatic backups. For a production deployment that requires high availability,
-set `clickhouse.enabled` to `false` and provide a separately managed ClickHouse:
+By default, the chart deploys an embedded ClickHouse instance (a single-node
+StatefulSet using the official ClickHouse image) to enable quick installation.
+This should not be used for production use, and an external ClickHouse instance
+is recommended.
 
-First, create the credentials Secret in the Helm release namespace. The Secret
-key must match `externalClickhouse.existingSecretPasswordKey`:
+For production, use a managed service such as
+[ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse
+separately, such as with the official
+[ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview).
+
+To use an external ClickHouse instance, first create the credentials Secret in
+the Helm release namespace. The Secret key must match
+`externalClickhouse.existingSecretPasswordKey`:
 
 ```shell
+NMP_NAMESPACE="your-release-namespace"
+CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password"
 kubectl create secret generic clickhouse-credentials \
-  --namespace <release-namespace> \
-  --from-literal=password='<clickhouse-password>'
+  --namespace "$NMP_NAMESPACE" \
+  --from-file="password=$CLICKHOUSE_PASSWORD_FILE"
 ```
 
 Then configure the external connection:
@@ -65,27 +72,8 @@ externalClickhouse:
   existingSecretPasswordKey: password
 ```
 
-The external user must be allowed to create the configured database, tables,
-materialized views, and indexes because Intake owns its ClickHouse migrations.
-
 ### ClickHouse sizing
 
-Use retained span count as an initial operational threshold, not as a disk-size
-estimate. Span `input`, `output`, and attribute payloads vary substantially.
-Measure `bytes_on_disk` from representative traffic before setting production
-storage:
-
-```sql
-SELECT
-    table,
-    sum(rows) AS physical_rows,
-    formatReadableSize(sum(bytes_on_disk)) AS disk
-FROM system.parts
-WHERE active AND database = 'intake'
-GROUP BY table
-ORDER BY table;
-```
-
 The following are starting points for the current Intake schema and interactive
 query workload:
 
@@ -184,22 +172,22 @@ and
 | api.tolerations | list | `[]` | Tolerations configuration for the API service. |
 | api.topologySpreadConstraints | list | `[]` | Topology spread constraints for the API service pods. See https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ |
 | basePlatformConfig | string | This object has the following default values for the base platform configuration. | Base platform configuration settings |
-| clickhouse | object | This object has the following default values for the embedded ClickHouse configuration. | Embedded ClickHouse configuration for Intake. The embedded deployment is a single-node convenience topology. Use an external, replicated ClickHouse deployment for production environments that require high availability. These values are used only when `clickhouse.enabled` is true. |
+| clickhouse | object | This object has the following default values for the embedded ClickHouse configuration. | Embedded ClickHouse configuration for Intake. These values are used only when `clickhouse.enabled` is true. |
 | clickhouse.affinity | object | `{}` | Affinity for the ClickHouse pod. |
 | clickhouse.annotations | object | `{}` | Annotations to add to the ClickHouse StatefulSet. |
 | clickhouse.auth.database | string | `"intake"` | ClickHouse database used by Intake. |
 | clickhouse.auth.existingSecret | string | `""` | Name of an existing Secret containing the ClickHouse password. If empty, the chart creates one. |
 | clickhouse.auth.existingSecretPasswordKey | string | `"password"` | Key in auth.existingSecret containing the ClickHouse password. |
 | clickhouse.auth.password | string | `nil` | ClickHouse password used when auth.existingSecret is empty. If unset, the chart generates a random password. |
 | clickhouse.auth.username | string | `"nemo"` | ClickHouse username used by Intake. |
-| clickhouse.enabled | bool | `true` | Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`. |
+| clickhouse.enabled | bool | `true` | Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`. It is not recommended to use the embedded ClickHouse for production deployments. It is enabled by default for ease of getting started with the platform. |
 | clickhouse.image.pullPolicy | string | `"IfNotPresent"` | ClickHouse image pull policy. |
 | clickhouse.image.repository | string | `"docker.io/clickhouse/clickhouse-server"` | ClickHouse image repository. Intake is tested against the 26.3 LTS release line. |
 | clickhouse.image.tag | string | `"26.3"` | ClickHouse image tag. |
 | clickhouse.livenessProbe | object | This object has the following default values for the liveness probe configuration. | Liveness probe configuration for the ClickHouse container. |
 | clickhouse.nodeSelector | object | `{}` | Node selector for the ClickHouse pod. |
 | clickhouse.persistence.enabled | bool | `true` | Whether to persist embedded ClickHouse data. |
-| clickhouse.persistence.size | string | `"20Gi"` | PersistentVolumeClaim size. See the Intake and ClickHouse sizing guidance in the chart README. |
+| clickhouse.persistence.size | string | `"20Gi"` | PersistentVolumeClaim size. |
 | clickhouse.persistence.storageClass | string | `""` | Storage class for the ClickHouse PVC. If unset, the cluster default is used. |
 | clickhouse.podAnnotations | object | `{}` | Annotations to add to the ClickHouse pod. |
 | clickhouse.podLabels | object | `{}` | Additional labels to add to the ClickHouse pod. |

@@ -31,22 +31,29 @@ secrets will not decrypt with a new key.
 
 ## Intake and ClickHouse
 
-Intake is included in the platform API service group. By default, the chart
-deploys a single-node embedded ClickHouse 26.3 LTS service, and Intake creates
-and migrates its own `intake` database on first use.
+NeMo Platform uses ClickHouse to store Intake data such as spans, traces,
+annotations, and evaluator results.
 
-The embedded ClickHouse is intended for development, evaluation, and
-non-critical single-node installations. It does not provide replication or
-automatic backups. For a production deployment that requires high availability,
-set `clickhouse.enabled` to `false` and provide a separately managed ClickHouse:
+By default, the chart deploys an embedded ClickHouse instance (a single-node
+StatefulSet using the official ClickHouse image) to enable quick installation.
+This should not be used for production use, and an external ClickHouse instance
+is recommended.
 
-First, create the credentials Secret in the Helm release namespace. The Secret
-key must match `externalClickhouse.existingSecretPasswordKey`:
+For production, use a managed service such as
+[ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse
+separately, such as with the official
+[ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview).
+
+To use an external ClickHouse instance, first create the credentials Secret in
+the Helm release namespace. The Secret key must match
+`externalClickhouse.existingSecretPasswordKey`:
 
 ```shell
+NMP_NAMESPACE="your-release-namespace"
+CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password"
 kubectl create secret generic clickhouse-credentials \
-  --namespace <release-namespace> \
-  --from-literal=password='<clickhouse-password>'
+  --namespace "$NMP_NAMESPACE" \
+  --from-file="password=$CLICKHOUSE_PASSWORD_FILE"
 ```
 
 Then configure the external connection:
@@ -65,27 +72,8 @@ externalClickhouse:
   existingSecretPasswordKey: password
 ```
 
-The external user must be allowed to create the configured database, tables,
-materialized views, and indexes because Intake owns its ClickHouse migrations.
-
 ### ClickHouse sizing
 
-Use retained span count as an initial operational threshold, not as a disk-size
-estimate. Span `input`, `output`, and attribute payloads vary substantially.
-Measure `bytes_on_disk` from representative traffic before setting production
-storage:
-
-```sql
-SELECT
-    table,
-    sum(rows) AS physical_rows,
-    formatReadableSize(sum(bytes_on_disk)) AS disk
-FROM system.parts
-WHERE active AND database = 'intake'
-GROUP BY table
-ORDER BY table;
-```
-
 The following are starting points for the current Intake schema and interactive
 query workload:
 

@@ -243,12 +243,11 @@ externalDatabase:
     key: ""
 
 # -- Embedded ClickHouse configuration for Intake.
-# The embedded deployment is a single-node convenience topology. Use an external,
-# replicated ClickHouse deployment for production environments that require high availability.
 # These values are used only when `clickhouse.enabled` is true.
 # @default -- This object has the following default values for the embedded ClickHouse configuration.
 clickhouse:
   # -- Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`.
+  # It is not recommended to use the embedded ClickHouse for production deployments. It is enabled by default for ease of getting started with the platform.
   enabled: true
   image:
     # -- ClickHouse image repository. Intake is tested against the 26.3 LTS release line.
@@ -278,7 +277,7 @@ clickhouse:
   persistence:
     # -- Whether to persist embedded ClickHouse data.
     enabled: true
-    # -- PersistentVolumeClaim size. See the Intake and ClickHouse sizing guidance in the chart README.
+    # -- PersistentVolumeClaim size.
     size: 20Gi
     # -- Storage class for the ClickHouse PVC. If unset, the cluster default is used.
     storageClass: ""