diff --git a/docs/requirements.mdx b/docs/requirements.mdx index b1e0b30d456..8c33fcc2a28 100644 --- a/docs/requirements.mdx +++ b/docs/requirements.mdx @@ -30,6 +30,7 @@ clusters, use the NeMo Platform Helm chart on Kubernetes. | Kubernetes | minikube, kind, EKS, AKS, GKE, OKE, OpenShift, or on-prem Kubernetes | Use a cluster with storage, networking, and image-pull access configured before installing the chart. | | Helm | Helm 3 | Required to install and upgrade the NeMo Platform chart. | | Storage | ReadWriteMany-capable persistent storage | Required for shared files and job storage in multi-pod deployments. | +| ClickHouse for Intake | External production service, or the chart's embedded development/testing instance | Embedded ClickHouse creates a separate RWO PVC. Use a separately managed ClickHouse service for production. See [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake). | | Image access | NGC credentials and Kubernetes image pull secrets | Required to pull NeMo Platform container images. | ## GPU Requirements diff --git a/docs/set-up/helm/backup-and-restore.mdx b/docs/set-up/helm/backup-and-restore.mdx index 84960042927..742b102a8f7 100644 --- a/docs/set-up/helm/backup-and-restore.mdx +++ b/docs/set-up/helm/backup-and-restore.mdx @@ -2,11 +2,12 @@ title: "Backup and Restore" description: "" --- -NeMo Platform use several storage systems that require backup consideration: +NeMo Platform uses several storage systems that require backup consideration: 1. **PostgreSQL database**: stores entity metadata and relationships -2. **Object storage**: stores model files, datasets, and other artifacts -3. **Persistent Volume Claims (PVC)**: stores Kubernetes persistent data +2. **ClickHouse database**: stores Intake spans, traces, annotations, and evaluator results +3. **Object storage**: stores model files, datasets, and other artifacts +4. **Persistent Volume Claims (PVC)**: stores Kubernetes persistent data ## PostgreSQL Backup @@ -15,6 +16,13 @@ For PostgreSQL backup and restore procedures, refer to the official PostgreSQL d - [PostgreSQL Backup and Restore](https://www.postgresql.org/docs/current/backup.html) - [Continuous Archiving and Point-in-Time Recovery (PITR)](https://www.postgresql.org/docs/current/continuous-archiving.html) +## ClickHouse Backup + +For ClickHouse backup and restore procedures, refer to the official ClickHouse documentation: + +- [Backup and restore in ClickHouse](https://clickhouse.com/docs/concepts/features/backup-restore/overview) +- [Review and restore ClickHouse Cloud backups](https://clickhouse.com/docs/products/cloud/guides/backups/review-and-restore-backups) + ## Object Storage Backup Backup procedures for object storage depend on your chosen storage solution: diff --git a/docs/set-up/helm/database-setup.mdx b/docs/set-up/helm/database-setup.mdx index c28de40e7b5..67efbd0f935 100644 --- a/docs/set-up/helm/database-setup.mdx +++ b/docs/set-up/helm/database-setup.mdx @@ -80,3 +80,44 @@ Use individual `host`, `port`, `user`, and `database` values and a secret that c existingSecret: my-db-password-secret existingSecretPasswordKey: "password" # key in the secret that holds the password ``` + +## ClickHouse for Intake + +The NeMo Platform uses ClickHouse to store Intake data such as spans, traces, annotations, and evaluator results. + +By default, the NeMo Platform Helm chart deploys an embedded ClickHouse instance (a single-node StatefulSet using the official ClickHouse image) to enable quick installation. This should not be used for production use, and an external ClickHouse instance is recommended. + +For production, use a managed service such as [ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse separately, such as with the official [ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview). + +### External ClickHouse Database + +To use an external ClickHouse database in the Helm chart: + +1. Set `clickhouse.enabled: false` and configure `externalClickhouse`. +2. Create a Secret containing the ClickHouse password in the same namespace as the NeMo Platform release: + + ```sh + NMP_NAMESPACE="your-release-namespace" + CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password" + kubectl create secret generic clickhouse-credentials \ + --namespace "$NMP_NAMESPACE" \ + --from-file="password=$CLICKHOUSE_PASSWORD_FILE" + ``` + +3. Configure the external HTTP endpoint in `values.yaml`: + + ```yaml + clickhouse: + enabled: false + + externalClickhouse: + host: clickhouse.example.internal + port: 8443 + secure: true + user: nemo + database: intake + existingSecret: clickhouse-credentials + existingSecretPasswordKey: password + ``` + + Set `host` to a hostname only, without a URL scheme or path. Set `secure: true` for HTTPS; the common HTTPS port is `8443`. diff --git a/docs/set-up/helm/index.mdx b/docs/set-up/helm/index.mdx index 94ed5bcc791..e3efbbd6973 100644 --- a/docs/set-up/helm/index.mdx +++ b/docs/set-up/helm/index.mdx @@ -26,7 +26,7 @@ Install the NeMo Platform using the Helm chart. -Set up an external database for the NeMo Platform. +Configure external PostgreSQL and ClickHouse databases. cluster-admin on-prem cloud diff --git a/docs/set-up/helm/install.mdx b/docs/set-up/helm/install.mdx index 80562eb0746..bdb72bc7ead 100644 --- a/docs/set-up/helm/install.mdx +++ b/docs/set-up/helm/install.mdx @@ -172,6 +172,11 @@ postgresql: persistence: storageClass: gp3 size: 20Gi + +clickhouse: + persistence: + storageClass: gp3 + size: 20Gi ``` If you store Files service data in S3, configure the file storage backend and make AWS credentials available through IRSA or another boto3-supported credential source: @@ -188,6 +193,8 @@ platformConfig: For multi-node GPU jobs on EKS, also configure EFA networking as described in [Multinode Networking](/documentation/self-managed-deployment/setup/helm/multinode-networking). +For production Intake deployments, use a separately managed ClickHouse service. See [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake). + ## Install Volcano Install the Volcano scheduler before installing the chart. This is required for customization jobs that leverage multiple nodes. diff --git a/docs/set-up/helm/persistent-volumes.mdx b/docs/set-up/helm/persistent-volumes.mdx index 068b7a14e71..1ca1bc45636 100644 --- a/docs/set-up/helm/persistent-volumes.mdx +++ b/docs/set-up/helm/persistent-volumes.mdx @@ -69,6 +69,19 @@ Replace `"nfs"` with your StorageClass name (e.g. `oci-nfs`, `gp3`). For NIM sca Refer to the [platform configuration documentation](/documentation/self-managed-deployment/config-reference) for the full config reference. +## Embedded ClickHouse Storage + +The embedded ClickHouse StatefulSet creates a separate ReadWriteOnce volume. Configure it with a low-latency block StorageClass: + +```yaml +clickhouse: + persistence: + storageClass: fast-rwo + size: 20Gi +``` + +The embedded deployment is not recommended for production. For production, use an external ClickHouse instance as described in [Database Setup](/documentation/self-managed-deployment/setup/helm/database-setup#clickhouse-for-intake). + ## Persistent volume options Use a ReadWriteMany-capable filesystem such as NFS or CephFS so the volume can be mounted read-write across nodes and pods. @@ -100,6 +113,8 @@ On AKS, Azure Disk (`managed-csi`) only supports ReadWriteOnce (RWO). Use **Azur 1. Enable the [Azure Files CSI driver](https://learn.microsoft.com/en-us/azure/aks/azure-files-csi) if not already installed in your cluster. 2. Set `core.storage.storageClass` to `azurefile` (or `azurefile-csi` on AKS 1.29+) and `postgresql.persistence.storageClass` to `managed-csi`. +For embedded ClickHouse, also set `clickhouse.persistence.storageClass` to `managed-csi`. + ## Oracle persistent volumes On OKE with OCI File Storage, see [Setting Up Storage for Kubernetes Clusters](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingpersistentvolumeclaim.htm) to make NFS-backed persistent volumes available, then set `core.storage.storageClass` (and NIM config if needed) accordingly. diff --git a/docs/set-up/security.mdx b/docs/set-up/security.mdx index 9d9d554ca48..16d5c3c023a 100644 --- a/docs/set-up/security.mdx +++ b/docs/set-up/security.mdx @@ -53,6 +53,8 @@ Alternatively, you can configure each microservice to use an external database d | 5432/TCP | NeMo Data Store Database | | 5432/TCP | NeMo Evaluator Database | | 5432/TCP | NeMo Entity Store Database | +| 8123/TCP | Embedded ClickHouse HTTP interface used by Intake | +| 9000/TCP | Embedded ClickHouse native TCP protocol | | 9091/TCP | Milvus metrics | | 19530/TCP | Milvus API | diff --git a/k8s/helm/README.md b/k8s/helm/README.md index 7f1ba1caa25..153cdd158b9 100644 --- a/k8s/helm/README.md +++ b/k8s/helm/README.md @@ -31,22 +31,29 @@ secrets will not decrypt with a new key. ## Intake and ClickHouse -Intake is included in the platform API service group. By default, the chart -deploys a single-node embedded ClickHouse 26.3 LTS service, and Intake creates -and migrates its own `intake` database on first use. +NeMo Platform uses ClickHouse to store Intake data such as spans, traces, +annotations, and evaluator results. -The embedded ClickHouse is intended for development, evaluation, and -non-critical single-node installations. It does not provide replication or -automatic backups. For a production deployment that requires high availability, -set `clickhouse.enabled` to `false` and provide a separately managed ClickHouse: +By default, the chart deploys an embedded ClickHouse instance (a single-node +StatefulSet using the official ClickHouse image) to enable quick installation. +This should not be used for production use, and an external ClickHouse instance +is recommended. -First, create the credentials Secret in the Helm release namespace. The Secret -key must match `externalClickhouse.existingSecretPasswordKey`: +For production, use a managed service such as +[ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse +separately, such as with the official +[ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview). + +To use an external ClickHouse instance, first create the credentials Secret in +the Helm release namespace. The Secret key must match +`externalClickhouse.existingSecretPasswordKey`: ```shell +NMP_NAMESPACE="your-release-namespace" +CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password" kubectl create secret generic clickhouse-credentials \ - --namespace \ - --from-literal=password='' + --namespace "$NMP_NAMESPACE" \ + --from-file="password=$CLICKHOUSE_PASSWORD_FILE" ``` Then configure the external connection: @@ -65,27 +72,8 @@ externalClickhouse: existingSecretPasswordKey: password ``` -The external user must be allowed to create the configured database, tables, -materialized views, and indexes because Intake owns its ClickHouse migrations. - ### ClickHouse sizing -Use retained span count as an initial operational threshold, not as a disk-size -estimate. Span `input`, `output`, and attribute payloads vary substantially. -Measure `bytes_on_disk` from representative traffic before setting production -storage: - -```sql -SELECT - table, - sum(rows) AS physical_rows, - formatReadableSize(sum(bytes_on_disk)) AS disk -FROM system.parts -WHERE active AND database = 'intake' -GROUP BY table -ORDER BY table; -``` - The following are starting points for the current Intake schema and interactive query workload: @@ -184,7 +172,7 @@ and | api.tolerations | list | `[]` | Tolerations configuration for the API service. | | api.topologySpreadConstraints | list | `[]` | Topology spread constraints for the API service pods. See https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ | | basePlatformConfig | string | This object has the following default values for the base platform configuration. | Base platform configuration settings | -| clickhouse | object | This object has the following default values for the embedded ClickHouse configuration. | Embedded ClickHouse configuration for Intake. The embedded deployment is a single-node convenience topology. Use an external, replicated ClickHouse deployment for production environments that require high availability. These values are used only when `clickhouse.enabled` is true. | +| clickhouse | object | This object has the following default values for the embedded ClickHouse configuration. | Embedded ClickHouse configuration for Intake. These values are used only when `clickhouse.enabled` is true. | | clickhouse.affinity | object | `{}` | Affinity for the ClickHouse pod. | | clickhouse.annotations | object | `{}` | Annotations to add to the ClickHouse StatefulSet. | | clickhouse.auth.database | string | `"intake"` | ClickHouse database used by Intake. | @@ -192,14 +180,14 @@ and | clickhouse.auth.existingSecretPasswordKey | string | `"password"` | Key in auth.existingSecret containing the ClickHouse password. | | clickhouse.auth.password | string | `nil` | ClickHouse password used when auth.existingSecret is empty. If unset, the chart generates a random password. | | clickhouse.auth.username | string | `"nemo"` | ClickHouse username used by Intake. | -| clickhouse.enabled | bool | `true` | Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`. | +| clickhouse.enabled | bool | `true` | Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`. It is not recommended to use the embedded ClickHouse for production deployments. It is enabled by default for ease of getting started with the platform. | | clickhouse.image.pullPolicy | string | `"IfNotPresent"` | ClickHouse image pull policy. | | clickhouse.image.repository | string | `"docker.io/clickhouse/clickhouse-server"` | ClickHouse image repository. Intake is tested against the 26.3 LTS release line. | | clickhouse.image.tag | string | `"26.3"` | ClickHouse image tag. | | clickhouse.livenessProbe | object | This object has the following default values for the liveness probe configuration. | Liveness probe configuration for the ClickHouse container. | | clickhouse.nodeSelector | object | `{}` | Node selector for the ClickHouse pod. | | clickhouse.persistence.enabled | bool | `true` | Whether to persist embedded ClickHouse data. | -| clickhouse.persistence.size | string | `"20Gi"` | PersistentVolumeClaim size. See the Intake and ClickHouse sizing guidance in the chart README. | +| clickhouse.persistence.size | string | `"20Gi"` | PersistentVolumeClaim size. | | clickhouse.persistence.storageClass | string | `""` | Storage class for the ClickHouse PVC. If unset, the cluster default is used. | | clickhouse.podAnnotations | object | `{}` | Annotations to add to the ClickHouse pod. | | clickhouse.podLabels | object | `{}` | Additional labels to add to the ClickHouse pod. | diff --git a/k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl b/k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl index 7b04f6ee977..c3a22e02c96 100644 --- a/k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl +++ b/k8s/helm/helm-docs-template/nemo-helm-readme.md.gotmpl @@ -31,22 +31,29 @@ secrets will not decrypt with a new key. ## Intake and ClickHouse -Intake is included in the platform API service group. By default, the chart -deploys a single-node embedded ClickHouse 26.3 LTS service, and Intake creates -and migrates its own `intake` database on first use. +NeMo Platform uses ClickHouse to store Intake data such as spans, traces, +annotations, and evaluator results. -The embedded ClickHouse is intended for development, evaluation, and -non-critical single-node installations. It does not provide replication or -automatic backups. For a production deployment that requires high availability, -set `clickhouse.enabled` to `false` and provide a separately managed ClickHouse: +By default, the chart deploys an embedded ClickHouse instance (a single-node +StatefulSet using the official ClickHouse image) to enable quick installation. +This should not be used for production use, and an external ClickHouse instance +is recommended. -First, create the credentials Secret in the Helm release namespace. The Secret -key must match `externalClickhouse.existingSecretPasswordKey`: +For production, use a managed service such as +[ClickHouse Cloud](https://clickhouse.com/cloud) or operate ClickHouse +separately, such as with the official +[ClickHouse Kubernetes Operator](https://clickhouse.com/docs/products/kubernetes-operator/overview). + +To use an external ClickHouse instance, first create the credentials Secret in +the Helm release namespace. The Secret key must match +`externalClickhouse.existingSecretPasswordKey`: ```shell +NMP_NAMESPACE="your-release-namespace" +CLICKHOUSE_PASSWORD_FILE="/path/to/clickhouse-password" kubectl create secret generic clickhouse-credentials \ - --namespace \ - --from-literal=password='' + --namespace "$NMP_NAMESPACE" \ + --from-file="password=$CLICKHOUSE_PASSWORD_FILE" ``` Then configure the external connection: @@ -65,27 +72,8 @@ externalClickhouse: existingSecretPasswordKey: password ``` -The external user must be allowed to create the configured database, tables, -materialized views, and indexes because Intake owns its ClickHouse migrations. - ### ClickHouse sizing -Use retained span count as an initial operational threshold, not as a disk-size -estimate. Span `input`, `output`, and attribute payloads vary substantially. -Measure `bytes_on_disk` from representative traffic before setting production -storage: - -```sql -SELECT - table, - sum(rows) AS physical_rows, - formatReadableSize(sum(bytes_on_disk)) AS disk -FROM system.parts -WHERE active AND database = 'intake' -GROUP BY table -ORDER BY table; -``` - The following are starting points for the current Intake schema and interactive query workload: diff --git a/k8s/helm/values.yaml b/k8s/helm/values.yaml index 3f3a6cd495d..cd3e4d3b15f 100644 --- a/k8s/helm/values.yaml +++ b/k8s/helm/values.yaml @@ -243,12 +243,11 @@ externalDatabase: key: "" # -- Embedded ClickHouse configuration for Intake. -# The embedded deployment is a single-node convenience topology. Use an external, -# replicated ClickHouse deployment for production environments that require high availability. # These values are used only when `clickhouse.enabled` is true. # @default -- This object has the following default values for the embedded ClickHouse configuration. clickhouse: # -- Whether to deploy the embedded ClickHouse. Set to false to use `externalClickhouse`. + # It is not recommended to use the embedded ClickHouse for production deployments. It is enabled by default for ease of getting started with the platform. enabled: true image: # -- ClickHouse image repository. Intake is tested against the 26.3 LTS release line. @@ -278,7 +277,7 @@ clickhouse: persistence: # -- Whether to persist embedded ClickHouse data. enabled: true - # -- PersistentVolumeClaim size. See the Intake and ClickHouse sizing guidance in the chart README. + # -- PersistentVolumeClaim size. size: 20Gi # -- Storage class for the ClickHouse PVC. If unset, the cluster default is used. storageClass: ""