diff --git a/docs/src/user-docs/guides-k8s-deployment.md b/docs/src/user-docs/guides-k8s-deployment.md index 29b952db49..6396a31d42 100644 --- a/docs/src/user-docs/guides-k8s-deployment.md +++ b/docs/src/user-docs/guides-k8s-deployment.md @@ -287,6 +287,11 @@ helm template clp . -f custom-values.yaml To use Presto as the query engine, see the [Using Presto][presto-guide] guide for setup instructions, including the Helm values file and installation steps. +### Using Spider as the scheduler + +To use [Spider] as the scheduler for compression and search jobs, see the [Using Spider scheduler][spider-guide] guide +for setup instructions, including the Helm values file and installation steps. + ### Component scheduling You can control where any CLP component is scheduled using standard Kubernetes scheduling primitives @@ -686,6 +691,7 @@ To tear down a `kubeadm` cluster: * [Using object storage][s3-storage]: Configuring S3 storage * [Configuring retention periods][retention-guide]: Setting up data retention policies * [Using Presto][presto-guide]: Distributed SQL queries on compressed logs +* [Using Spider scheduler][spider-guide]: Spider-orchestrated compression [admin-tools]: reference-sbin-scripts/admin-tools.md [aks]: https://azure.microsoft.com/en-us/products/kubernetes-service @@ -711,5 +717,7 @@ To tear down a `kubeadm` cluster: [s3-storage]: guides-using-object-storage/index [SeaweedFS]: https://github.com/seaweedfs/seaweedfs [seaweedfs-setup]: guides-docker-compose-deployment.md#setting-up-seaweedfs +[Spider]: https://github.com/y-scope/spider +[spider-guide]: guides-using-spider-scheduler.md [webui-clp-json]: quick-start/clp-json.md#searching-from-the-ui [webui-clp-text]: quick-start/clp-text.md#searching-from-the-ui diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md new file mode 100644 index 0000000000..cdfcc6408c --- /dev/null +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -0,0 +1,217 @@ +# Using Spider scheduler + +[Spider][spider] is a distributed task scheduling and execution framework. It schedules tasks across +configured workers and manages their execution lifecycle. + +CLP uses Spider as its next-generation framework for orchestrating compression and search jobs, +providing the following benefits: + +* **Improved fault tolerance** +* **Minimized scheduling overhead** +* **Maximized utilization of compute resources** + +:::{note} +In the current release, CLP uses Spider only for compression jobs coordinated by +`compression-coordinator`. Support for orchestrating search jobs with Spider is planned for a future +release. + +Spider is planned to become the default job execution framework for CLP, eventually replacing the +existing Celery-based job orchestration components. +::: + +## Deployment + +When Spider is enabled, compatible CLP coordinators are automatically deployed and submit supported +jobs to Spider for scheduling and execution. Currently, Spider and its related CLP components can be +enabled only in [Kubernetes deployments][k8s-deployment] using the CLP Helm chart. + +:::{note} +Spider is not supported in Docker Compose deployments in the current release. Support for Docker +Compose is planned for a future release. +::: + +### Kubernetes (Helm) + +When deploying CLP on Kubernetes using Helm, enable Spider by setting `spider.enabled` to `true`. +The CLP Helm chart will then automatically deploy the following components: + +* **Spider subchart**: Deploys all the Spider services required to schedule and execute jobs. +* **compression-coordinator**: Coordinates CLP compression jobs and submits them to Spider for + execution. + + :::{note} + In the current release, `compression-coordinator` supports only compression jobs that use certain + CLP package configurations. See "Using compression-coordinator" for the current limitations. + ::: + +#### Set up + +1. Create a values file to enable Spider and its related CLP components. + + ```{code-block} yaml + :caption: spider-values.yaml + + spider: + enabled: true + + # Optional Spider configuration passed directly to the Spider subchart. + # The values below are the defaults and can be adjusted as needed. + spiderConfig: + storage: + log_level: "INFO" + port: 50051 + scheduler: + log_level: "INFO" + worker: + replicas: 4 + + clpConfig: + # Optional `compression-coordinator` configuration. + # The values below are the defaults and can be adjusted as needed. + compression_coordinator: + logging_level: "INFO" + # How often the coordinator checks for newly created compression jobs. + job_polling_interval_millisecs: 100 + # Maximum number of times a failed compression task can be retried. + compression_task_max_retry: 1 + # Maximum number of times a failed commit task can be retried. + commit_task_max_retry: 1 + # Maximum number of compression jobs that can be submitted to Spider concurrently. + max_concurrent_jobs: 1000 + + logs_input: + # In the current release, compression jobs coordinated by `compression-coordinator` require + # input logs to be stored in S3. + type: "s3" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + + archive_output: + storage: + # In the current release, `compression-coordinator` supports only S3 archive output. + type: "s3" + s3_config: + region_code: "us-east-2" + bucket: "" + key_prefix: "archives/" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + ``` + +2. Install (or upgrade) the Helm chart with the Spider values: + + ```bash + helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml + ``` + +3. Verify that `compression-coordinator` and Spider pods are running: + + ```bash + kubectl get pods | grep -E "spider|compression-coordinator" + ``` + + The pods may take a few minutes to become ready. During startup, some Spider pods may restart + while the Spider database is being initialized. A small number of restarts during this period is + expected. + + The output should eventually include entries similar to the following: + + ```text + NAME READY STATUS RESTARTS AGE + clp-compression-coordinator-... 1/1 Running 0 5m + clp-spider-database-0 1/1 Running 0 5m + clp-spider-scheduler-... 1/1 Running 2 5m + clp-spider-storage-... 1/1 Running 2 5m + clp-spider-worker-... 1/1 Running 2 5m + ``` + +## Configure Spider + +Spider can be configured to meet the resource, storage, and authentication requirements of your +workload. This section describes commonly adjusted settings for Spider in a CLP deployment, followed +by options for more advanced performance tuning. + +Configure these settings in the CLP Helm values file. + +### Common settings + +The following settings are commonly adjusted when deploying Spider with CLP. + +:::{confval} spider.enabled +:type: `bool` +:default: `false` + +Enables Spider and deploys its related CLP components. +::: + +:::{confval} spider.spiderConfig.worker.replicas +:type: `int` +:default: 4 + +Sets the number of Spider worker pods to deploy. Increase this value to allow more tasks to run +concurrently. +::: + +### Advanced tuning + +The following settings provide additional control over Spider's runtime behavior and performance. +Adjust them as needed to suit your workload and deployment environment. + +:::{confval} spider.spiderConfig.scheduler.config.active_job_queue_size +:type: `int` +:default: 8 + +Sets the maximum number of jobs that can actively make progress in Spider at the same time. Jobs +beyond this limit remain queued until an active slot becomes available. Currently, queued jobs are +admitted in first-in, first-out (FIFO) order. + +Active jobs share the available Spider workers in a round-robin fashion. As a result, this setting +controls the trade-off between job-level concurrency and the amount of compute capacity available +to each active job. + +In the current release, Spider executes only compression jobs, so this setting effectively limits +the number of compression jobs that can make progress concurrently. + +Consider increasing this value when you want more jobs to make progress concurrently and sufficient +worker capacity is available. Decrease it when you prefer to concentrate compute resources on fewer +jobs at a time. +::: + +:::{confval} spider.spiderConfig.scheduler.config.dispatch_queue_capacity +:type: `int` +:default: 16 + +Sets the capacity of the scheduler's task dispatch queue. This queue buffers tasks that are ready to +be dispatched to Spider workers. + +As a general guideline, configure this value to approximately two to four times the number of +Spider workers. If you increase the number of workers, consider increasing this value as well so +that the scheduler can keep enough tasks available for dispatch and avoid underutilizing workers. +::: + +:::{confval} spider.spiderConfig.scheduler.config.ready_task_capacity +:type: `int` +:default: 1048576 + +Sets the maximum number of ready tasks that the Spider scheduler can buffer for scheduling. + +A larger capacity allows the scheduler to consider more ready tasks at once, which can improve +fairness across active jobs and better accommodate workloads with high job submission rates. +However, buffering more tasks also increases the scheduler's memory usage. + +Consider: + +* Increasing this value for workloads that produce large numbers of ready tasks or have high job + submission rates. +* Decreasing this value if scheduler memory usage is a concern and the workload does not require a + large ready-task buffer. +::: + +[k8s-deployment]: guides-k8s-deployment.md +[spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index f10656eafe..aeace35656 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -118,6 +118,24 @@ Use Presto for distributed SQL queries on compressed logs. --- +## Scheduling & performance + +Configure how CLP schedules jobs and uses compute resources. + +::::{grid} 1 1 2 2 +:gutter: 2 + +:::{grid-item-card} +:link: guides-using-spider-scheduler +Using Spider scheduler +^^^ +Use Spider to schedule and execute supported CLP jobs across distributed workers. +::: + +:::: + +--- + ## Core Try CLP's core binaries directly, without deploying the full CLP package. @@ -259,6 +277,13 @@ guides-mcp-server/index guides-using-presto ::: +:::{toctree} +:hidden: +:caption: Scheduling & performance + +guides-using-spider-scheduler +::: + :::{toctree} :hidden: :caption: Core