Skip to content
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/src/user-docs/guides-k8s-deployment.md
Comment thread
20001020ycx marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,11 @@ helm template clp . -f custom-values.yaml
To use Presto as the query engine, see the [Using Presto][presto-guide] guide for setup
instructions, including the Helm values file and installation steps.

### Using Spider as the scheduler

To use [Spider] as the scheduler for compression and search jobs, see the [Using Spider scheduler][spider-guide] guide
for setup instructions, including the Helm values file and installation steps.

### Component scheduling

You can control where any CLP component is scheduled using standard Kubernetes scheduling primitives
Expand Down Expand Up @@ -686,6 +691,7 @@ To tear down a `kubeadm` cluster:
* [Using object storage][s3-storage]: Configuring S3 storage
* [Configuring retention periods][retention-guide]: Setting up data retention policies
* [Using Presto][presto-guide]: Distributed SQL queries on compressed logs
* [Using Spider scheduler][spider-guide]: Spider-orchestrated compression

[admin-tools]: reference-sbin-scripts/admin-tools.md
[aks]: https://azure.microsoft.com/en-us/products/kubernetes-service
Expand All @@ -711,5 +717,7 @@ To tear down a `kubeadm` cluster:
[s3-storage]: guides-using-object-storage/index
[SeaweedFS]: https://github.com/seaweedfs/seaweedfs
[seaweedfs-setup]: guides-docker-compose-deployment.md#setting-up-seaweedfs
[Spider]: https://github.com/y-scope/spider
[spider-guide]: guides-using-spider-scheduler.md
[webui-clp-json]: quick-start/clp-json.md#searching-from-the-ui
[webui-clp-text]: quick-start/clp-text.md#searching-from-the-ui
215 changes: 215 additions & 0 deletions docs/src/user-docs/guides-using-spider-scheduler.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# Using Spider scheduler

[Spider][spider] is a distributed task scheduling and execution framework. It schedules tasks across
configured workers and manages their execution lifecycle.

CLP uses Spider as its next-generation framework for orchestrating compression and search jobs,
providing the following benefits:

* **Improved fault tolerance**
* **Minimized scheduling overhead**
* **Maximized utilization of compute resources**

:::{note}
In the current release, CLP uses Spider only for compression jobs coordinated by
`compression-coordinator`. Support for orchestrating search jobs with Spider is under development
and will be introduced in a future release.

Spider is planned to become the default job execution framework for CLP, eventually replacing the
existing Celery-based job orchestration components.
:::

## Deployment

When Spider is enabled, compatible CLP coordinators are automatically deployed and submit supported
jobs to Spider for scheduling and execution. Currently, Spider and its related CLP components can be
enabled only in [Kubernetes deployments][k8s-deployment] using the CLP Helm chart.

:::{note}
Spider is not supported in Docker Compose deployments in the current release. Support for Docker
Compose is planned for a future release.
:::

### Kubernetes (Helm)

When deploying CLP on Kubernetes using Helm, enable Spider by setting `spider.enabled` to `true`.
The CLP Helm chart will then automatically deploy the following components:

* **Spider subchart**: Deploys all the Spider services required to schedule and execute jobs.
* **compression-coordinator**: Coordinates supported CLP compression jobs and submits them to
Spider for execution.

:::{note}
In the current release, `compression-coordinator` supports only compression jobs that use certain
CLP package configurations. See "Using compression-coordinator" for the current limitations.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #2456 is merged we should refer to the coordinator's guide.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to myself, do a pass after all deps PRs are merged

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we should probably merge this PR first since it's self-contained. #2456 will refer to content in this PR as well but it needs to refer to specific sections.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but be sure to add linking in the content we wrote in this PR back in #2456, for example, "Using compression-coordinator" in this sentence right here worth a link

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will add it after this PR is merged.

:::

#### Set up

1. Create a values file to enable Spider and its related CLP components.

```{code-block} yaml
:caption: spider-values.yaml

spider:
enabled: true

# Optional Spider configuration passed directly to the Spider subchart.
# The values below are the defaults and can be adjusted as needed.
spiderConfig:
storage:
log_level: "INFO"
port: 50051
scheduler:
log_level: "INFO"
worker:
replicas: 4

clpConfig:
# Optional `compression-coordinator` configuration.
# The values below are the defaults and can be adjusted as needed.
compression_coordinator:
logging_level: "INFO"
# How often the coordinator checks for newly created compression jobs.
job_polling_interval_millisecs: 100
# Maximum number of times a failed compression task can be retried.
compression_task_max_retry: 1
# Maximum number of times a failed commit task can be retried.
commit_task_max_retry: 1
Comment thread
20001020ycx marked this conversation as resolved.
Resolved

logs_input:
# In the current release, compression jobs coordinated by `compression-coordinator` require
# input logs to be stored in S3.
type: "s3"
aws_authentication:
type: "credentials"
credentials:
access_key_id: "<access-key-id>"
secret_access_key: "<secret-access-key>"

archive_output:
storage:
# In the current release, `compression-coordinator` supports only S3 archive output.
type: "s3"
s3_config:
region_code: "us-east-2"
bucket: "<archives-bucket>"
key_prefix: "archives/"
aws_authentication:
type: "credentials"
credentials:
access_key_id: "<access-key-id>"
secret_access_key: "<secret-access-key>"
Comment thread
20001020ycx marked this conversation as resolved.
Resolved
```

2. Install (or upgrade) the Helm chart with the Spider values:

```bash
helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

3. Verify that `compression-coordinator` and Spider pods are running:

```bash
kubectl get pods | grep -E "spider|compression-coordinator"
```

The pods may take a few minutes to become ready. During startup, some Spider pods may restart
while the Spider database is being initialized. A small number of restarts during this period is
expected.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

The output should eventually include entries similar to the following:

```text
NAME READY STATUS RESTARTS AGE
clp-compression-coordinator-... 1/1 Running 0 5m
clp-spider-database-0 1/1 Running 0 5m
clp-spider-scheduler-... 1/1 Running 2 5m
clp-spider-storage-... 1/1 Running 2 5m
clp-spider-worker-... 1/1 Running 2 5m
```

## Configure Spider

Spider can be configured to meet the resource, storage, and authentication requirements of your
workload. This section describes commonly adjusted settings for Spider in a CLP deployment, followed
by options for more advanced performance tuning.

Configure these settings in the CLP Helm values file.

### Common settings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note for myself, add external database set up config in #2493


The following settings are commonly adjusted when deploying Spider with CLP.

:::{confval} spider.enabled
:type: `bool`
:default: `false`

Enables Spider and deploys its related CLP components.
:::

:::{confval} spider.spiderConfig.worker.replicas
:type: `int`
:default: 4

Sets the number of Spider worker pods to deploy. Increase this value to allow more tasks to run
concurrently.
:::

### Advanced tuning

The following settings provide additional control over Spider's runtime behavior and performance.
Adjust them as needed to suit your workload and deployment environment.

:::{confval} spider.spiderConfig.scheduler.config.active_job_queue_size
:type: `int`
:default: 8

Sets the maximum number of jobs that can actively make progress in Spider at the same time. Jobs
beyond this limit remain queued until an active slot becomes available. Currently, queued jobs are
admitted in first-in, first-out (FIFO) order.

Active jobs share the available Spider workers in a round-robin fashion. As a result, this setting
controls the trade-off between job-level concurrency and the amount of compute capacity available
to each active job.

In the current release, Spider executes only compression jobs, so this setting effectively limits
the number of compression jobs that can make progress concurrently.

Consider increasing this value when you want more jobs to make progress concurrently and sufficient
worker capacity is available. Decrease it when you prefer to concentrate compute resources on fewer
jobs at a time.
:::

:::{confval} spider.spiderConfig.scheduler.config.dispatch_queue_capacity
:type: `int`
:default: 16

Sets the capacity of the scheduler's task dispatch queue. This queue buffers tasks that are ready to
be dispatched to Spider workers.

As a general guideline, configure this value to approximately two to four times the number of
Spider workers. If you increase the number of workers, consider increasing this value as well so
that the scheduler can keep enough tasks available for dispatch and avoid underutilizing workers.
:::

:::{confval} spider.spiderConfig.scheduler.config.ready_task_capacity
:type: `int`
:default: 1048576

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe be explicit about 1048576 (1M)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, what do you mean?


Sets the maximum number of ready tasks that the Spider scheduler can buffer for scheduling.

A larger capacity allows the scheduler to consider more ready tasks at once, which can improve
fairness across active jobs and better accommodate workloads with high task submission rates.
However, buffering more tasks also increases the scheduler's memory usage.

Consider:

* Increasing this value for workloads that produce large numbers of ready tasks or have high task
submission rates.
* Decreasing this value if scheduler memory usage is a concern and the workload does not require a
large ready-task buffer.
:::

[k8s-deployment]: guides-k8s-deployment.md
[spider]: https://github.com/y-scope/spider
25 changes: 25 additions & 0 deletions docs/src/user-docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,24 @@ Use Presto for distributed SQL queries on compressed logs.

---

## Scheduling & performance

Configure how CLP schedules jobs and uses compute resources.

::::{grid} 1 1 2 2
:gutter: 2

:::{grid-item-card}
:link: guides-using-spider-scheduler
Using Spider scheduler
^^^
Use Spider to schedule and execute supported CLP jobs across distributed workers.
:::

::::

---

## Core

Try CLP's core binaries directly, without deploying the full CLP package.
Expand Down Expand Up @@ -259,6 +277,13 @@ guides-mcp-server/index
guides-using-presto
:::

:::{toctree}
:hidden:
:caption: Scheduling & performance

guides-using-spider-scheduler
:::

:::{toctree}
:hidden:
:caption: Core
Expand Down
Loading