diff --git a/docs/src/dev-docs/design-deployment-orchestration.md b/docs/src/dev-docs/design-deployment-orchestration.md index 0046df73e4..86b1e64ed1 100755 --- a/docs/src/dev-docs/design-deployment-orchestration.md +++ b/docs/src/dev-docs/design-deployment-orchestration.md @@ -45,9 +45,7 @@ graph LR results_cache["results-cache (MongoDB)"] compression_scheduler["compression-scheduler"] query_scheduler["query-scheduler"] - spider_scheduler["spider-scheduler"] compression_worker["compression-worker"] - spider_compression_worker["spider-compression-worker"] query_worker["query-worker"] reducer["reducer"] api_server["api-server"] @@ -77,25 +75,23 @@ graph LR query_scheduler -->|healthy| reducer linkStyle 6 stroke:#800080 - %% Link 7-15: Database initialization job --> Services + %% Link 7-13: Database initialization job --> Services db_table_creator -->|completed_successfully| api_server db_table_creator -->|completed_successfully| compression_scheduler db_table_creator -->|completed_successfully| garbage_collector db_table_creator -->|completed_successfully| log_ingestor db_table_creator -->|completed_successfully| mcp_server db_table_creator -->|completed_successfully| query_scheduler - db_table_creator -->|completed_successfully| spider_compression_worker - db_table_creator -->|completed_successfully| spider_scheduler db_table_creator -->|completed_successfully| webui - linkStyle 7,8,9,10,11,12,13,14,15 stroke:#0000ff + linkStyle 7,8,9,10,11,12,13 stroke:#0000ff - %% Link 16-20: Results cache initialization job --> Services + %% Link 14-18: Results cache initialization job --> Services results_cache_indices_creator -->|completed_successfully| api_server results_cache_indices_creator -->|completed_successfully| garbage_collector results_cache_indices_creator -->|completed_successfully| mcp_server results_cache_indices_creator -->|completed_successfully| reducer results_cache_indices_creator -->|completed_successfully| webui - linkStyle 16,17,18,19,20 stroke:#008000 + linkStyle 14,15,16,17,18 stroke:#008000 subgraph Databases database @@ -114,12 +110,10 @@ graph LR subgraph Schedulers compression_scheduler query_scheduler - spider_scheduler end subgraph Workers compression_worker - spider_compression_worker query_worker reducer end @@ -137,8 +131,6 @@ graph LR %% Subgraph styles style celery_dependencies fill:#ffffe0 - style spider_compression_worker fill:#008080 - style spider_scheduler fill:#008080 +++ **Figure 1**: Orchestration architecture of the services in the CLP package. @@ -157,10 +149,8 @@ graph LR | redis | Task result storage for workers | | compression_scheduler | Scheduler for compression jobs | | query_scheduler | Scheduler for search/aggregation jobs | -| spider_scheduler | Scheduler for Spider distributed task execution framework | | results_cache | Storage for the workers to return search results to the UI | | compression_worker | Worker processes for compression jobs using Celery | -| spider_compression_worker | Worker processes for compression jobs using Spider | | query_worker | Worker processes for search/aggregation jobs using Celery | | reducer | Reducers for performing the final stages of aggregation jobs | | api_server | API server for submitting queries | @@ -251,23 +241,14 @@ Services require persistent storage for logs, data, archives, and streams. ### Deployment types -CLP supports multiple deployment configurations based on the compression scheduler and query engine. +CLP supports two deployment configurations, depending on the query engine: -| Deployment Type | Compression Scheduler | Query Engine | -|-----------------|-----------------------|------------------------------| -| Base | Celery | [Presto][presto-integration] | -| Full | Celery | Native | -| Spider Base | Spider | [Presto][presto-integration] | -| Spider Full | Spider | Native | +* **Full**: Uses Celery for query orchestration. +* **Base**: Excludes the Celery-based query orchestration in Full, to support the + [Presto integration][presto-integration]. -:::{note} -Spider support is not yet available for Helm. -::: - -Docker Compose selects the appropriate compose file (e.g., `docker-compose.yaml` for Full, -`docker-compose-spider.yaml` for Spider Full) and uses `deploy.replicas` with environment -variables (e.g., `CLP_MCP_SERVER_ENABLED`) to toggle optional services. Helm uses conditional -templating to include/exclude resources. +Docker Compose uses `deploy.replicas` with environment variables (e.g., `CLP_MCP_SERVER_ENABLED`) +to toggle optional services. Helm uses conditional templating to include or exclude resources. ## Troubleshooting diff --git a/docs/src/user-docs/guides-docker-compose-deployment.md b/docs/src/user-docs/guides-docker-compose-deployment.md index b6958d4db3..0526934bb0 100755 --- a/docs/src/user-docs/guides-docker-compose-deployment.md +++ b/docs/src/user-docs/guides-docker-compose-deployment.md @@ -199,12 +199,6 @@ docker compose \ up compression-scheduler \ --no-deps --wait -# Start Spider scheduler (optional, only if using Spider) -docker compose \ - --project-name "clp-package-$(cat var/log/instance-id)" \ - up spider-scheduler \ - --no-deps --wait - # Start query scheduler docker compose \ --project-name "clp-package-$(cat var/log/instance-id)" \ @@ -245,17 +239,11 @@ docker compose \ # Worker services (can be started on multiple hosts) ################################################################################ -# Start compression worker (if using Celery) +# Start compression worker docker compose \ --project-name "clp-package-$(cat var/log/instance-id)" \ up compression-worker \ --no-deps --wait - -# Start Spider compression worker (optional, only if using Spider) -docker compose \ - --project-name "clp-package-$(cat var/log/instance-id)" \ - up spider-compression-worker \ - --no-deps --wait # Start query worker docker compose \ diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md deleted file mode 100644 index 94f67699ea..0000000000 --- a/docs/src/user-docs/guides-using-spider.md +++ /dev/null @@ -1,66 +0,0 @@ -# Using Spider with CLP - -[Spider] is a fast and scalable distributed task execution engine that can be used to run tasks. -This guide describes how to set up and use Spider with CLP. - -:::{note} -Spider is under active development, and its integration with CLP may change in the future. -Right now, Spider only supports executing CLP compression tasks. Support for search tasks will be added -later. -::: - -## Requirements - -* [CLP][clp-releases] v0.8.0 or higher -* [Docker] v28 or higher -* [Docker Compose][docker-compose] v2.20.2 or higher - -## Set up - -To use Spider for CLP compression tasks, you need to [set up CLP](#setting-up-clp-with-spider) with -Spider in configuration. - -### Setting up CLP with Spider - -1. Follow the [quick-start](quick-start/index.md) guide to download and extract the CLP package, - but don't start the package just yet. -2. Before starting the package, update the package's config file (`etc/clp-config.yaml`) as follows: - - * Change the `compression_scheduler.type` field to `"spider"`. - - ```yaml - compression_scheduler: - type: "spider" - ``` - - * (Optional) Override `database.names.spider` to avoid name conflicts when using self-provisioned database instances. - - ```yaml - database: - names: - spider: "spider-db" - ``` - - * (Optional) Override the `spider_scheduler` default config to change listening host or avoid port conflicts. - - ```yaml - spider_scheduler: - host: "localhost" - port: 6000 - ``` - -3. (Optional) If you do not intend to use generated credentials, set your own Spider credentials in - `etc/credentials.yaml` before starting the package. - - ```yaml - database: - spider_username: "spider-user" - spider_password: "spider-pass" - ``` - -4. Continue following the [quick-start](./quick-start/index.md#using-clp) guide to start CLP. - -[clp-releases]: https://github.com/y-scope/clp/releases -[docker-compose]: https://docs.docker.com/compose/install/ -[Docker]: https://docs.docker.com/engine/install/ -[Spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index 8621c65398..f10656eafe 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -114,12 +114,6 @@ Using Presto Use Presto for distributed SQL queries on compressed logs. ::: -:::{grid-item-card} -:link: guides-using-spider -Using Spider -^^^ -Use Spider for compression and query job task distribution. -::: :::: --- @@ -263,7 +257,6 @@ guides-retention guides-using-the-api-server guides-mcp-server/index guides-using-presto -guides-using-spider ::: :::{toctree}