docs(spider): Remove legacy Spider integration doc. #2436
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,9 +45,7 @@ graph LR | |
| results_cache["results-cache (MongoDB)"] | ||
| compression_scheduler["compression-scheduler"] | ||
| query_scheduler["query-scheduler"] | ||
| compression_worker["compression-worker"] | ||
| query_worker["query-worker"] | ||
| reducer["reducer"] | ||
| api_server["api-server"] | ||
|
|
@@ -77,25 +75,23 @@ graph LR | |
| query_scheduler -->|healthy| reducer | ||
| linkStyle 6 stroke:#800080 | ||
|
|
||
| %% Link 7-13: Database initialization job --> Services | ||
| db_table_creator -->|completed_successfully| api_server | ||
| db_table_creator -->|completed_successfully| compression_scheduler | ||
| db_table_creator -->|completed_successfully| garbage_collector | ||
| db_table_creator -->|completed_successfully| log_ingestor | ||
| db_table_creator -->|completed_successfully| mcp_server | ||
| db_table_creator -->|completed_successfully| query_scheduler | ||
| db_table_creator -->|completed_successfully| webui | ||
| linkStyle 7,8,9,10,11,12,13 stroke:#0000ff | ||
|
|
||
| %% Link 14-18: Results cache initialization job --> Services | ||
| results_cache_indices_creator -->|completed_successfully| api_server | ||
| results_cache_indices_creator -->|completed_successfully| garbage_collector | ||
| results_cache_indices_creator -->|completed_successfully| mcp_server | ||
| results_cache_indices_creator -->|completed_successfully| reducer | ||
| results_cache_indices_creator -->|completed_successfully| webui | ||
| linkStyle 14,15,16,17,18 stroke:#008000 | ||
|
|
||
| subgraph Databases | ||
| database | ||
|
|
@@ -114,12 +110,10 @@ graph LR | |
| subgraph Schedulers | ||
| compression_scheduler | ||
| query_scheduler | ||
| end | ||
|
|
||
| subgraph Workers | ||
| compression_worker | ||
| query_worker | ||
| reducer | ||
| end | ||
|
|
@@ -137,8 +131,6 @@ graph LR | |
|
|
||
| %% Subgraph styles | ||
| style celery_dependencies fill:#ffffe0 | ||
|
|
||
| +++ | ||
| **Figure 1**: Orchestration architecture of the services in the CLP package. | ||
|
|
@@ -157,10 +149,8 @@ graph LR | |
| | redis | Task result storage for workers | | ||
| | compression_scheduler | Scheduler for compression jobs | | ||
| | query_scheduler | Scheduler for search/aggregation jobs | | ||
| | results_cache | Storage for the workers to return search results to the UI | | ||
| | compression_worker | Worker processes for compression jobs using Celery | | ||
| | query_worker | Worker processes for search/aggregation jobs using Celery | | ||
| | reducer | Reducers for performing the final stages of aggregation jobs | | ||
| | api_server | API server for submitting queries | | ||
|
|
@@ -251,23 +241,14 @@ Services require persistent storage for logs, data, archives, and streams. | |
|
|
||
| ### Deployment types | ||
|
Member
There was a problem hiding this comment. Changes in this section may require a sanity check from @junhaoliao.
Member
There was a problem hiding this comment. |
||
|
|
||
| CLP supports two deployment configurations, depending on the query engine: | ||
|
|
||
| * **Full**: Uses Celery for query orchestration. | ||
| * **Base**: Excludes the Celery-based query orchestration in Full, to support the | ||
| [Presto integration][presto-integration]. | ||
|
LinZhihao-723 marked this conversation as resolved.
|
||
|
|
||
| Docker Compose uses `deploy.replicas` with environment variables (e.g., `CLP_MCP_SERVER_ENABLED`) | ||
| to toggle optional services. Helm uses conditional templating to include or exclude resources. | ||
|
Comment on lines
242
to
+251
Member
There was a problem hiding this comment. Since there's only one type of compression scheduler now, shall we remove the "Compression Scheduler" column?
Member
There was a problem hiding this comment.
i agree. it might be easier to read today by turning the table into a simple list. e.g., CLP supports two deployment configurations, depending on the query engine:
|
||
|
|
||
| ## Troubleshooting | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do have a docker-compose file for the legacy spider setup. We don't need to remove it in this PR, but we should probably make an issue to keep track of it for removal.