Repository navigation
feat(docker-compose): Add Docker Compose deployment. #427
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Copy this file to `.env` and replace the development credentials before starting Spider. | ||
|
20001020ycx marked this conversation as resolved.
Outdated
|
||
|
|
||
| # Required database credentials. | ||
| SPIDER_DATABASE_ROOT_PASSWORD=spider-root-password | ||
| SPIDER_STORAGE_DB_PASSWORD=spider-password | ||
| SPIDER_STORAGE_DB_USERNAME=spider-user | ||
|
|
||
| # Optional deployment settings and their defaults. | ||
| SPIDER_IMAGE_TAG=main | ||
| SPIDER_PULL_POLICY=always | ||
|
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== candidate files =="
git ls-files | rg '(^|/)(\.env\.example|docker\.yaml|compose\.ya?ml)$' || true
echo
echo "== spider compose env example =="
if [ -f tools/deployment/spider-compose/.env.example ]; then
cat -n tools/deployment/spider-compose/.env.example
fi
echo
echo "== taskfiles/docker.yaml relevant references =="
if [ -f taskfiles/docker.yaml ]; then
rg -n "SPIDER_|spider|compose|pull|build|task" taskfiles/docker.yaml -C 3
fi
echo
echo "== compose files referencing SPIDER_PULL_POLICY or spider pull_policy =="
for f in $(git ls-files | rg '(^|/)(docker|docker-compose|compose)\.ya?ml$' -i); do
echo "-- $f --"
rg -n "SPIDER_PULL_POLICY|pull_policy|services:" "$f" -C 3 || true
doneRepository: y-scope/spider Length of output: 2845 🌐 Web query:
💡 Result: If Docker Compose is failing to use a locally cached image when you expect it to, it is typically due to a mismatch between the expected image tag/digest or the configuration of the pull_policy. The pull_policy attribute controls how Docker Compose handles image retrieval [1][2]. When you believe an image should be found locally but it is not, check the following: 1. Verify Image Existence and Tags: Ensure the image name and tag (or digest) exactly match what is defined in your docker-compose.yml file. You can verify this by running the command docker images [1]. If the tag in your local cache does not match the tag defined in the compose file, Compose will treat it as missing [1][3]. 2. Check the pull_policy Setting: - missing (or if_not_present): This is the default policy if no build section is defined [2]. It uses the local image if it exists; otherwise, it attempts to pull from the registry [1][4]. - never: This tells Compose to use the local image only and never attempt to pull from a registry [3][4]. If the image is not in your local cache, the service will fail to start [5][3]. - build: If a build section is present, Compose may prioritize building the image from source over using an existing local image [2][6]. 3. Interaction with the build Directive: If your service includes a build section, Docker Compose's behavior changes [6]. Even if an image exists locally, Compose might trigger a rebuild or attempt to pull if not configured correctly [6][7]. If you want to force the use of a local image while a build section is present, setting pull_policy: never can prevent accidental pulls, though you may still need to manage the build behavior explicitly [2][7]. Troubleshooting Tips: - Verify with docker images: Confirm that the exact image name and tag reside in the Docker daemon's image store [1][3]. - Overriding policies: You can force a specific behavior during execution by using the --pull flag with the docker compose up command (e.g., --pull always or --pull never) to override the file configuration for a single run [2]. - Clean up: If you suspect issues with local cache integrity, ensure that there are no dangling images or conflicting tags that might be confusing the resolution process [3]. Citations:
🌐 Web query:
💡 Result: In Docker Compose, the pull_policy attribute defines the behavior for fetching container images from a registry [1][2]. The pull_policy: always setting instructs Docker Compose to pull the image from the registry every time the service starts, regardless of whether the image already exists in the local platform cache [1][2]. This is useful for tags that are frequently updated (such as:latest or development branch tags) to ensure you are always using the most recent version available on the registry [2]. The following pull_policy options are available in the Compose Specification [1]: - always: Pulls the image from the registry every time [1][2]. - missing (or if_not_present): The default behavior. Compose pulls the image only if it is not already available in the local platform cache [1][2]. Note that the:latest tag is always checked/pulled even with this policy [1][2]. - never: Does not attempt to pull the image from a registry. It relies entirely on the locally cached image and will report a failure if the image is missing [1][2]. - build: Compose builds the image from source, even if it is already present [1][2]. - daily: Checks the registry if the last pull occurred more than 24 hours ago [1][2]. - weekly: Checks the registry if the last pull occurred more than 7 days ago [1][2]. - every_: Checks for updates if the last pull took place before the specified duration (e.g., every_12h, every_30m) [1][2]. You can also override the configuration file setting using command-line flags. For example, using docker compose up --pull always will force a pull for all services regardless of their individual pull_policy settings [2]. Likewise, the docker compose pull command respects the pull_policy defined in your file unless a different policy is specified via the --policy flag [3][2]. Citations:
🌐 Web query:
💡 Result: In Docker Compose, the pull_policy attribute is used within a service definition to control when and how the service's image is pulled from a registry [1][2]. While command-line flags like --pull (available in commands such as docker compose up and docker compose create) explicitly accept values such as "always", "missing", "never", and sometimes "build" [3][4], the pull_policy attribute within the compose.yaml file itself functions based on the Compose Specification's defined behavior for image resolution [5][6]. When pull_policy is configured in a service definition, it determines the image pulling strategy: - always: Compose always attempts to pull the image from the registry, even if it is already present locally [5]. - missing: Compose pulls the image only if it is not already available in the local image store [1][2]. - never: Compose does not attempt to pull the image from the registry and relies entirely on the local image. If pull_policy is omitted, the default behavior of Compose is to attempt to pull the image if it is missing, and in scenarios where a build section is also defined, Compose manages the precedence of pulling versus building from source according to the Compose Build Specification [5][1][2]. Note that the --pull command-line flag provides additional control during runtime execution, separate from the persistent configuration set in the Compose file [7][3][4]. Citations:
🌐 Web query:
💡 Result: The term SPIDER_WORKER_CONTAINER_IMAGE_REF is not a standard tool or universal environment variable, but rather an application-specific configuration identifier. It is primarily associated with projects using containerized worker architectures, such as those found in custom CI/CD pipelines or distributed systems [1]. In the context of the requested path tools/docker/build.sh, this script is a common pattern used in various open-source projects (such as Apache TVM, Apache Karaf, and Microsoft Playwright) to automate the building and tagging of Docker images [2][3][4]. While build.sh scripts generally perform the same function—compiling source and wrapping it in a container—they are specific to the repository where they reside [2][3][4]. If you are encountering SPIDER_WORKER_CONTAINER_IMAGE_REF in a configuration file or environment, it is likely being used as a reference to define which Docker image tag or registry path the worker service should pull or utilize for a given task [1]. To effectively use a tools/docker/build.sh script, you should: 1. Navigate to the directory containing the script. 2. Check the script's usage by running./tools/docker/build.sh --help (or equivalent), as most of these scripts require arguments like --tag, --platform, or specific environment variables to function correctly [3][4]. 3. Ensure any required dependencies (e.g., node_modules, cargo builds, or specific build-time binaries) are installed or prepared as indicated by the script's internal logic [2][4]. If you are working with a specific project, consult that project's documentation or repository root README for how their build system handles image references [5][6]. Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== files under tools =="
git ls-files tools | sed -n '1,200p'
echo
echo "== tools/docker/build.sh references if present =="
if [ -f tools/docker/build.sh ]; then
wc -l tools/docker/build.sh
rg -n "SPIDER_IMAGE_TAG|SPIDER_WORKER_CONTAINER_IMAGE_REF|IMAGE_TAG|tag|build|DOCKER|docker" tools/docker/build.sh -C 2 || true
fi
echo
echo "== compose image references =="
cat -n tools/deployment/spider-compose/compose.yamlRepository: y-scope/spider Length of output: 7466 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== taskfiles/docker.yaml referenced build images and compose default =="
cat -n taskfiles/docker.yaml
echo
echo "== tools/docker/build.sh =="
cat -n tools/docker/build.sh
echo
echo "== tools/docker/Dockerfile stages =="
sed -n '1,220p' tools/docker/Dockerfile | rg -n "FROM|AS " -C 2 || sed -n '1,220p' tools/docker/DockerfileRepository: y-scope/spider Length of output: 4176 Use a local-image-friendly default pull policy.
If local builds are meant to work with this Compose setup, use an example value matching the local tag or document that local builds need an explicit override. 🤖 Prompt for AI Agents
Contributor
There was a problem hiding this comment. I wonder this could be the workflow of a spider developer who wishes to spin up a docker compose against their local chnage:
CLP does this automatically, particularly the 2nd step, like a single Shall we provide wiring and script for the same user experience. |
||
| SPIDER_RESTART_POLICY=on-failure:3 | ||
| SPIDER_SCHEDULER_LOG_LEVEL=INFO | ||
| SPIDER_STORAGE_HOST_PORT=50051 | ||
| SPIDER_STORAGE_LOG_LEVEL=INFO | ||
| SPIDER_STORAGE_PUBLISH_IP=127.0.0.1 | ||
| SPIDER_WORKER_LOG_LEVEL=INFO | ||
| SPIDER_WORKER_REPLICAS=4 | ||
|
20001020ycx marked this conversation as resolved.
Outdated
|
||
|
|
||
| # A complete worker image reference takes precedence over `SPIDER_IMAGE_TAG` when set. | ||
| # SPIDER_WORKER_CONTAINER_IMAGE_REF=ghcr.io/y-scope/spider/worker:main | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| name: "spider" | ||
|
|
||
| x-service-defaults: &service_defaults | ||
| logging: | ||
| driver: "local" | ||
| pull_policy: "${SPIDER_PULL_POLICY:-always}" | ||
| restart: "${SPIDER_RESTART_POLICY:-on-failure:3}" | ||
| stop_grace_period: "60s" | ||
|
|
||
| x-healthcheck-defaults: &healthcheck_defaults | ||
| interval: "30s" | ||
| retries: 3 | ||
| start_interval: "2s" | ||
| start_period: "60s" | ||
| timeout: "2s" | ||
|
|
||
| services: | ||
| spider-database: | ||
| <<: *service_defaults | ||
| image: "mariadb:10.11.16" | ||
| networks: | ||
| - "spider-internal" | ||
| environment: | ||
| MYSQL_DATABASE: "spider-db" | ||
| MYSQL_PASSWORD: "${SPIDER_STORAGE_DB_PASSWORD:?Please set a value.}" | ||
| MYSQL_ROOT_PASSWORD: "${SPIDER_DATABASE_ROOT_PASSWORD:?Please set a value.}" | ||
| MYSQL_USER: "${SPIDER_STORAGE_DB_USERNAME:?Please set a value.}" | ||
| volumes: | ||
| - type: "volume" | ||
| source: "spider-database-data" | ||
| target: "/var/lib/mysql" | ||
| healthcheck: | ||
| <<: *healthcheck_defaults | ||
| test: [ | ||
| "CMD", | ||
| "mysqladmin", "ping", | ||
| "--silent", | ||
| "-h", "127.0.0.1", | ||
| "-u", "${SPIDER_STORAGE_DB_USERNAME:?Please set a value.}", | ||
| "--password=${SPIDER_STORAGE_DB_PASSWORD:?Please set a value.}", | ||
| ] | ||
|
|
||
| spider-storage: | ||
| <<: *service_defaults | ||
| image: "ghcr.io/y-scope/spider/storage:${SPIDER_IMAGE_TAG:-main}" | ||
| networks: | ||
| - "default" | ||
| - "spider-internal" | ||
| command: [ | ||
| "spider-storage", | ||
| "--config", "/etc/spider/storage.yaml", | ||
| ] | ||
| environment: | ||
| RUST_LOG: "${SPIDER_STORAGE_LOG_LEVEL:-INFO}" | ||
| SPIDER_STORAGE_DB_PASSWORD: "${SPIDER_STORAGE_DB_PASSWORD:?Please set a value.}" | ||
| SPIDER_STORAGE_DB_USERNAME: "${SPIDER_STORAGE_DB_USERNAME:?Please set a value.}" | ||
| ports: | ||
| - host_ip: "${SPIDER_STORAGE_PUBLISH_IP:-127.0.0.1}" | ||
| published: "${SPIDER_STORAGE_HOST_PORT:-50051}" | ||
| target: 50051 | ||
| configs: | ||
| - source: "spider-storage-config" | ||
| target: "/etc/spider/storage.yaml" | ||
| depends_on: | ||
| spider-database: | ||
| condition: "service_healthy" | ||
| restart: true | ||
| healthcheck: | ||
| <<: *healthcheck_defaults | ||
| test: [ | ||
| "CMD", | ||
| "timeout", "2", | ||
| "bash", "-c", "</dev/tcp/127.0.0.1/50051", | ||
| ] | ||
|
|
||
| spider-scheduler: | ||
| <<: *service_defaults | ||
| image: "ghcr.io/y-scope/spider/scheduler:${SPIDER_IMAGE_TAG:-main}" | ||
| networks: | ||
| - "spider-internal" | ||
| command: [ | ||
| "spider-scheduler", | ||
| "--config", "/etc/spider/scheduler.yaml", | ||
| ] | ||
| environment: | ||
| RUST_LOG: "${SPIDER_SCHEDULER_LOG_LEVEL:-INFO}" | ||
| configs: | ||
| - source: "spider-scheduler-config" | ||
| target: "/etc/spider/scheduler.yaml" | ||
| depends_on: | ||
| spider-storage: | ||
| condition: "service_healthy" | ||
| restart: true | ||
| healthcheck: | ||
| <<: *healthcheck_defaults | ||
| test: [ | ||
| "CMD", | ||
| "timeout", "2", | ||
| "bash", "-c", "</dev/tcp/127.0.0.1/50052", | ||
| ] | ||
|
|
||
| spider-worker: | ||
| <<: *service_defaults | ||
| image: >- | ||
| ${SPIDER_WORKER_CONTAINER_IMAGE_REF:-ghcr.io/y-scope/spider/worker:${SPIDER_IMAGE_TAG:-main}} | ||
| networks: | ||
| - "default" | ||
| - "spider-internal" | ||
|
20001020ycx marked this conversation as resolved.
|
||
| command: [ | ||
| "spider-execution-manager", | ||
| "--config", "/etc/spider/worker.yaml", | ||
| ] | ||
| environment: | ||
| RUST_LOG: "${SPIDER_WORKER_LOG_LEVEL:-INFO}" | ||
| configs: | ||
| - source: "spider-worker-config" | ||
| target: "/etc/spider/worker.yaml" | ||
| depends_on: | ||
| spider-scheduler: | ||
| condition: "service_healthy" | ||
| restart: true | ||
| spider-storage: | ||
| condition: "service_healthy" | ||
| restart: true | ||
| deploy: | ||
| replicas: "${SPIDER_WORKER_REPLICAS:-4}" | ||
|
|
||
| configs: | ||
| spider-scheduler-config: | ||
| file: "./config/scheduler.yaml" | ||
| spider-storage-config: | ||
| file: "./config/storage.yaml" | ||
| spider-worker-config: | ||
| file: "./config/worker.yaml" | ||
|
|
||
| networks: | ||
| default: | ||
| spider-internal: | ||
| internal: true | ||
|
|
||
| volumes: | ||
| spider-database-data: | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| connection_pool_size: 4 | ||
| host: "0.0.0.0" | ||
| port: 50052 | ||
| runtime: | ||
| advertised_endpoint: | ||
| host: "spider-scheduler" | ||
| port: 50052 | ||
| scheduler: | ||
| policy: "round_robin" | ||
| config: | ||
| active_job_queue_capacity: 64 | ||
| cleanup_ready_task_capacity: 256 | ||
| commit_ready_task_capacity: 256 | ||
| dispatch_queue_capacity: 64 | ||
| finalizing_job_expiration_timeout_sec: 300 | ||
| ready_task_capacity: 65536 | ||
| storage_poll_timeout_ms: 10 | ||
| tick_interval_ms: 5 | ||
| storage_endpoint: | ||
| host: "spider-storage" | ||
| port: 50051 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| host: "0.0.0.0" | ||
| port: 50051 | ||
| runtime: | ||
| db_config: | ||
| host: "spider-database" | ||
| max_connections: 64 | ||
| name: "spider-db" | ||
| port: 3306 | ||
| ready_queue_config: | ||
| cleanup_capacity: 256 | ||
| commit_capacity: 256 | ||
| task_capacity: 1048576 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| connection_pool_size: 4 | ||
| liveness: | ||
| scheduler_heartbeat_interval_sec: 10 | ||
| storage_heartbeat_interval_sec: 10 | ||
| scheduler: | ||
| host: "spider-scheduler" | ||
| port: 50052 | ||
| scheduler_poll_wait_ms: 1000 | ||
| storage: | ||
| host: "spider-storage" | ||
| port: 50051 | ||
| task_executor: | ||
| bin_path: "/usr/local/bin/spider-task-executor" | ||
| inherited_env: [] | ||
| max_log_line_bytes: 65536 | ||
| package_dir: "/opt/spider/packages" |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: y-scope/spider
Length of output: 299
🏁 Script executed:
Repository: y-scope/spider
Length of output: 29282
🏁 Script executed:
Repository: y-scope/spider
Length of output: 23494
🌐 Web query:
Docker Compose up --wait-timeout documentation💡 Result:
The --wait-timeout option in Docker Compose is used with the --wait flag to specify a maximum duration, in seconds, to wait for the services in a project to reach a running or healthy state [1][2][3]. Key details regarding this option: Purpose: It prevents the docker compose up command from waiting indefinitely when services fail to start or reach a healthy state within a reasonable timeframe [4][5]. Usage: It must be used in conjunction with the --wait flag, which signifies that the command should wait for services to be running or healthy [1][2]. Because the --wait flag implies detached mode, --wait-timeout is typically used when running services in the background [1][3]. Syntax: You can specify the timeout in seconds as follows: docker compose up -d --wait --wait-timeout Note: This option is distinct from the --timeout (or -t) flag, which is used to set the timeout for container shutdown [1][6]. While the --wait-timeout feature was introduced to address issues where processes would hang, users should be aware of ongoing maintenance and potential bugs in specific edge cases, such as complex dependency conditions (e.g., service_completed_successfully) [7][4].
Citations:
updocker/compose#10276Restore the bounded startup timeout.
tcompose:upusesdocker compose up --detach --waitwithout--wait-timeout, so startup waits are no longer bounded by the documented 120-second limit. AddSPIDER_COMPOSE_WAIT_TIMEOUT=120totools/deployment/spider-compose/.env.example, load the variable in the Taskfile, and pass--wait-timeout ${.SPIDER_COMPOSE_WAIT_TIMEOUT:-120}to the compose command.🤖 Prompt for AI Agents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sitaowang1998 sounds reasonable, can you confirm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed offline, add a 5 minutes timeout.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.