Add fast integration tests workflow for CI

[paultranvan]

Summary

• Add new integration_tests.yaml GitHub Actions workflow that starts the full OpenRAG stack and verifies /health_check responds
• Target runtime: under 5 minutes (vs 45-75 min for smoke_test)

Key optimizations

• Uses official pre-built vLLM CPU image from AWS ECR (v0.9.2)
• Pre-pulls all Docker images in parallel
• Aggressive health check timeouts (5s interval vs 30s)
• Caches HuggingFace models
• Only tests health check endpoint (no document indexing)

Files added

• .github/workflows/integration_tests.yaml - Main workflow
• .github/workflows/integration_tests/.env - Minimal config for health check

Test plan

• Verify workflow runs successfully on push to dev/main
• Verify workflow completes in under 5 minutes
• Verify /health_check endpoint responds correctly

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Added an automated integration test pipeline (GitHub Actions) that boots a full-stack test environment, performs health checks, validates service readiness, captures failure diagnostics, and cleans up resources.
  • Added environment configuration for the integration tests to control service endpoints, health-check behavior, and resource limits for reliable, repeatable runs.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

Adds a GitHub Actions "Integration Tests" workflow and accompanying environment file to spin up a Docker Compose full-stack test environment (OpenRAG, vLLM, Milvus, etcd, MinIO, PostgreSQL), run health checks, validate the RAG API endpoint, gather diagnostics on failure, and clean up resources.

Changes

Cohort / File(s)

Summary

Integration test CI & config .github/workflows/integration_tests.yaml, .github/workflows/integration_tests/.env

New GitHub Actions workflow and env file: triggers on push/PR, logs into GHCR, pre-pulls images, generates a docker-compose.override for OpenRAG/vLLM/Milvus/etcd/MinIO/Postgres with aggressive health checks, caches HF models, boots stack under CPU profile, polls service readiness, verifies /health (expects "RAG API is up"), emits failure logs/status dumps, and always performs cleanup. Env file includes local endpoints, model placeholders, resource/parallelism limits, and disables contextual retrieval/reranker for tests.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant GH Actions as "GitHub Actions"
  participant GHCR as "GHCR (registry)"
  participant Runner as "Self-hosted / GitHub Runner"
  participant Docker as "Docker Engine"
  participant Compose as "docker-compose (integration override)"
  participant Services as "OpenRAG / vLLM / Milvus / etcd / MinIO / Postgres"

  Note over GH Actions,Runner: Workflow start (push/PR/workflow_dispatch)
  GH Actions->>GHCR: Authenticate (ghcr login)
  GH Actions->>Runner: dispatch job
  Runner->>GHCR: pre-pull images (parallel)
  Runner->>Docker: create docker-compose.integration.yaml
  Runner->>Docker: docker compose up --profile cpu -d
  Docker->>Compose: start defined services
  Compose->>Services: bring containers up with health checks
  loop Poll until healthy / timeout
    Runner->>Services: GET /health
    Services-->>Runner: health response ("RAG API is up") or error
  end
  alt healthy
    Runner->>GH Actions: success
  else failure
    Runner->>Docker: docker compose ps, logs, inspect -> collect diagnostics
    Runner->>Docker: docker compose down -v
    Runner->>GH Actions: fail with diagnostics
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 Hops across CI fields I go,

Containers hum and services glow,
Health checks sing, the compose stack springs,
Logs gathered neat as victory rings,
A rabbit cheers — the tests can flow!

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a GitHub Actions workflow for fast integration tests in CI.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

.github/workflows/integration_tests.yaml (1)

9-13: Consider filtering PR branches to avoid redundant runs.

The pull_request: trigger without a branches: filter will run on PRs targeting any branch. If you only want integration tests for PRs targeting main or dev, add a filter:

🔎 Suggested change

 on:
   push:
     branches: [main, dev]
-  pull_request:
+  pull_request:
+    branches: [main, dev]
   workflow_dispatch:

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 397bb7b and c42bbdc.

📒 Files selected for processing (2)

• .github/workflows/integration_tests.yaml
• .github/workflows/integration_tests/.env

🧰 Additional context used

🪛 dotenv-linter (4.0.0)

.github/workflows/integration_tests/.env

[warning] 4-4: [UnorderedKey] The API_KEY key should go before the BASE_URL key

(UnorderedKey)

---

[warning] 9-9: [UnorderedKey] The VLM_API_KEY key should go before the VLM_BASE_URL key

(UnorderedKey)

---

[warning] 14-14: [UnorderedKey] The EMBEDDER_BASE_URL key should go before the EMBEDDER_MODEL_NAME key

(UnorderedKey)

---

[warning] 28-28: [UnorderedKey] The RAY_MAX_TASKS_PER_WORKER key should go before the RAY_POOL_SIZE key

(UnorderedKey)

---

[warning] 33-33: [LowercaseKey] The RAY_task_retry_delay_ms key should be in uppercase

(LowercaseKey)

---

[warning] 34-34: [UnorderedKey] The RAY_ENABLE_UV_RUN_RUNTIME_ENV key should go before the RAY_task_retry_delay_ms key

(UnorderedKey)

---

[warning] 35-35: [LowercaseKey] The RAY_memory_monitor_refresh_ms key should be in uppercase

(LowercaseKey)

---

[warning] 35-35: [UnorderedKey] The RAY_memory_monitor_refresh_ms key should go before the RAY_task_retry_delay_ms key

(UnorderedKey)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: index-backup-restore

🔇 Additional comments (6)

.github/workflows/integration_tests.yaml (5)

45-56: Good optimization with parallel image pulls.

Backgrounding the pulls and using wait is an effective way to parallelize and reduce startup time.

---

89-94: Health check configuration is well-tuned for CI.

The aggressive polling (5s interval, 30s start_period, 30 retries) balances fast feedback with reasonable tolerance for slow container startup.

---

195-208: Health check polling logic looks solid.

The 180s timeout with status updates provides good observability while waiting. The curl -sf flags correctly handle silent failures.

---

211-220: Health check verification is straightforward and effective.

The substring match and informative failure message are appropriate for this validation.

---

241-245: Cleanup step is correctly configured.

Using if: always() ensures resources are cleaned up regardless of test outcome, and || true prevents cleanup failures from masking the actual test result.

.github/workflows/integration_tests/.env (1)

1-35: Environment file is appropriate for integration testing.

The configuration is well-organized with clear comments explaining each section. Placeholder API keys (sk-) are acceptable since this is only used for health check verification and actual LLM/VLM endpoints are not exercised.

The Ray environment variables (RAY_task_retry_delay_ms, RAY_memory_monitor_refresh_ms) use the correct case-sensitive naming convention expected by Ray and are properly formatted.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Self-copy command is a no-op.

Line 185 copies .hydra_config/* into .hydra_config/, which copies a directory's contents into itself—this does nothing. If the intent was to copy from a source location (e.g., a template directory), the source path needs correction. Otherwise, remove this line.

🔎 Suggested fix (if the line is unnecessary)

       - name: Create required directories
         run: |
           mkdir -p model_weights data db logs .hydra_config
-          cp -r .hydra_config/* .hydra_config/ 2>/dev/null || true

🤖 Prompt for AI Agents

.github/workflows/integration_tests.yaml around lines 182 to 185: the cp command
copies `.hydra_config/*` into the same `.hydra_config/` directory (a no-op);
either remove that cp line entirely, or change the source to the actual
template/source directory you intended to copy from (e.g., replace
`.hydra_config/*` with the correct path like `config_templates/.hydra_config/*`
or similar), and keep the `2>/dev/null || true` behavior if you want to ignore
missing sources.

[Ahmath-Gadji]

No longer relevant #194