Coderabbit labels

[paultranvan]

Summary by CodeRabbit

• New Features

  • Added version endpoint to API.
  • Added text sanitization and markdown table spacing cleanup.
  • Enhanced chunk processing with configurable timeouts and concurrency limits.
  • Support for custom URL schemes in API responses.

• Bug Fixes

  • Improved error handling for external resource failures.
  • Enhanced HTTP error reporting in LLM operations.
  • Fixed exception handling in extract operations.

• Documentation

  • Added environment variable configuration documentation.
  • Added development guide for the project.

• Tests

  • Added comprehensive API integration test suite.
  • Added unit tests for text processing and error detection.

• Chores

  • Version bumped to 1.1.6.
  • Updated configuration and Docker setup.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Caution

Review failed

Failed to post review comments

📝 Walkthrough

Walkthrough

This PR introduces comprehensive API testing infrastructure, text sanitization utilities, enhanced error handling for external resource failures, and refinements to document chunking with contextualization. It includes a complete CI/CD testing workflow with Docker Compose setup, mock VLLM server, and extensive integration tests alongside core feature enhancements and configuration updates.

Changes

Cohort / File(s)	Summary
Configuration & Version Management .coderabbit.yaml, .env.example, pyproject.toml, pytest.ini, .hydra_config/chunker/base.yaml, .hydra_config/config.yaml	Configuration additions for auto-review labels, preferred URL scheme, contextualization timeouts/concurrency limits, pytest markers; version bump to 1.1.6 and build-system configuration.
API Testing Infrastructure .github/workflows/api_tests.yml, .github/workflows/api_tests/Dockerfile.mock-vllm, .github/workflows/api_tests/docker-compose.yaml, .github/workflows/api_tests/mock_vllm.py, .github/workflows/api_tests/fixtures/sample.txt	New GitHub Actions workflow for API integration tests with health checks, Docker Compose setup orchestrating mock-vllm, PostgreSQL, etcd, MinIO, and Milvus services, and a lightweight mock VLLM server implementing embeddings and chat completion endpoints.
API Tests Suite tests/api_tests/__init__.py, tests/api_tests/conftest.py, tests/api_tests/test_*.py (health, indexer, search, partition, queue, tools, extract, actors, users, openai_compat)	Comprehensive integration test fixtures and test cases covering health checks, file indexing, document search, partition CRUD, queue management, tool execution, and OpenAI-compatible API endpoints.
Text Sanitization & Utilities openrag/components/indexer/utils/text_sanitizer.py, openrag/components/indexer/utils/test_text_sanitizer.py, openrag/components/indexer/utils/__init__.py	New text sanitization module with Unicode normalization, control character removal, whitespace normalization, and Markdown table spacing utilities; comprehensive test coverage.
Error Handling for External Resources openrag/utils/external_resource_errors.py, openrag/utils/test_external_resource_errors.py	New utilities to classify external resource errors (HTTP status codes, network/SSL/connection errors) with URL extraction; applied to image description fetching in loaders.
Ray Actor Timeout Management openrag/components/ray_utils.py	New helper function call_ray_actor_with_timeout to wrap Ray futures with configurable timeouts, handling cancellation and error propagation.
Chunker Enhancements openrag/components/indexer/chunker/chunker.py, openrag/components/indexer/chunker/utils.py, openrag/components/indexer/chunker/test_chunking.py	Adds contextualization timeout and concurrency limits, batches contextualization calls, handles empty chunks gracefully, injects filename context, skips image placeholders via IMAGE_PLACEHOLDER constant, sanitizes text before chunking.
PDF & Media Loaders openrag/components/indexer/loaders/pdf_loaders/marker.py, openrag/components/indexer/loaders/pdf_loaders/openai.py, openrag/components/indexer/loaders/base.py, openrag/components/indexer/loaders/test_media_loader.py	Marker worker cleanup via destructor, timeout-aware PDF processing; removes Ray-based PDF-to-images worker in favor of async function; external resource error detection for image description fetching; comprehensive media loader tests.
API & Router Enhancements openrag/api.py, openrag/routers/indexer.py, openrag/routers/openai.py, openrag/routers/tools.py, openrag/routers/extract.py, openrag/routers/utils.py	Dynamic version retrieval and /version endpoint; URL scheme customization for generated links; improved error handling in streaming responses; text sanitization in tool execution; HTTPException re-raising; enhanced logging with error details.
Core Utilities & LLM openrag/components/utils.py, openrag/components/llm.py, openrag/components/indexer/utils/files.py, openrag/components/indexer/vectordb/vectordb.py	Reduced CPU spin via small sleeps in semaphore cleanup; Ray restart policy adjustment (max_restarts 5); JSON/HTTP error handling in LLM completions; timeout-aware serialization via new helper; chunk ID validation in vector DB.
Documentation & Metadata CLAUDE.md, docs/content/docs/documentation/API.mdx, docs/content/docs/documentation/env_vars.md, openrag/consts.py, openrag/test_version.py, openrag/scripts/filter-logs.py	New Claude development guide, API version endpoint documentation, environment variable documentation, IMAGE_PLACEHOLDER constant, version tests, and log filtering script for timestamp-based filtering.
Infrastructure & Configuration .gitignore, docker-compose.yaml, entrypoint.sh, openrag/utils/logger.py	Minor .gitignore adjustments, Docker Compose network naming and idle sleep for VLLM, entrypoint.sh syntax fix, file logging permission error handling.
File Organization openrag/components/indexer/indexer.py, openrag/components/indexer/utils/test_files.py	Import path consolidation: serialize_file and related utilities moved under indexer/utils/ with updated references throughout.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• fix: Improve error logging for external resource access failures #186 — Both add external resource error handling (openrag/utils/external_resource_errors.py) and apply it in openrag/components/indexer/loaders/base.py.
• correct IndexError caused by empty list of chunks when performing chu… #202 — Both modify chunker contextualization logic, add IMAGE_PLACEHOLDER handling, and enhance filename plumbing in openrag/components/indexer/chunker/chunker.py.
• Add lightweight API test workflow with mock VLLM #194 — Both add the same API testing infrastructure (.github/workflows/api_tests.yml, mock_vllm.py, docker-compose setup, and comprehensive test suite under tests/api_tests/).

Suggested labels

chore, tests, feature, documentation

Suggested reviewers

• Ahmath-Gadji

Poem

🐰 A rabbit's whimsical ode to thorough testing

With mocks and fixtures, tests align,
Mock VLLM servers help us shine!
Timeouts caught and errors sorted,
Clean text sanitized, nicely sorted—
From chunking flows to API tests so fine,
Infrastructure blooms—the future's mine! 🥕

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 77.51% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The PR title 'Coderabbit labels' is vague and does not clearly describe the main changes. While the PR does add configuration for suggested_labels and auto_apply_labels in .coderabbit.yaml, the title fails to convey what these changes accomplish or their significance.	Consider a more descriptive title such as 'Enable auto-labeling in Coderabbit configuration' or 'Add Coderabbit suggested and auto-apply labels configuration' to better communicate the primary change.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}