refactor(phase-9): isolate Ray into services/workers, delete legacy indexer components - #419
Merged
Conversation
Type-safe registry for pluggable components. Each domain (embedder,
reranker, llm, vlm, chunking, parser) will have its own Registry
instance. Implementations register via @registry.register("name")
decorator and are instantiated via registry.create("name", **kwargs).
Includes RegistryError with helpful message listing available
implementations when a lookup fails.
Consolidates all exception classes into core/utils/exceptions.py. Preserves the existing OpenRAGError API (message, code, status_code, to_dict()) and all existing VDB/Embedding subclasses for backward compatibility. Adds new exception categories for the hexagonal architecture: - ConfigError, RegistryError, PipelineError - AuthError, AuthenticationError (401) - ValidationError (422), NotFoundError (404) with domain subtypes - QuotaExceededError (429) - ServiceUnavailableError (503), CircuitBreakerOpenError - InferenceError with LLMParsingError (502), timeout (504), connection (503) - StorageError with MilvusError, PostgresError Status codes preserved from existing codebase for backward compat. Will be moved to api/error_handlers.py mapping in Phase 10.
Updates utils/exceptions/{__init__,base,vectordb,embeddings}.py to
re-export from openrag.core.utils.exceptions. All existing consumer
imports continue to work unchanged.
New code should import from openrag.core.utils.exceptions directly.
These shims will be removed in Phase 12.
CatalogStore ABC will live in core/ports/catalog_store.py (alongside the repository ABCs it composes), not in a separate core/catalog/ folder. Decision logged with Phase 1 entries.
Pure text cleaning functions moved from components/indexer/utils/text_sanitizer.py. No infrastructure imports — only re and unicodedata. Includes sanitize_text(), clean_markdown_table_spacing(), and sanitize_extracted_text().
Pure filename functions extracted from components/indexer/utils/files.py. Only the infrastructure-free parts: sanitize_filename() and make_unique_filename(). The rest (save_file_to_disk, serialize_file) stays in the old location until Phase 5+.
Pure utility for detecting when errors originate from external HTTP resources (VLM image fetches, etc.) rather than internal failures. Moved from utils/external_resource_errors.py.
The unit of indexable/retrievable text. Includes from_langchain() and to_langchain() boundary converters for migration compatibility. Imports are deferred in converter methods so core/ stays pure.
Document is the input to the indexing pipeline. ProcessedDocument is the result after parsing (text blocks + images). Includes DocumentType enum and from_langchain/to_langchain converters.
Domain user model with role enum (viewer/editor/owner), partition memberships carried inline, and OIDC session model for cookie-based auth. Derived from SQLAlchemy models in components/indexer/vectordb/.
Catalog models for tracking document lifecycle (QUEUED -> COMPLETED) and batch indexation jobs. Derived from TaskStateManager + File table.
Query input model, per-chunk scored results (RetrievalResult, ScoredChunk), and end-to-end retrieval output (RetrievalResponse).
Conversation + Message for chat history persistence. ContextualizedQuery for LLM query rewriting (HyDE, multi-query). Prompt + PromptType enum for template management.
Consumers can now import cleanly: from openrag.core.models import Chunk, Document, User, RetrievalQuery
Frozen Pydantic model with dict-like backward compatibility. Existing code using config.section.get(), config.section["key"], dict(config.section), and **config.section keeps working.
endpoints.py — LLMConfig, VLMConfig, EmbedderConfig, SemaphoreConfig, LLMContextConfig (all model endpoint settings in one file). chunking.py — ChunkerConfig (strategy name, chunk size, overlap, etc.).
Discriminated unions for reranker (infinity/openai) and retriever (single/multiQuery/hyde). Plus RAGConfig, MapReduceConfig, and WebSearchConfig (staan provider).
LoaderConfig with all nested configs: FileLoadersConfig, MimetypesConfig, TranscriberConfig, OpenAILoaderConfig, LocalWhisperConfig, plus Marker and Docling settings.
infrastructure.py — VectorDBConfig, RDBConfig, RayConfig (with indexer concurrency groups, serve, semaphore), PathsConfig, ServerConfig, VerboseConfig, PromptsConfig. auth.py — OIDCConfig for Keycloak/external IdP integration.
root.py — Settings class composing all sub-models. loader.py — load_config() with YAML defaults + env var overrides + Pydantic validation. Exact same logic as config/loader.py. The old config/ package will be updated to re-export from here for backward compatibility.
Updates config/__init__.py to import Settings and load_config from openrag.core.config. All existing consumer imports (from config import load_config) continue to work unchanged.
Async-native embedder interface: embed(), embed_single(), dimension property. Registry for pluggable implementations via @embedder_registry.register().
Async reranker interface: rerank(query, documents, top_k) returning (original_index, score) pairs sorted by relevance.
3 tasks
hedhoud
changed the title
The dispatcher is the production replacement for the old indexer_ray_shim — it routes new indexing jobs to IndexerPool via .remote() and bookkeeps task state on the TaskStateManager actor. That makes services/workers/ its natural home; living under services/storage/ violated the "Ray only in services/workers/ + startup" rule from the phase-9 plan. No behavioural change: same class, same factory, only the import path changes. di.container and the dispatcher test suite are updated to the new location.
components/indexer/loaders/audio/openai.py was the last non-worker loader making raw .remote() calls — it grabbed the singleton WhisperActor and invoked detect_language for the language-detection hook the OpenAI audio client uses when use_whisper_lang_detector=true. Move the actor lookup + .remote() call into a new helper ``detect_language_via_actor`` in services/workers/parsers/whisper_workers.py. The loader now imports the helper and wraps it in the existing language_detector callable, so the components/ side is fully Ray-free.
utils/dependencies.py was the last non-worker, non-startup module holding ``import ray`` + module-level ``ray.get_actor`` / .remote() calls. The bootstrap belongs under services/workers/ alongside the worker actors it creates. - New: services/workers/bootstrap.py — same actor-creation helpers and module-level bootstrap calls, no behaviour change. - Deleted: utils/dependencies.py (only consumers were routers/actors.py and di/container.py; both updated to the new path). - main.py now explicitly imports services.workers.bootstrap so the worker actors come up at startup without relying on a transitive import through routers/actors. - Test stubs (test_auth_router, test_utils_partition_access) re-pointed at the new module name; the stubs no longer fake the whole ``services`` package, which had been masking the real services.auth subpackage. - Loader-shim docstrings updated to cite the new path. 9F verification now shows zero Ray imports outside services/workers/, main.py, and the two explicit carve-outs (distributed_semaphore, routers/actors).
The auth_service.py imports ``mask_email`` from ``utils.logger`` (added for PII masking in OIDC logs); the OIDC lifecycle test stubs ``utils.logger`` in ``sys.modules`` but only exposed ``get_logger`` and ``escape_markup``, so collecting the test crashed at import time. Mirror the same helper that the other router-test stubs already define.
hedhoud
added a commit
that referenced
this pull request
May 26, 2026
This was referenced Jun 16, 2026
This was referenced Jul 15, 2026
Closed
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Phase 9 needs the indexing flow split into explicit worker stages before the Ray actor can become a thin runtime wrapper. This PR starts that path while keeping the current production indexing path untouched.
Problem
The worker package had Ray helpers and parser workers, but no concrete stage functions for parse through store. That made the next pipeline-builder step impossible to review independently.
Expected behavior
The stage slice is testable as normal async Python. Each stage mutates a row consistently, marks failures, and removes credentials after use. The obsolete local copilot HTML harness is also removed.
Validation
Summary by CodeRabbit
New Features
Tests