refactor(parsers): remove legacy loaders and doc-serializer shims

README.md

@@ -148,7 +148,7 @@ All supported file format parsers are pre-configured. For PDF processing, **[Mar
 <details>
 <summary>For more PDF options</summary>
 
-For CPU-only deployments or lightweight testing scenarios, you can consider switching to **`PyMuPDF4LLMLoader`** or **`PyMuPDFLoader`**. To change the loader, set the **`PDFLoader`** variable like this `PDFLoader=PyMuPDF4LLMLoader`.
+For CPU-only deployments or lightweight testing scenarios, you can consider switching to **`PyMuPDFLoader`**. To change the loader, set the **`PDFLoader`** variable like this `PDFLoader=PyMuPDFLoader`.
 
 > ⚠️ **Important**: These alternative loaders have limitations - they cannot process non-searchable (image-based) PDFs and do not extract or handle embedded images.
 </details>

docs/content/docs/documentation/env_vars.md

@@ -22,10 +22,10 @@ Openrag loads all files into a pivot markdown file format before proceeding to c
 | `IMAGE_CAPTIONING_URL` | `bool` | `true` | If `true`, HTTP/HTTPS image URLs in markdown files are fetched and described by the VLM. |
 | `SAVE_MARKDOWN` | `bool` | `false` | If `true`, the pivot-format markdown produced during parsing is saved. Useful for debugging and verifying the correctness of the generated markdown. |
 |`SAVE_UPLOADED_FILES`|`bool`|`false`| When `true`, uploaded files are stored on disk. You must enable this option if you want Chainlit to show sources while chatting.|
-| `PDFLoader` | `str` | `MarkerLoader` | Specifies the PDF parsing engine to use. Available options: `PyMuPDFLoader`, `PyMuPDF4LLMLoader`, `MarkerLoader` and `DotsOCRLoader`.|
+| `PDFLoader` | `str` | `MarkerLoader` | Specifies the PDF parsing engine to use. Available options: `PyMuPDFLoader`, `MarkerLoader` and `DotsOCRLoader`.|
 
 :::caution
-`PyMuPDFLoader` and `PyMuPDF4LLMLoader` are lightweight pdf loaders that cannot process non-searchable (image-based) PDFs and do not extract or handle embedded images.
+`PyMuPDFLoader` is a lightweight pdf loader that cannot process non-searchable (image-based) PDFs and does not extract or handle embedded images.
 :::
 
 #### PDF Loader

docs/content/docs/getting_started/quickstart.mdx

@@ -11,7 +11,7 @@ OpenRAG is an open source Retrieval Augmented Generation (RAG) solution. This gu
 ### Prerequisites
 - [Docker](https://www.docker.com/get-started) and **Docker Compose**
 - Your hardware should meet these specifications:
-  - **CPU deployment**: Minimum **13 GiB** RAM for light PDF parsers (**`PyMuPDF4LLMLoader`, `PyMuPDFLoader`**), or **23 GiB** RAM for heavier parsers like **`MarkerLoader`** (refer to [this section](/openrag/getting_started/quickstart/#3-file-parser-configuration) for details)
+  - **CPU deployment**: Minimum **13 GiB** RAM for light PDF parsers (**`PyMuPDFLoader`**), or **23 GiB** RAM for heavier parsers like **`MarkerLoader`** (refer to [this section](/openrag/getting_started/quickstart/#3-file-parser-configuration) for details)
   - **GPU deployment**: **16 GB** GPU memory recommended (for systems with separate CPU and GPU memory)
 
 ### Installation and Configuration
@@ -37,7 +37,7 @@ Here is a brief overview of key environment variables to configure:
 All supported file format parsers are pre-configured. For PDF processing, **[MarkerLoader](https://github.com/datalab-to/marker)** serves as the default parser, offering comprehensive support for OCR-scanned documents, complex layouts, tables, and embedded images. MarkerLoader operates efficiently on both GPU and CPU environments.
 
 :::note 
-For **`CPU-only deployments`** or lightweight testing scenarios, you can consider switching to **`PyMuPDF4LLMLoader`** or **`PyMuPDFLoader`**. To change the loader, set the **`PDFLoader`** variable like this `PDFLoader=PyMuPDF4LLMLoader`.
+For **`CPU-only deployments`** or lightweight testing scenarios, you can consider switching to **`PyMuPDFLoader`**. To change the loader, set the **`PDFLoader`** variable like this `PDFLoader=PyMuPDFLoader`.
 :::caution[Important]
 These alternative loaders have limitations - they cannot process non-searchable (image-based) PDFs and do not extract or handle embedded images.
 :::

openrag/api/main.py

@@ -155,7 +155,7 @@ async def lifespan(app: FastAPI):
 
     # ``ensure_worker_bootstrap`` imports ``services.workers.bootstrap``
     # for its side effect: creating the long-lived detached worker
-    # actors (TaskStateManager, DocSerializer, MarkerPool, semaphores).
+    # actors (TaskStateManager, MarkerPool, semaphores).
     # The indirection through :mod:`di.workers` keeps API code free of
     # direct ``services.workers`` imports.
     logger.info("Startup: initializing worker bootstrap")

openrag/api/routers/admin/tools.py

@@ -1,8 +1,8 @@
 """Tools routes — thin HTTP layer over :class:`ConversionService`.
 
-Phase 8E: the ``extractText`` serialization moved to
-``services.orchestrators.conversion_service.ConversionService`` (the Ray
-``DocSerializer`` actor now sits behind the ``FileSerializer`` port).
+The ``extractText`` serialization lives in
+``services.orchestrators.conversion_service.ConversionService``, which calls
+the ``FileSerializer`` port (an in-process parser-dispatcher serializer).
 This module keeps HTTP transport only: the saved-file IO + cleanup,
 tool validation/dispatch, and the 4xx/5xx error mapping whose exact
 ``{"detail": ...}`` body the legacy endpoint returned via

openrag/core/indexing/serializer.py

@@ -1,15 +1,15 @@
-"""Transitional port for the document-serialization operation.
+"""Port for the document-serialization operation (file → raw text).
 
-``ConversionService`` (Phase 8E) exposes the ``extractText`` tool —
-serialize an uploaded file to raw text. The work runs in the
-``DocSerializer`` Ray actor; defining it on a dedicated port keeps the
-orchestrator Ray-free (8H: no Ray import / remote call under
-``services/orchestrators/``). A small shim in ``services/storage/``
-adapts the actor to this interface during the shim period; Phase 9
-swaps it for a direct serializer call and deletes the shim.
+``ConversionService`` exposes the ``extractText`` tool / ``/extract`` route —
+serialize an uploaded file to raw text. Defining the operation on a dedicated
+port keeps the orchestrator decoupled from the parser/Ray infrastructure
+(no Ray import / remote call under ``services/orchestrators/``); the
+composition root injects the concrete implementation
+(``services/workers/parsers/file_serializer.py::ParserFileSerializer``, which
+runs the parser dispatcher in-process).
 
-No Ray / LangChain types leak across this boundary — the serialized
-document is returned as its plain text content.
+No Ray / LangChain types leak across this boundary — the serialized document
+is returned as its plain text content.
 """
 
 from __future__ import annotations

openrag/core/models/document.py

@@ -118,6 +118,10 @@ def detect_content_type(filename: str) -> DocumentType:
             "png": DocumentType.IMAGE,
             "jpg": DocumentType.IMAGE,
             "jpeg": DocumentType.IMAGE,
+            "svg": DocumentType.IMAGE,
+            "gif": DocumentType.IMAGE,
+            "webp": DocumentType.IMAGE,
+            "bmp": DocumentType.IMAGE,
             "mp3": DocumentType.AUDIO,
             "wav": DocumentType.AUDIO,
             "flac": DocumentType.AUDIO,

openrag/di/container.py

@@ -589,17 +589,18 @@ def job_service(self) -> JobService:
     def conversion_service(self) -> ConversionService:
         """ConversionService — lazily built, cached for the container's lifetime.
 
-        The serializer is the Ray-backed ``SerializerRayShim`` during the
-        Phase-8 shim period (Ray cleanup is Phase 9); the DocSerializer
-        actor is resolved lazily per call inside the shim.
+        The serializer is the in-process ``ParserFileSerializer`` — it runs the
+        parser dispatcher directly (GPU backends still dispatch to their pool
+        actors) and implements the ``FileSerializer`` port, so the orchestrator
+        stays decoupled from the parser/Ray infrastructure.
         """
         if self._conversion_service is None:
             from services.orchestrators.conversion_service import ConversionService
-            from services.workers.parsers.doc_serializer_adapter import from_ray_namespace
+            from services.workers.parsers.file_serializer import build_file_serializer
 
             settings = self._require_settings()
             self._conversion_service = ConversionService(
-                serializer=from_ray_namespace(),
+                serializer=build_file_serializer(),
                 vector_store=self.vector_store,
                 collection=settings.vectordb.collection_name,
             )

openrag/services/orchestrators/conversion_service.py

@@ -4,9 +4,9 @@
 tool) and ``routers/extract.py`` (chunk-by-id lookup). Both were thin
 wrappers; this service keeps them Ray-free:
 
-- serialization runs in the ``DocSerializer`` Ray actor, reached through
-  the :class:`~core.indexing.serializer.FileSerializer` port (the
-  container injects the ``SerializerRayShim`` during the shim period);
+- serialization goes through the :class:`~core.indexing.serializer.FileSerializer`
+  port (the container injects the in-process ``ParserFileSerializer``, which runs
+  the parser dispatcher directly — GPU backends still dispatch to their pools);
 - chunk lookup goes through the clean :class:`VectorStore` port
   (``query_chunks_by_filter`` on the Milvus ``_id``), mirroring how
   PartitionService reads chunks — no LangChain ``Document`` leaks out.

openrag/services/workers/bootstrap.py

@@ -4,7 +4,6 @@
 detached actors the request path looks up by name:
 
 * TaskStateManager  — shared task-state actor
-* DocSerializer     — loader dispatcher
 * MarkerPool / DoclingPool / WhisperPool / WhisperActor — GPU parsers
 * llmSemaphore / vlmSemaphore / audioSemaphore — cluster-wide rate limiters
 
@@ -65,20 +64,14 @@ def get_task_state_manager():
     return get_or_create_actor("TaskStateManager", TaskStateManager, lifetime="detached")
 
 
-def get_serializer():
-    from services.workers.parsers.doc_serializer import DocSerializer
-
-    return get_or_create_actor("DocSerializer", DocSerializer, lifetime="detached")
-
-
 def get_marker_pool():
     from services.workers.parsers.docling_workers import DoclingPool
     from services.workers.parsers.marker_workers import MarkerPool
 
     config = _require_settings()
     pdf_loader = config.loader.file_loaders.pdf
     match pdf_loader:
-        case "DoclingLoader2":
+        case "DoclingLoader":
             return get_or_create_actor("DoclingPool", DoclingPool, lifetime="detached")
         case "MarkerLoader":
             return get_or_create_actor("MarkerPool", MarkerPool, lifetime="detached")
@@ -146,4 +139,3 @@ def initialize_worker_bootstrap(settings: "Settings") -> None:
     init_audio_actor()
     get_marker_pool()
     get_task_state_manager()
-    get_serializer()

openrag/services/workers/indexer_actor.py

@@ -1,12 +1,12 @@
 from __future__ import annotations
 
+import asyncio
 import traceback
 from datetime import datetime
 from pathlib import Path
 from typing import Any
 
 from core.models.document import Document
-from services.workers.parsers.doc_serializer_bridge import INDEXATION_CONFIG_METADATA_KEY
 from services.workers.pipeline_builder import IndexingPipeline
 
 
@@ -61,13 +61,13 @@ async def process_file(
         """
         await self._tsm.set_state.remote(task_id, "SERIALIZING")
         try:
-            document = _load_document(path, metadata, partition, indexation_config=indexation_config)
+            document = await _load_document(path, metadata, partition)
             # One indexation timestamp for this file, shared by the Milvus chunks
             # (via the store stage) and the Postgres catalog row, so they agree.
             row: dict[str, Any] = {
                 "document": document,
                 "partition": partition,
-                "filename": Path(path).name,
+                "filename": document.filename,
                 "language": metadata.get("language", "en"),
                 "replace": replace,
                 "user": user,
@@ -178,24 +178,47 @@ async def _replace_topic_tags_if_needed(
     )
 
 
-def _load_document(
+async def _load_document(
     path: str,
     metadata: dict[str, Any],
     partition: str,
-    *,
-    indexation_config: dict[str, Any] | None = None,
 ) -> Document:
     p = Path(path)
-    document_metadata = dict(metadata)
-    if indexation_config is not None:
-        document_metadata[INDEXATION_CONFIG_METADATA_KEY] = dict(indexation_config)
+    file_id = metadata.get("file_id")
+    if not file_id:
+        # file_id is a required route path param, force-set by
+        # IndexingService._build_metadata. Missing here means a broken upstream
+        # contract — fail loudly rather than silently persisting chunks under a
+        # non-queryable id (e.g. the temp upload's basename).
+        raise ValueError("_load_document requires metadata['file_id']")
+    # ``Document.id`` is the file's identity, not a random uuid: parsers set
+    # ``ProcessedDocument.document_id = document.id`` and the chunker uses that as
+    # ``Chunk.document_id`` / ``file_id``. If this defaulted to uuid4, chunks would
+    # persist under an id the ``/partition/{partition}/file/{file_id}`` lookup
+    # never queries by (zero chunks found).
+    #
+    # Per-partition indexation_config reaches the pipeline via ``row["indexation_config"]``
+    # (see IndexerWorker.process_file); it is intentionally not stamped into the
+    # document metadata so it never leaks into chunk metadata.
+    filename = _display_filename(path, metadata)
+    raw_bytes = await asyncio.to_thread(p.read_bytes)
     return Document(
-        filename=metadata.get("file_id") or p.name,
-        raw_bytes=p.read_bytes(),
-        content_type=Document.detect_content_type(p.name),
+        id=file_id,
+        filename=filename,
+        raw_bytes=raw_bytes,
+        content_type=Document.detect_content_type(filename),
         partition=partition,
-        metadata=document_metadata,
+        metadata=dict(metadata),
     )
 
 
+def _display_filename(path: str, metadata: dict[str, Any]) -> str:
+    """Return the user-facing filename while falling back to the stored path."""
+
+    filename = metadata.get("original_filename") or metadata.get("filename")
+    if filename:
+        return str(filename)
+    return Path(path).name
+
+
 __all__ = ["IndexerWorker"]

openrag/services/workers/indexer_pool.py

@@ -22,12 +22,13 @@ def __init__(self) -> None:
         from core.embeddings import embedder_registry
         from services.storage.milvus_store import MilvusVectorStore
         from services.storage.postgres_store import PostgresStore
-        from services.workers.parsers.doc_serializer_bridge import DocSerializerBridgeParser
+        from services.workers.parsers.parser_dispatcher import build_caption_vlm, build_parser_dispatcher
         from services.workers.pipeline_builder import build_indexing_pipeline
 
         cfg = load_config()
 
-        parser = DocSerializerBridgeParser(config=cfg)
+        parser = build_parser_dispatcher(cfg)
+        vlm = build_caption_vlm(cfg)
         chunker = _build_chunker(cfg)
         embedder_factory = _build_embedder_factory(cfg)
         contextualizer_factory = _build_contextualizer_factory(cfg)
@@ -51,6 +52,8 @@ def __init__(self) -> None:
             chunker=chunker,
             embedder=embedder,
             vector_store=self._vector_store,
+            vlm=vlm,
+            image_captioning=cfg.loader.image_captioning,
             chunker_factory=_build_chunker_from_config,
             embedder_factory=embedder_factory,
             contextualizer_factory=contextualizer_factory,