fix(embedder): avoid vLLM exact-max_model_len hang; tune batch_size default

[Ahmath-Gadji]

Problem

When indexing certain files, embedding completes every batch except one, which hangs indefinitely — systematically reproducible for the same file, while other files index fine in parallel (the server keeps serving). So it's content-related, not load or client.

Root cause: vLLM issue #29496 — Qwen3-Embedding (and other vLLM pooling models) hang forever on an embedding request whose input is exactly max_model_len tokens long. Reducing the input by a single token fixes it; the issue was closed "not planned".

Our embedder sent truncate_prompt_tokens = max_model_len, so any chunk ≥ max_model_len was truncated straight onto that hang boundary, wedging that one batch while the rest of the document (and other files) embedded normally.

Changes

1. Truncate one token below the boundary. _embed_batch now sends truncate_prompt_tokens = max(1, max_model_len - 1) so a truncated chunk lands at max_model_len - 1, never on the hang boundary. Applies to every VLLMEmbedder (default and named-endpoint paths). Floored at 1 for safety.

2. Lower the default embedder batch_size 64 → 32 across the three sources of truth (conf/config.yaml, EmbedderConfig, VLLMEmbedder constructor). Smaller batches shrink each request and the blast radius of any single slow/stuck batch, and align the legacy embedder default with the named-endpoint ModelEndpointConfig default (also 32). timeout (120s) and embed_concurrency (4) are unchanged — already reasonable; all three remain overridable via EMBEDDER_TIMEOUT / EMBEDDER_BATCH_SIZE / EMBEDDER_CONCURRENCY.

Tests

• test_truncate_prompt_tokens_is_one_below_max_model_len (8192 → 8191) and a new test_truncate_prompt_tokens_floors_at_one.
• Full inference / config / di unit suites pass (280 passed); ruff clean.

Summary by CodeRabbit

• Bug Fixes

  • Fixed token truncation logic to properly handle edge cases when processing inputs near model token limits.

• Chores

  • Optimized default embedding batch size to improve resource efficiency and processing stability.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f9f5ec97-6feb-40e3-9248-9e974fd875ae

📥 Commits

Reviewing files that changed from the base of the PR and between 79e8d58 and 80e1a4f.

📒 Files selected for processing (4)

• conf/config.yaml
• openrag/core/config/endpoints.py
• openrag/services/inference/vllm_client.py
• tests/unit/services/inference/test_vllm_client.py

---

📝 Walkthrough

Walkthrough

batch_size defaults are reduced from 64 to 32 in conf/config.yaml, EmbedderConfig, and VLLMEmbedder.__init__. The _embed_batch method changes truncate_prompt_tokens from max_model_len to max(1, max_model_len - 1) to avoid an off-by-one boundary, with two new unit tests covering the normal and floor-at-1 cases.

Changes

Embedder batch size and truncation fixes

Layer / File(s)	Summary
batch_size default reduction openrag/core/config/endpoints.py, conf/config.yaml, openrag/services/inference/vllm_client.py	EmbedderConfig.batch_size default and VLLMEmbedder.__init__ default are both lowered from 64 to 32; the YAML config value is updated to match.
truncate_prompt_tokens off-by-one fix and tests openrag/services/inference/vllm_client.py, tests/unit/services/inference/test_vllm_client.py	_embed_batch now sets truncate_prompt_tokens to max(1, self._max_model_len - 1) with comments explaining the boundary condition; test_truncate_prompt_tokens_included is replaced by test_truncate_prompt_tokens_is_one_below_max_model_len and test_truncate_prompt_tokens_floors_at_one.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• linagora/openrag#465: Introduced batch_size batching via the same VLLMEmbedder constructor parameter and EmbedderConfig field that this PR modifies.

Suggested labels

fix

Poem

🐇 Thirty-two chunks, not sixty-four,
A gentler batch through the embedder's door.
One token shy of the model's max len,
No off-by-one shall trip us again!
Hippity-hop, the tests all pass —
This rabbit fixed the boundary at last. 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the two main changes: fixing a vLLM hanging issue by adjusting truncation behavior and reducing the default batch_size from 64 to 32.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/embedder-truncate-hang

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}