fix(rag): forward retrieval_filter from retrieval_config to vector store search

[devin-ai-integration]

TLDR

Problem this solves:

• /v1/rag/query dropped retrieval_config.retrieval_filter on the way to the vector store
• Bedrock Knowledge Bases then returned chunks from every metadata partition

How it solves it:

• Forward the filter to the vector store search as its filters argument
• Accept retrieval_config.filters and the SDK's top-level filters keyword too, retrieval_filter winning
• Return the upstream error's own status code on a failed query (a rejected filter is a 400, an unknown knowledge base a 404) instead of a blanket 500; the error body shape is unchanged

User Flow

Before: a developer who scopes a RAG query to one document partition gets an answer built from documents outside it

1. They send POST https://litellm-domain/v1/rag/query with a Bedrock chat model and retrieval_config: {"vector_store_id": "<knowledge base id>", "custom_llm_provider": "bedrock", "top_k": 5, "retrieval_filter": {"andAll": [{"equals": {"key": "department", "value": "billing"}}, {"equals": {"key": "doc_type", "value": "manual"}}]}}
2. The response is 200, but choices[0].message.provider_specific_fields.search_results.data lists chunks whose attributes.department is support or sales, and the answer quotes them

After: the same request only retrieves chunks from the filtered partition, so the answer is built from the right documents

1. They send the same POST https://litellm-domain/v1/rag/query
2. The response is 200 and every entry in choices[0].message.provider_specific_fields.search_results.data carries attributes.department: "billing" and attributes.doc_type: "manual"; the other partitions no longer appear
3. retrieval_config.filters works as an alias of retrieval_filter, and when both are set retrieval_filter wins
4. A filter the knowledge base rejects (an andAll with a single clause, say) comes back as a 400 carrying Bedrock's validation message

Relevant issues

Affected release

Linear ticket

Resolves LIT-4754

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Recording: Slack thread. Before was 226b1e1, after was b4f9e31, and current-main merge candidate d03556e710 was revalidated

Shared setup for both legs: one config, real Bedrock Knowledge Base L7INRFMVQT (us-west-2, five ingested text files, x-amz-bedrock-kb-source-uri metadata on each chunk), real bedrock/us.anthropic.claude-sonnet-5 completions, no mocks. Each leg is its own worktree and proxy booted with --num_workers 2 on a random port (Before 59140 at the merge base, After 39860 at the PR tip). The three cases are the same on both sides: the customer's retrieval_filter shape (an andAll of two clauses that only test_auth_v2.txt satisfies), the filters alias (a plain equals on test_bedrock_71f3a050.txt), a no-filter control, and a filter Bedrock rejects (an andAll with one clause)

model_list:
  - model_name: bedrock/us.anthropic.claude-sonnet-5
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-west-2

general_settings:
  master_key: sk-1234

python litellm/proxy/proxy_cli.py --config lit4754_config.yaml --port <port> --detailed_debug --num_workers 2

Before (226b1e1)

retrieval_filter (customer shape, andAll of two clauses)

1. Run the query against the live proxy

   curl -s http://localhost:59140/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "retrieval_filter": {"andAll": [{"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_auth_v2.txt"}}, {"notEquals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_bedrock_71f3a050.txt"}}]}}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "basic_ingest_a0a725e7.txt",
       "ingest_query_c500aeea.txt",
       "test_bedrock_a9a4a586.txt",
       "test_bedrock_71f3a050.txt",
       "test_bedrock_61f947df.txt"
     ],
     "answer": "Looking at the context provided, there are **five different test documents**, each associated with a different ingestion"
   }

filters alias (plain equals)

1. Run the query against the live proxy

   curl -s http://localhost:59140/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "filters": {"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_bedrock_71f3a050.txt"}}}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "basic_ingest_a0a725e7.txt",
       "ingest_query_c500aeea.txt",
       "test_bedrock_a9a4a586.txt",
       "test_bedrock_71f3a050.txt",
       "test_bedrock_61f947df.txt"
     ],
     "answer": "I don't see a specific document ID indicated in your question, and the context contains multiple test documents with dif"
   }

no filter (control)

1. Run the query against the live proxy

   curl -s http://localhost:59140/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "basic_ingest_a0a725e7.txt",
       "ingest_query_c500aeea.txt",
       "test_bedrock_a9a4a586.txt",
       "test_bedrock_71f3a050.txt",
       "test_bedrock_61f947df.txt"
     ],
     "answer": "Based on the context provided, there isn't a single definitive answer since multiple test documents are listed with diff"
   }

malformed filter (andAll with one clause, which Bedrock rejects)

1. Run the query against the live proxy

   curl -s -o out.json -w 'HTTP %{http_code}\n' http://localhost:59140/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "retrieval_filter": {"andAll": [{"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_auth_v2.txt"}}]}}}'; jq -c 'if .choices then {files: [.choices[0].message.provider_specific_fields.search_results.data[].filename]} else . end' out.json

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   HTTP 200
   {"files":["basic_ingest_a0a725e7.txt","ingest_query_c500aeea.txt","test_bedrock_a9a4a586.txt","test_bedrock_71f3a050.txt","test_bedrock_61f947df.txt"]}

After (b4f9e31)

retrieval_filter (customer shape, andAll of two clauses)

1. Run the query against the live proxy

   curl -s http://localhost:39860/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "retrieval_filter": {"andAll": [{"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_auth_v2.txt"}}, {"notEquals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_bedrock_71f3a050.txt"}}]}}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "test_auth_v2.txt"
     ],
     "answer": "This document was ingested using **BaseAWSLLM authentication**.\n\nAccording to the context provided, LiteLLM supports mul"
   }

filters alias (plain equals)

1. Run the query against the live proxy

   curl -s http://localhost:39860/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "filters": {"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_bedrock_71f3a050.txt"}}}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "test_bedrock_71f3a050.txt"
     ],
     "answer": "Based on the context provided, this document (Test document 71f3a050) was ingested for **Bedrock Knowledge Base** purpos"
   }

no filter (control)

1. Run the query against the live proxy

   curl -s http://localhost:39860/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5}}' | jq '{files: [.choices[0].message.provider_specific_fields.search_results.data[].filename], answer: .choices[0].message.content[:120]}'

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   {
     "files": [
       "basic_ingest_a0a725e7.txt",
       "ingest_query_c500aeea.txt",
       "test_bedrock_a9a4a586.txt",
       "test_bedrock_71f3a050.txt",
       "test_bedrock_61f947df.txt"
     ],
     "answer": "Based on the context provided, there are multiple documents mentioned, each indicating a different ingestion method:\n\n- "
   }

malformed filter (andAll with one clause, which Bedrock rejects)

1. Run the query against the live proxy

   curl -s -o out.json -w 'HTTP %{http_code}\n' http://localhost:39860/v1/rag/query -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model": "bedrock/us.anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "How was this document ingested?"}], "retrieval_config": {"vector_store_id": "L7INRFMVQT", "custom_llm_provider": "bedrock", "top_k": 5, "retrieval_filter": {"andAll": [{"equals": {"key": "x-amz-bedrock-kb-source-uri", "value": "s3://bedrock-kb-test-888602223428/data/test_auth_v2.txt"}}]}}}'; jq -c 'if .choices then {files: [.choices[0].message.provider_specific_fields.search_results.data[].filename]} else . end' out.json

2. Observed output (files = search_results chunks the proxy retrieved, answer = first 120 characters of the completion, or the status and error body when the query fails)

   HTTP 400
   {"detail":{"error":"litellm.BadRequestError: BedrockException - {\"message\":\"1 validation error detected: Value at 'retrievalConfiguration.vectorSearchConfiguration.filter.andAll' failed to satisfy constraint: Member must have length greater than or equal to 2\"}"}}

Observations from the run:

• Before ignores both filter spellings; five files come back every time
• After returns only the file the filter allows, answer follows
• Control case identical on both legs, so no unfiltered regression
• Rejected filter: Before 200 unfiltered, After 400 with Bedrock's message
• Unknown knowledge base id: Before 500, After 404, same message
• Streaming with a filter answers from the filtered file on After
• Managed-storage KBs reject vectorSearchConfiguration filters (aws CLI check); pre-existing
• Current-main merge candidate d03556e710 (main 0b3e564) matched the exact-tip results for retrieval_filter, alias, control, 400/404 errors, and streaming

Type

🐛 Bug Fix

Caveats (if any)

Low

• A filter the vector store cannot apply now fails the request with the store's status and message; before this PR it was silently ignored
• Every failed /v1/rag/query now carries the upstream status code (404 for an unknown knowledge base, say) where it used to be a 500; the {"detail": {"error": ...}} body is unchanged
• Top-level filters remains SDK-only; raw /v1/rag/query requests must place filters inside retrieval_config

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/359598ee8147428892f5f8da6f9492c1

• b4f9e31 passes /live-pr-risk

---

Note

Medium Risk
Changes RAG retrieval semantics (invalid filters now fail instead of being ignored) and alters HTTP status codes on /v1/rag/query errors, which may affect clients that assumed 500 for all failures.

Overview
Fixes RAG queries that ignored metadata filters, so scoped retrieval (e.g. Bedrock Knowledge Base retrieval_filter) now reaches vector_stores.asearch as filters. Resolution order is retrieval_config.retrieval_filter, then retrieval_config.filters, then the SDK filters kwarg; when several are set, retrieval_filter wins.

The proxy /v1/rag/query error path no longer maps every failure to 500—it uses _upstream_status_code so provider errors like 400 (bad filter) or 404 (unknown KB) propagate while keeping the same {"detail": {"error": ...}} body. RAGRetrievalConfig types now document retrieval_filter and the filter fields.

^{Reviewed by Cursor Bugbot for commit b4f9e31. Bugbot is set up for automated code reviews on this repo. Configure here.}

Open in Devin Desktop: https://app.devin.ai/desktop/session/359598ee8147428892f5f8da6f9492c1?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This PR forwards RAG retrieval filters to vector-store searches and preserves upstream error status codes.

• Adds support for retrieval_filter, retrieval_config.filters, and top-level filters, with explicit precedence.
• Removes top-level filters from forwarded keyword arguments to avoid the previously reported duplicate-keyword collision.
• Extends RAG retrieval types and adds regression coverage for filter forwarding and upstream errors.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain within the scope of this follow-up review.

Important Files Changed

Filename	Overview
litellm/rag/main.py	Selects and forwards retrieval filters while removing the top-level keyword that previously collided with the explicit search argument.
litellm/proxy/rag_endpoints/endpoints.py	Preserves integer upstream status codes when translating RAG query failures into HTTP responses.
litellm/types/rag.py	Adds typed support for the new retrieval-filter spelling and tightens retrieval configuration field types.
tests/test_litellm/rag/test_main.py	Adds coverage for filter aliases, precedence, top-level forwarding, and unfiltered behavior.
tests/test_litellm/proxy/rag_endpoints/test_rag_endpoints.py	Adds endpoint coverage for preserving upstream error status codes.

_{Reviews (3): Last reviewed commit: "fix(proxy): surface the upstream status ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptile the P1 (top-level filters kwarg colliding with the explicit filters arg) is fixed in 232b9e8 — the pipeline now pops filters off kwargs and uses it as a fallback source, with a regression test (test_aquery_top_level_filters_kwarg_does_not_collide) that reproduces the TypeError on the prior commit. Please re-review the current head.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_bedrock_rag_retrieval_filter (b4f9e31) with main (0b3e564)}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b4f9e31. Configure here.}

[devin-ai-integration]

Revalidated b4f9e31 and its current-main merge candidate against a real Bedrock Knowledge Base. Filters, streaming, and 400/404 propagation passed. Recording

[mateo-berri]

LGTM