test(vcr): fix CI failures from PR #27159 (JSONL crash + replay-incompatible tests)

[mateo-berri]

Stacks on top of #27159. Fixes the six CI jobs that flipped to failing after VCR auto-marking was extended to additional test directories.

Failures and root causes

I checked all six failing jobs on the latest CircleCI workflow for #27159 and compared them against the most recent baseline run on litellm_internal_staging. The baseline itself is heavily red right now because of RateLimitError: You exceeded your current quota from the OpenAI test key — the PR actually clears most of those by replaying cached responses. The remaining failures break down as follows.

Genuinely caused by the PR (this PR fixes)

Job	Failing test(s)	Why
batches_testing	test_async_create_file	vcrpy's stock body matcher crashes (json.JSONDecodeError: Extra data) on the JSONL S3 PUT body, before it can return "no match".
logging_testing	test_basic_s3_logging[*] (4 params)	vcrpy's boto3_stubs intercepts the S3 PUT done by the s3 success-callback. The test then does a real S3 LIST to verify; cached LIST replay never contains the freshly generated response_id.
logging_testing	test_async_embedding_azure	Failure-path test passes api_key="my-bad-key". We scrub auth headers, so the bad-key request matches the prior good-key cassette and replays a 200; failure callback never fires.
local_testing_part1	test_amazing_sync_embedding	Same root cause as above.
litellm_assistants_api_testing	step timeout (15m)	OpenAI Assistants polling (while run.status != "completed": ...) mints fresh thread/run/message IDs every recording session. Polled GETs against new IDs can never match cached cassettes, so every CI run effectively re-records and the suite blows past the no_output_timeout.

Pre-existing on baseline (not touched here)

• litellm_router_testing :: test_acompletion_caching_with_ttl_on_router — also fails on baseline.
• llm_translation_testing :: test_function_calling_with_tool_response — single gw5 worker crash; flake.

Fix 1: safe body matcher

vcrpy's body matcher in vcr/matchers.py inspects Content-Type and unconditionally runs json.loads on application/json bodies. JSON Lines payloads (the Bedrock batch S3 PUT and similar upload paths) crash that. Added _safe_body_matcher to tests/_vcr_conftest_common.py, registered it as safe_body, and replaced "body" with SAFE_BODY_MATCHER_NAME in the shared match_on tuple.

The matcher compares request bodies as bytes (with str → bytes normalization). It is strictly more conservative than vcrpy's default — the only thing it gives up is "different JSON key order is treated as the same body", which doesn't apply to deterministic litellm-built request payloads. There is no risk of cross-contamination: it can never produce a false positive that the default would have rejected. The trade-off is that bodies containing nondeterministic values (UUIDs, timestamps) produce a cache miss; the right fix for those cases is a before_record_request scrubber, not a smarter matcher.

Confirmed via a small reproduction that vcrpy's default body matcher raises json.JSONDecodeError on a JSONL body, and that _safe_body_matcher accepts it. The reproduction is encoded in tests/test_litellm/test_vcr_safe_body_matcher.py (9 unit tests, all passing locally).

Fix 2: skip lists for replay-incompatible tests

The existing apply_vcr_auto_marker_to_items helper already supported skip_files and skip_nodeid_suffixes. Added entries for the four cases above:

• tests/logging_callback_tests/conftest.py:
  • _VCR_INCOMPATIBLE_FILES = {"test_amazing_s3_logs.py"}
  • _VCR_INCOMPATIBLE_NODEID_SUFFIXES = ("::test_async_embedding_azure",)
• tests/local_testing/conftest.py:
  • _VCR_INCOMPATIBLE_FILES = {"test_assistants.py"}
  • _VCR_INCOMPATIBLE_NODEID_SUFFIXES = ("::test_amazing_sync_embedding",)

These tests fall back to the pre-PR behavior (live calls, no cache). The remaining tests in each directory still benefit from caching.

Tests

$ uv run --no-sync python -m pytest tests/test_litellm/test_vcr_safe_body_matcher.py -v
...
============================== 9 passed in 0.37s ===============================

The 9 unit tests cover: matcher registration in match_on, identical-bytes match, str/bytes equivalence, JSONL acceptance (the bug we're fixing), JSONL difference rejection, byte difference rejection, None body equality, JSON-key-order non-normalization (documenting the conservative-vs-default trade-off), and the documentation test that confirms vcrpy's default body matcher crashes on JSONL with json.JSONDecodeError.

Slack Thread

  

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}