test: add strict stateless provider replay identity

[yuneng-berri]

TLDR

Problem this solves:

• Replay ignores semantic headers and collapses literal request values
• Repeated query values can reuse the wrong recorded response

How it solves it:

• Add opt-in stateless_v1 matching for chat and messages
• Bind requests to destination, headers, queries and exact JSON
• Preserve legacy fixtures and reject incompatible strict bundles

User Flow

Before: an E2E contributor can replay success after changing a prompt

1. Record POST http://127.0.0.1:<edge-port>/openai/v1/chat/completions with prompt 2031-04-05
2. Switch to replay and send the same request with 2032-06-07
3. Receive the recorded HTTP 200 despite changing the request

After: the contributor enables E2E_REPLAY_MATCH_PROFILE=stateless_v1 for recording and replay

1. Record POST http://127.0.0.1:<edge-port>/openai/v1/chat/completions with prompt 2031-04-05
2. Switch to replay and send the same request with 2032-06-07
3. Receive HTTP 599 naming a body mismatch, without an upstream call

The same contract applies to /anthropic/v1/messages. Leaving the profile unset preserves legacy behavior

Relevant issues

Affected release

Linear ticket

Resolves LIT-7733

Pre-Submission checklist

• I have added meaningful tests
• The focused harness files pass locally: 156 passed, no reruns
• My PR passes all required CI/CD checks
• Scope is limited to opt-in test-harness request matching
• Greptile has independently reviewed the current head at 5/5

Screenshots / Proof of Fix

This is test-harness behavior. All HTTP evidence uses temporary synthetic providers, with no proxy/database deployment or paid-provider calls

Shared setup: run the existing harness edge against its synthetic HTTP provider with an empty temporary fixture directory. Record one request, close the recording edge, reload the bundle, and start a replay edge with the same provider destination. The provider counter starts at zero. $EDGE is the allocated edge address. Use E2E_REPLAY_MATCH_PROFILE=stateless_v1 for the updated capture and replay

The request command is:

curl --silent --show-error --noproxy '*' -w '\nHTTP %{http_code}\n' \
  -H 'content-type: application/json' -H 'anthropic-beta: feature-a' \
  --data-binary '{"model":"synthetic","messages":[{"role":"user","content":"2031-04-05"}]}' \
  "$EDGE/openai/v1/chat/completions"

Use the same command with /anthropic/v1/messages for the messages case

Before (d1fc231)

Chat completions

1. Execute the recording command: HTTP 200, response echoes 2031-04-05
2. Against the replay edge, change the prompt to 2032-06-07: HTTP 200, response still echoes 2031-04-05
3. Provider counter: one capture call and zero replay calls

Messages

1. Execute the recording command at /anthropic/v1/messages: HTTP 200, response echoes 2031-04-05
2. Against the replay edge, change the prompt to 2032-06-07: HTTP 200, response still echoes 2031-04-05
3. Provider counter: one capture call and zero replay calls

After (37bde0b)

Chat completions

1. Execute the strict recording command: HTTP 200, response echoes 2031-04-05
2. Against the strict replay edge, change the prompt to 2032-06-07: HTTP 599 with stateless_v1 replay mismatch: body; re-record with E2E_FIXTURE_MODE=record
3. Provider counter: one capture call and zero replay calls

Messages

1. Execute the strict recording command at /anthropic/v1/messages: HTTP 200, response echoes 2031-04-05
2. Against the strict replay edge, change the prompt to 2032-06-07: HTTP 599 with stateless_v1 replay mismatch: body; re-record with E2E_FIXTURE_MODE=record
3. Provider counter: one capture call and zero replay calls

Qualification

The five harness files passed 156 tests with reruns disabled. Coverage includes precise numbers and JSON types, header presence/value changes, destination changes, ordered queries, credential redaction, fresh-process replay, legacy bundle loading, finite consumption, leftovers, multipart/binary compatibility, stream chunk boundaries and truncation

A temporary functional fault omitted anthropic-beta from identity. Both protocol checks failed at 200 == 599 after a changed header incorrectly received the recorded success. Restoring the implementation passed both controls. No diagnostic fault or fixture bundle is committed

The scoped repository check script passed E2E typing, test-tree lint, raw HTTP restrictions and test-quality budgets. Workflow startup and CI coverage/slice checks passed. The CircleCI provider_replay_harness job explicitly executes the five markerless harness files and uploads JUnit results. It uses the existing CircleCI trigger policy; this PR adds no automatic PR trigger

Independent review identified numeric precision, query decoding and authentication edge cases; each was corrected and verified. Strict JSON now preserves numeric tokens exactly (including 1, 1e0, -0 and enormous exponents), while only authorization scheme casing is normalized. Semantic header values remain exact. Legacy interaction serialization was also compared byte-for-byte with the base

The implementation at 37bde0bdbe passed all 156 tests in the previous GHA run, received independent Greptile 5/5, and passed Veria with no security issues. The latest change moves execution to CircleCI and removes the added GHA step. CircleCI configuration validates; the new job has not run in CircleCI yet. Fresh current-head review is pending. Bugbot has no result available and is not counted as a pass

The strict tests live in tests/code_coverage_tests/test_provider_replay_harness.py, preserving all eight methods and including their path in the CI selector. The synthetic provider stores immutable request snapshots

The Vertex job on the earlier head 37bde0bdbe failed two Gemini embedding cost assertions that also failed on its exact base. That result is historical: the later main merge and CI move require fresh current-head checks

Type

Test

Caveats

Low

• Strict enrollment covers only stateless chat and messages
• Credentials, model availability and transport performance remain unverified
• SDK timeout/retry metadata is excluded from request identity
• Inbound transfer encoding is unsupported; use content-length framing

QA runbook

Strict harness tests live outside tests/e2e, alongside CI tooling tests. They reuse the existing synthetic provider and transport helpers: actual HTTP requests cross the edge and synthetic upstream, recordings are written to disk and loaded for replay, and one case replays in a fresh process. They exercise the record/storage/replay boundary without mocked client calls or direct canonicalizer-only assertions. No provider credentials or live proxy are required

• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_roundtrip_rejects_semantic_changes
  • Record chat and messages, then change date, UUID, model, array order, query order, header values/presence or auth scheme; expect HTTP 599
  • Reorder JSON keys and header casing; expect the original bytes and no extra provider call
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_json_values_remain_distinct
  • Record each null/empty/false/zero/array/object/precise-number value and replay a different value; expect HTTP 599 naming body
  • Replay the original value; expect HTTP 200, including 1e400
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_ineligible_capture_never_calls_provider
  • Send unsupported endpoints, headers, bodies, invalid UTF-8 queries, duplicate JSON keys; expect eligibility errors
  • Confirm no provider call or interaction file was created
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_destination_is_part_of_actual_http_identity
  • Record a request, change its upstream base path and replay; expect HTTP 599 naming upstream
  • Confirm the provider counter still records only the capture call
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_credentials_are_not_identity_and_fresh_process_replays
  • Capture synthetic auth, cookie and query credentials; inspect stored requests and verify original values reached the provider
  • Start a fresh replay process with different credential values; expect original response bytes and no provider call
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_profiles_cannot_load_each_others_bundles
  • Load each bundle using the other profile; expect a profile mismatch with a rerecord instruction
  • Confirm neither incompatible bundle is silently upgraded
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_strict_stream_preserves_chunks_and_truncation
  • Record complete and interrupted streams, then replay; compare exact transfer chunks and ending
  • Confirm all expected interactions were consumed without contacting the provider
• tests/code_coverage_tests/test_provider_replay_harness.py::TestStrictIdentity::test_auth_scheme_survives_missing_credentials
  • Record credential-less Bearer, then replay Basic or Digest; expect HTTP 599 naming auth
  • Replay Bearer with synthetic credentials; expect success because credential validity is outside matching

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Finite tests cannot establish that regressions are impossible

[greptile-apps]

Greptile Summary

This PR adds an opt-in strict replay profile for stateless OpenAI chat-completion and Anthropic messages requests.

• Matches exact JSON values, ordered query pairs, semantic headers, authorization schemes, and upstream destinations while redacting credentials.
• Separates strict and legacy bundle formats and rejects incompatible fixture bundles.
• Adds focused replay, streaming, redaction, compatibility, and fresh-process tests.
• Runs the provider replay harness in every admitted CircleCI workflow by removing the restrictive branch filter.

Confidence Score: 5/5

The PR appears safe to merge; the previously identified CircleCI coverage gap is fixed and no new actionable issue was found.

Removing the harness job’s branch filter ensures every pipeline admitted by the existing workflow runs the replay suite, so ordinary feature branches no longer skip this coverage. All six previous threads were manually resolved without explanatory replies; the current code also directly addresses their reported concerns through exact numeric identities, normalized authorization schemes, test relocation, immutable request snapshots, compliant formatting, and unrestricted harness job execution.

Important Files Changed

Filename	Overview
.circleci/config.yml	Adds the provider replay harness job and now runs it in every admitted CircleCI workflow without the feature-branch-excluding filter.
tests/e2e/fixture_profile.py	Defines strict request eligibility and identity construction with exact numeric tokens, ordered queries, semantic headers, credential redaction, and normalized authorization schemes.
tests/e2e/provider_edge.py	Integrates profile-aware identity generation, recording, replay matching, diagnostics, and bundle loading into the test provider edge.
tests/e2e/fixture_bundle.py	Adds profile-specific bundle metadata, format validation, strict identities, and legacy-compatible serialization.
tests/code_coverage_tests/test_provider_replay_harness.py	Adds focused end-to-end coverage for strict identity mismatches, numeric fidelity, eligibility, destination matching, credential handling, profile compatibility, and streaming.
tests/e2e/test_provider_edge.py	Extends the synthetic provider with immutable request snapshots used by the strict replay tests.

_{Reviews (6): Last reviewed commit: "ci: run replay harness on every admitted..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptileai review

[yuneng-berri]

@greptileai review. Strict harness tests now live outside tests/e2e; all 156 HTTP, persistence, and replay controls pass.

[yuneng-berri]

@greptileai review. Request history now uses immutable snapshots, and the payload expression fits the line limit; all 156 tests pass.

[yuneng-berri]

@greptileai review. Harness execution moved from GHA to CircleCI under existing triggers; configuration validates and all 156 tests pass locally.

[yuneng-berri]

@greptileai review. Removed the harness job branch filter so every admitted CircleCI pipeline runs it; run-ci and cadence triggering remain unchanged.