test: add semantic cache e2e test suite skeleton

[Pratham-Mishra04]

Summary

Adds a standalone end-to-end test suite for the semantic_cache plugin under tests/semanticcache. The suite validates the full caching lifecycle against a live Bifrost instance — plugin creation/teardown, cache miss/hit assertions, cross-provider behavior, streaming, and log cross-checking — without provisioning any infrastructure itself.

Changes

• e2e_test.go — TestMain entry point: loads config, initializes the report directory, checks Bifrost reachability, enforces plugin-absent precondition (with RUN_FORCE=1 auto-delete), runs all phases, and performs best-effort teardown on exit.
• preconditions_test.go — Phase 0 checks: Bifrost reachable, OpenAI configured, optional providers (Gemini, Anthropic) present with warnings if absent.
• http_test.go — HTTP helpers for all request types: chat completions (streaming and non-streaming), text completions, embeddings, image generation, and the Responses API. Each helper dumps full request/response bodies to the report directory for forensics.
• plugin_test.go — Plugin lifecycle helpers (pluginCreate, pluginUpdate, pluginDelete, pluginGet) mirroring the exact wire format the UI sends to /api/plugins.
• assert_test.go — Assertion helpers (assertMiss, assertHit, assertNoCacheDebug, assertSameCacheID, assertDifferentCacheID) plus a configurable async write-settle wait (SC_WRITE_SETTLE_MS) to account for the plugin's async PostLLMHook store write.
• cache_test.go — Cache management helpers (clearByCacheID, clearByCacheKey) wrapping the /api/cache/clear/* endpoints.
• logs_crosscheck_test.go — Cross-checks the persisted log row's cache_debug against the in-flight response stamp, with polling to handle Bifrost's async logging pipeline and float epsilon tolerance for JSON encoder differences.
• fixtures_test.go — Hand-curated paraphrase pairs for Phase 2 semantic cases, designed to land well above (canonical→paraphrase) or well below (canonical→unrelated) the default 0.8 similarity threshold.
• log_test.go — Structured per-run logging to reports/<UTC-timestamp>/run.log with optional TRAIL_SESSION_ID stamping for trail integration.
• go.mod — Standalone module (github.com/maximhq/bifrost/tests/semanticcache), consistent with the tests/governance pattern, excluded from the repo's go.work.
• README.md — Documents prerequisites, env vars, run commands, trail integration, and report output format.
• .gitignore — Excludes reports/ and *.log from version control.

Notable design decisions: the suite is intentionally verify-only (no infrastructure provisioning), uses a dedicated vector store namespace (BifrostSemanticCachePluginE2E) to isolate test data, and writes full wire-level request/response artifacts per step to support post-mortem debugging without re-running.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Requires a running Bifrost instance with Weaviate configured, OpenAI (required), and optionally Gemini and Anthropic providers.

cd tests/semanticcache

# All phases
GOWORK=off go test -v ./...

# Single phase
GOWORK=off go test -v -run TestPhase1_DirectOnly ./...

# Auto-delete any pre-existing plugin row before run
RUN_FORCE=1 GOWORK=off go test -v ./...

# Keep plugin after run for post-mortem inspection
RUN_KEEP_PLUGIN=1 GOWORK=off go test -v ./...

Environment variables:

Variable	Default	Purpose
BIFROST_URL	http://localhost:8080	Bifrost base URL
SC_CHAT_MODEL_OPENAI	openai/gpt-4o-mini	OpenAI chat model
SC_CHAT_MODEL_OPENAI_ALT	openai/gpt-4o	Alternate OpenAI model for cache-by-model cases
SC_EMBED_MODEL_OPENAI	text-embedding-3-small	Embedding model for Phase 2
SC_CHAT_MODEL_GEMINI	gemini/gemini-2.5-flash	Gemini chat model
SC_CHAT_MODEL_ANTHROPIC	anthropic/claude-haiku-4-5	Anthropic chat model
SC_NAMESPACE	BifrostSemanticCachePluginE2E	Vector store namespace
SC_WRITE_SETTLE_MS	500	Async write settle wait in ms
RUN_FORCE	unset	1 to delete pre-existing plugin before run
RUN_KEEP_PLUGIN	unset	1 to skip teardown on exit
TRAIL_SESSION_ID	unset	Stamped onto every log line for trail integration

Screenshots/Recordings

N/A

Breaking changes

• No

Related issues

N/A

Security considerations

No secrets are stored in the test suite. API keys are consumed from the existing Bifrost provider configuration and never passed directly through the test harness.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: abc17d8e-a99e-4c01-ac7d-1587cca94b3a

📥 Commits

Reviewing files that changed from the base of the PR and between 197b4b6 and 3579222.

📒 Files selected for processing (12)

• tests/semanticcache/.gitignore
• tests/semanticcache/README.md
• tests/semanticcache/assert_test.go
• tests/semanticcache/cache_test.go
• tests/semanticcache/e2e_test.go
• tests/semanticcache/fixtures_test.go
• tests/semanticcache/go.mod
• tests/semanticcache/http_test.go
• tests/semanticcache/log_test.go
• tests/semanticcache/logs_crosscheck_test.go
• tests/semanticcache/plugin_test.go
• tests/semanticcache/preconditions_test.go

✅ Files skipped from review due to trivial changes (2)

• tests/semanticcache/.gitignore
• tests/semanticcache/README.md

🚧 Files skipped from review as they are similar to previous changes (10)

• tests/semanticcache/preconditions_test.go
• tests/semanticcache/go.mod
• tests/semanticcache/logs_crosscheck_test.go
• tests/semanticcache/e2e_test.go
• tests/semanticcache/plugin_test.go
• tests/semanticcache/fixtures_test.go
• tests/semanticcache/log_test.go
• tests/semanticcache/cache_test.go
• tests/semanticcache/assert_test.go
• tests/semanticcache/http_test.go

---

📝 Walkthrough

Summary by CodeRabbit

• Tests

  • Added a comprehensive end-to-end semantic cache test suite with HTTP harness, logging, fixtures, cross-checks, plugin lifecycle helpers, and utilities for verifying cache hits/misses and cache-debug metadata.
  • Included helpers for clearing cache by ID/key and test precondition checks.

• Documentation

  • New README describing test setup, required services/endpoints, environment variables, run instructions, and generated report artifacts.

• Chores

  • Test artifacts and log files now ignored via .gitignore.

Walkthrough

This PR adds a complete end-to-end test suite for Bifrost's semantic cache functionality. It introduces a modular test infrastructure with HTTP utilities for OpenAI-compatible APIs (chat, embeddings, text completion, image generation), cache validation assertions, plugin management helpers, test fixtures, and precondition checks.

Changes

Semantic Cache End-to-End Test Suite

Layer / File(s)	Summary
Module setup and test harness entry point tests/semanticcache/go.mod, tests/semanticcache/e2e_test.go	Go module declaration and TestMain entry point that loads config, initializes logging, verifies Bifrost reachability, checks/deletes plugin preconditions (RUN_FORCE optional), runs tests, and performs teardown cleanup. Low-level HTTP helpers (doRaw, readAllSafe) and providersList decode provider responses.
Logging and reporting infrastructure tests/semanticcache/log_test.go	Per-run timestamped reports/<timestamp>/ directory, shared run.log with mutex, structured logf formatting (timestamp, level, phase/case/step), optional TRAIL_SESSION_ID correlation, reportPath, and dumpJSON for forensic artifacts.
HTTP configuration and request execution foundation tests/semanticcache/http_test.go (lines 1–45, 183–217)	Environment-driven config (Bifrost URL, model names, namespace, 120s HTTP client). Wire-schema types for cache debug and model payloads. doJSON marshals request bodies, POSTs to configured Bifrost endpoints, and returns status/body/headers with wrapped errors.
Chat completions API types and handlers tests/semanticcache/http_test.go (lines 47–112, 113–182, 219–283, 482–652)	Chat request/response/message/tool types with pointer semantics, cache header builder, postChat helper that dumps request/response JSON and extracts cache_debug, and SSE streaming support parsing data: events, aggregating chunks, detecting [DONE], and deriving cache-debug metadata.
Text completion, embeddings, image generation, and /v1/responses endpoints tests/semanticcache/http_test.go (lines 284–333, 334–381, 382–431, 432–480, 654–674)	Request/response types and postTextCompletion, postEmbedding, postImageGen, and postResponses helpers following shared execution, JSON dump, HTTP 200 enforcement, unmarshalling, and centralized logCacheDebugFields logging.
Plugin management helpers tests/semanticcache/plugin_test.go	Plugin request/response models and envelope types, config builders for direct-only and semantic modes, and HTTP helpers: pluginGet (404-aware), pluginCreate (expects 201), pluginUpdate (PUT with enabled+config), and pluginDelete (accepts 200/404).
Cache operations and cache debug assertions tests/semanticcache/cache_test.go, tests/semanticcache/assert_test.go	HTTP DELETE helpers clearByCacheID and clearByCacheKey. Configurable cacheWriteSettle (500ms default, overridable). Assertion helpers assertMiss, assertHit, assertNoCacheDebug, assertSameCacheID, assertDifferentCacheID, and nil-safe dereference helpers.
Test fixtures and log cross-validation tests/semanticcache/fixtures_test.go, tests/semanticcache/logs_crosscheck_test.go	Curated paraphrase and image prompt fixtures with lookup helpers. findLogByCacheDebug polls /api/logs for matching cache_debug. assertLogMatchesResponseCacheDebug compares response vs persisted log cache_debug fields with float epsilon tolerance and pointer equality checks, failing with aggregated diffs on mismatch.
Environment preconditions (Phase 0) tests/semanticcache/preconditions_test.go	TestPreconditions verifies Bifrost reachability (GET /api/plugins), requires openai provider, and emits PASS/WARN for optional gemini/anthropic providers via case-insensitive lookup helper.
Project documentation and gitignore tests/semanticcache/README.md, tests/semanticcache/.gitignore	README documents prerequisites, execution commands, required GOWORK=off, environment variables, trail log/session correlation, report layout (reports/<UTC-timestamp>/), and current implementation scope. .gitignore ignores reports/ and *.log.

🎯 3 (Moderate) | ⏱️ ~25 minutes

🐰 I hopped through tests with careful art,

fixtures tucked and helpers smart,
chat streams parsed and logs aligned,
cache hits traced and crosschecks signed,
semantic cache, well-tested start.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 49.02% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'test: add semantic cache e2e test suite skeleton' clearly and concisely describes the main change: adding a new end-to-end test suite for semantic cache, matching the substantial test file additions.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering all template sections: clear summary of purpose, detailed changes for each file, correct type (Chore/CI), affected areas (Plugins), thorough testing instructions with environment variables, and complete checklist.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch test-semanticcache-scaffolding

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Pratham-Mishra04]

• feat: add test-semantic-cache and test-semantic-cache-complete Makefile targets #3429
• test: add semantic cache plugin lifecycle tests #3428
• test: add semantic cache e2e test suite #3427
• test: add direct cache e2e test suite #3426
• test: add semantic cache e2e test suite skeleton #3425 👈 (View in Graphite)
• fix: decouple cache telemetry from write decision and guard no-op search paths in semantic cache #3424
• fix: resolve cache plugin at request time to support post-boot loads and plugin reloads #3423
• refactor: semantic cache ui revamp #3331
• feat: remove cleanup_on_shutdown from semantic cache plugin config #3330
• refactor: semantic cache plugin #3210 : 1 other dependent PR (#3240 )
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

This is a new, additive test module with no changes to production code paths; safe to merge.

All changes are confined to a new standalone test module that has no effect on the main Bifrost binary. The suite is verify-only and the only finding is a defensive gap in the pre-check that would produce confusing downstream failures rather than data loss or security risk.

tests/semanticcache/e2e_test.go — the plugin pre-check fallthrough behaviour is worth a second look before Phase 2+ cases are wired in.

Important Files Changed

Filename	Overview
tests/semanticcache/e2e_test.go	TestMain entry point with plugin pre-check, doRaw helper, and providersList; the pre-check silently treats all non-200 responses as plugin-absent, including 5xx errors
tests/semanticcache/http_test.go	HTTP helpers for all Bifrost request types; logCacheDebugFields hardcodes status=200 (previously flagged); cacheHeaders.apply idiom using a dummy http.Request is functional but unusual
tests/semanticcache/assert_test.go	Assertion helpers for cache hit/miss/no-debug and cache ID equality, with configurable async write-settle wait; clean and well-documented
tests/semanticcache/plugin_test.go	Plugin lifecycle helpers (create/update/delete/get) mirroring UI wire format; pluginGet decodes directly into pluginResponse while create/update use pluginEnvelope
tests/semanticcache/logs_crosscheck_test.go	Cross-checks persisted log cache_debug against in-flight response stamp with polling and float epsilon tolerance; well-reasoned design
tests/semanticcache/fixtures_test.go	Hand-curated paraphrase pairs with wide margin from 0.8 threshold on both sides; good design to avoid flaky boundary cases

_{Reviews (3): Last reviewed commit: "test(semanticcache): e2e suite scaffoldi..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/semanticcache/cache_test.go`:
- Around line 10-40: The tests currently append raw cache identifiers into
endpoint paths in clearByCacheID and clearByCacheKey which can break routes for
reserved characters; fix by URL-escaping the dynamic segment (use
net/url.PathEscape or url.PathEscape) before concatenating it into the path
passed to doJSON (e.g., replace direct use of cacheID/key with an escaped
variable), and add the necessary import for net/url if missing.

In `@tests/semanticcache/e2e_test.go`:
- Around line 41-42: The test currently ignores the error return from doRaw when
making the pre-check GET for "/api/plugins/"+pluginName; update calls to doRaw
(used in the plugin pre-checks around the GET for pluginName and similar calls
later) to capture the returned error, assert it is nil, and fail the test
immediately with a clear message (e.g., using t.Fatalf or t.Fatal) if an error
occurred, rather than allowing status to be 0 and producing misleading
downstream failures; modify both the initial plugin GET (where doRaw returns
status, body, _, _) and the later doRaw usages (the block around lines 52–56) to
handle and check the error return.
- Around line 125-133: The test incorrectly rejects a wrapped-but-empty response
because the condition in the providers-list parsing only returns when
len(wrap.Providers) > 0; change the logic in the decoding block that uses
json.Unmarshal into the variable wrap (and the field Providers) to accept the
wrapped shape even when empty (i.e., return wrap.Providers whenever
json.Unmarshal(body, &wrap) succeeds), so the fallback bare-list decoding is
only used when the wrapped decode fails.

In `@tests/semanticcache/http_test.go`:
- Around line 560-562: The test is mutating the caller's chatRequest by setting
req.Stream directly; instead create a local copy of the request before changing
the Stream field (e.g., copy := *req; copy.Stream = &streamFlag; and pass &copy
to postChatStream) so the original req remains unchanged; apply the same pattern
for the other occurrences noted (lines 567-575) to avoid leaking stream=true
into later calls.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: df3b7305-29b0-46ba-bb81-bea190c64650

📥 Commits

Reviewing files that changed from the base of the PR and between 6c1cd4b and 197b4b6.

📒 Files selected for processing (12)

• tests/semanticcache/.gitignore
• tests/semanticcache/README.md
• tests/semanticcache/assert_test.go
• tests/semanticcache/cache_test.go
• tests/semanticcache/e2e_test.go
• tests/semanticcache/fixtures_test.go
• tests/semanticcache/go.mod
• tests/semanticcache/http_test.go
• tests/semanticcache/log_test.go
• tests/semanticcache/logs_crosscheck_test.go
• tests/semanticcache/plugin_test.go
• tests/semanticcache/preconditions_test.go

[akshaydeo]

Merge activity

• May 12, 7:30 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 12, 7:43 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.