feat(llm-api-gateway): add prompt cache key to chat completions

[FamousDirector]

TL;DR

• Adds OpenAI-compatible prompt_cache_key to Chat Completions using the existing NVCF cache-affinity path.
• Preserves the raw body and response-header value while sending only a SHA-256-derived router key.

Additional Details

• Precedence is existing request context, nonempty body key, validated session header, then messages hash.
• Reuses the 256-byte and control-character validation; an invalid session header still returns 400 before body selection.
• Keeps header-only and messages-hash behavior compatible. No Stargate, configuration, chart, CLI, dependency, or response-body change is needed.
• Documents body-key and returned-header reuse with a gateway-to-router sequence.

For the Reviewer

• Review affinity precedence and source-sensitive hashing in api/session_affinity.go.
• Review raw-body and hashed-header assertions in provider/stargate_test.go.

For QA

• go test ./api ./provider
• go test ./...
• go build ./...
• bazel test //src/invocation-plane-services/llm-api-gateway/... --test_output=errors --flaky_test_attempts=2
• ./tools/ci/check-docs
• git diff --check
• Separate manual QA is not required. Go and docs checks retain their existing advisory local-environment warnings.

Issues

Closes #822

Checklist

• Read the Contributing Guidelines.
• Commit includes the Developer Certificate of Origin (DCO) sign-off.
• Tests cover the implementation.
• Documentation is up to date.

Summary by CodeRabbit

• New Features

  • Added optional prompt_cache_key support for chat completion and Responses requests.
  • Improved session stickiness by prioritizing prompt cache keys and returning a reusable session ID.
  • Preserved prompt cache keys in forwarded requests while securely deriving routing affinity values.

• Bug Fixes

  • Added validation for invalid or oversized cache keys and session identifiers.

• Documentation

  • Expanded configuration, routing behavior, precedence rules, and validation guidance.

• Tests

  • Added coverage for streaming, non-streaming, hashing, fallback, and validation scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6122e8cf-817a-4789-bbcd-445c30008eb8

📥 Commits

Reviewing files that changed from the base of the PR and between 6384071 and deffba0.

📒 Files selected for processing (8)

• docs/user/llm-gateway.md
• docs/user/llm-request-router-load-balancing.md
• src/invocation-plane-services/llm-api-gateway/README.md
• src/invocation-plane-services/llm-api-gateway/api/openai_routes_test.go
• src/invocation-plane-services/llm-api-gateway/api/session_affinity.go
• src/invocation-plane-services/llm-api-gateway/api/session_affinity_test.go
• src/invocation-plane-services/llm-api-gateway/models/openai.go
• src/invocation-plane-services/llm-api-gateway/provider/stargate_test.go

---

📝 Walkthrough

Walkthrough

The gateway now accepts prompt_cache_key for Chat Completions. It prioritizes and validates the key, preserves the raw value, returns it in the session header, and forwards only its SHA-256-derived affinity value. Documentation and tests cover unary, streaming, Responses, and Stargate behavior.

Changes

Prompt cache affinity

Layer / File(s)	Summary
Prompt cache affinity contract src/invocation-plane-services/llm-api-gateway/models/openai.go, src/invocation-plane-services/llm-api-gateway/api/session_affinity.go	ChatCompletionRequest accepts prompt_cache_key. Chat affinity prioritizes and validates it, then hashes the selected session value for internal routing.
Affinity behavior validation src/invocation-plane-services/llm-api-gateway/api/*_test.go, src/invocation-plane-services/llm-api-gateway/provider/stargate_test.go	Tests cover precedence, fallback, invalid values, streaming and non-streaming response headers, hashed routing headers, and raw payload preservation.
Affinity usage documentation docs/user/llm-gateway.md, docs/user/llm-request-router-load-balancing.md, src/invocation-plane-services/llm-api-gateway/README.md	Documentation describes prompt cache key support, hashing, request-body forwarding, session-header reuse, router configuration, and validation steps.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: ⚪ Minimal · up to deffb

The PR adds prompt-cache affinity handling while preserving existing request and response behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ChatCompletions
  participant SessionAffinity
  participant Stargate
  Client->>ChatCompletions: Send prompt_cache_key
  ChatCompletions->>SessionAffinity: Select and validate affinity source
  SessionAffinity->>Stargate: Forward raw body and hashed x-cache-affinity-key
  Stargate-->>Client: Return x-multi-turn-session-id

Loading

Suggested reviewers: vrv3814

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title uses valid Conventional Commits syntax and accurately describes the primary feature change.
Linked Issues check	✅ Passed	The changes implement Chat Completions prompt_cache_key support, precedence, validation, hashing, forwarding, documentation, and required tests for issue [#822].
Out of Scope Changes check	✅ Passed	The code, tests, provider coverage, and documentation changes are directly related to the linked issue objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jcameron/feat/prompt-cache-key-chat-completions

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-jcameron-feat-prompt-cache-key-chat-completions.docs.buildwithfern.com/nvcf

[balajinvda]

🎉 This PR is included in version nvcf-llm-api-gateway-v0.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀