fix: include provider in cache key for uniqueness

[steebchen]

Summary

• Fixes cache key uniqueness by including the provider field in the cache payload

Changes

Core Functionality

• Added provider: usedProvider to the cache payload in apps/gateway/src/chat/chat.ts to ensure cache keys are unique across different providers

Test plan

• Verify caching behavior with multiple providers
• Confirm cache hits and misses are accurate when provider changes
• Run existing tests to ensure no regressions in chat functionality

🌿 Generated by Terry

---

ℹ️ Tag @terragon-labs to ask questions and address PR feedback

📎 Task: https://www.terragonlabs.com/task/c20ebd4e-8179-41c6-818c-3cbaa6ab4976

Summary by CodeRabbit

• Bug Fixes
  • Chat responses now reliably reflect the selected AI provider by isolating cached results per provider.
  • Prevents cross-provider mix-ups and stale answers after switching providers, improving conversation accuracy.
  • Reduces unexpected tone/format differences caused by reused responses from a different provider.
  • Delivers more consistent behavior and trustworthiness without changing existing workflows.

[coderabbitai]

Walkthrough

The chat OpenAPI caching logic in apps/gateway/src/chat/chat.ts now includes the provider (usedProvider) in the cache payload alongside model, messages, and other fields, altering the cache key composition to distinguish cached responses by provider.

Changes

Cohort / File(s)	Summary of Changes
Chat caching key update apps/gateway/src/chat/chat.ts	Expanded cachePayload to include provider: usedProvider so cache entries are partitioned by provider in addition to model/messages and existing fields.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant C as Client
  participant G as Gateway Chat
  participant K as Cache
  participant P as LLM Provider

  Note over G: Build cachePayload now includes provider
  C->>G: Chat request (messages, model, provider)
  G->>K: Get(cache key: model + messages + provider + ...)
  alt Cache hit
    K-->>G: Cached response
    G-->>C: Return cached response
  else Cache miss
    G->>P: Forward request (provider-specific)
    P-->>G: Response/stream
    G->>K: Set(cache key incl. provider, response)
    G-->>C: Return response
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• feat(cache): add streaming cache support for chat responses #529 — Also updates apps/gateway/src/chat/chat.ts caching to include provider and touches streaming cache behavior, indicating closely related caching logic changes.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly describes the core change—adding the provider field to the cache key for uniqueness—which directly matches the code update in the pull request and uses clear, specific language without extra noise.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch terragon/fix-cache-key-uniqueness

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 60fbedf and e931ec7.

📒 Files selected for processing (1)

• apps/gateway/src/chat/chat.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx}: Never use any or as any in this TypeScript project unless absolutely necessary
Always use top-level import; never use require or dynamic import()

Files:

• apps/gateway/src/chat/chat.ts

apps/{api,gateway}/**/*.ts

📄 CodeRabbit inference engine (CLAUDE.md)

For database reads, use Drizzle query helpers: db().query.

.findMany() or findFirst()

Files:

• apps/gateway/src/chat/chat.ts

{apps,packages}/**/src/**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

Always use top-level import; never use require() or dynamic import()

Files:

• apps/gateway/src/chat/chat.ts

apps/{api,gateway}/src/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

apps/{api,gateway}/src/**/*.{ts,tsx}: For reads, use db().query.

.findMany() or db().query..findFirst()
Run pnpm generate after API route changes to update OpenAPI schemas

Files:

• apps/gateway/src/chat/chat.ts

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)

• GitHub Check: e2e-shards (4)
• GitHub Check: e2e-shards (5)
• GitHub Check: e2e-shards (1)
• GitHub Check: e2e-shards (2)
• GitHub Check: e2e-shards (3)
• GitHub Check: autofix
• GitHub Check: generate / run
• GitHub Check: test / run
• GitHub Check: lint / run
• GitHub Check: build / run

🔇 Additional comments (1)

apps/gateway/src/chat/chat.ts (1)

1329-1338: Fix ensures cache isolation per provider

Including provider: usedProvider in the cache payload closes the cross-provider collision gap; cache lookups will now respect provider-specific responses. Nicely scoped fix.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment