[#20367][fix] Accept prompt_cache_key in Chat Completions requests by lucifer1004 · Pull Request #20368 · NVIDIA/TensorRT-LLM

lucifer1004 · 2026-10-15T04:28:38Z

Description

Fixes #20367.

ChatCompletionRequest is extra="forbid", so a request carrying OpenAI's standard prompt_cache_key field was rejected with HTTP 400 (extra_forbidden). This PR adds the field as Optional[str] = None, accepted and otherwise ignored, like user and parallel_tool_calls: KV cache reuse needs no client hint, and cache_salt remains the field that isolates cached prefixes. ResponsesRequest already accepts the field through extra="allow".

Other unknown fields are still rejected, and non-string values of prompt_cache_key fail validation.

tensorrt_llm/serve/openai_protocol.py: new field on ChatCompletionRequest.
tests/unittest/api_stability/references/trtllm_serve_api.yaml: the matching reference row (kind: openai, status: stable).

Test Coverage

New tests/unittest/llmapi/apps/test_openai_protocol_prompt_cache_key.py (CPU only), registered in l0_cpu.yml: accepted as keyword and from JSON, default None, non-string values rejected, other fields unchanged by it, unknown fields still rejected.
tests/unittest/api_stability/test_serve_api.py covers the reference row.
An equivalent change (with a stricter variant of these tests) passed our CPU test run of the serve unit tests; this branch's new test file has not been run yet (no local environment with the package built); I will report the result here.
pre-commit on the changed files: clean.

PR Checklist

PR description clearly explains what and why.
PR follows the TRT-LLM coding guidelines to the best of my knowledge.
Test cases are provided for new code paths.
API change: one new optional OpenAI-compatible request field (backward compatible); the serve API reference YAML is updated.
No new dependencies, no CODEOWNERS changes.
Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

ChatCompletionRequest adds optional prompt_cache_key: str, defaulting to None, so chat-completion requests can include this OpenAI field without changing other request data. The field is accepted for compatibility and ignored; other unknown fields remain rejected. The stable API reference records the field as optional.

QA Engineer Review

The new CPU-only test file covers string values, including an empty string; the None default; JSON input; invalid types; unchanged fields; and rejection of unknown fields. The test is listed in tests/integration/test_lists/test-db/l0_cpu.yml. CI comments report that the new test passed, and a later pipeline completed successfully. The tests cover model validation but not endpoint-level serving behavior. Coverage: needs follow-up for endpoint-level behavior.

Per-File QA Perspective

tensorrt_llm/serve/openai_protocol.py: Verify chat-completion requests accept and ignore prompt_cache_key, default it to None, and continue to reject invalid types and other unknown fields.
tests/unittest/llmapi/apps/test_openai_protocol_prompt_cache_key.py: Covers accepted values, defaults, JSON input, invalid types, unchanged fields, and unknown-field rejection. Listed in the CPU CI test list.
tests/integration/test_lists/test-db/l0_cpu.yml: Adds the new CPU-only test to the CI list.
tests/unittest/api_stability/references/trtllm_serve_api.yaml: Adds prompt_cache_key as a stable optional API field. Verify that the reference stays consistent with the request model.

ChatCompletionRequest forbids unknown fields, so requests carrying OpenAI's standard prompt_cache_key were rejected with 400. Accept the field and ignore it, like user and parallel_tool_calls. Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

coderabbitai · 2026-10-15T04:33:34Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Last reviewed commit: fb68c1e
Latest commit, not reviewed: 7dc1606

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: NVIDIA/TensorRT-LLM
Branch: main
File: .coderabbit.yaml
Commit: 315f04f
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: 28dcb930-dce9-4105-b429-69477d2cfd07

Commits

Reviewing files that changed from the base of the PR and between d33a514 and fb68c1e.

Files selected for processing: 2

tensorrt_llm/serve/openai_protocol.py
tests/integration/test_lists/test-db/l0_cpu.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

ChatCompletionRequest accepts an optional prompt_cache_key string and ignores it. The change adds API stability metadata and CPU tests for parsing, defaults, validation, and preservation of other fields.

Changes

Chat completion request compatibility

Layer / File(s)	Summary
Add and validate the request field `tensorrt_llm/serve/openai_protocol.py`, `tests/unittest/api_stability/references/trtllm_serve_api.yaml`, `tests/unittest/llmapi/apps/test_openai_protocol_prompt_cache_key.py`, `tests/integration/test_lists/test-db/l0_cpu.yml`	`ChatCompletionRequest` accepts the optional `prompt_cache_key` field and ignores it. The API stability reference records the field as optional and stable. CPU tests cover valid strings, the default value, JSON parsing, unchanged other fields, and rejection of unknown fields and non-string values. The CPU test list includes the new test module.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

[Bug]: trtllm-serve /v1/chat/completions rejects the standard OpenAI field safety_identifier with 400 #20962: Reports the same ChatCompletionRequest extra-field rejection for another standard OpenAI field and identifies prompt_cache_key support as a counterpart change.

Possibly related PRs

NVIDIA/TensorRT-LLM#20864: Adds an optional ignored store field to ChatCompletionRequest using the same schema, stability-reference, and test pattern.
NVIDIA/TensorRT-LLM#20963: Adds an optional ignored safety_identifier field to ChatCompletionRequest using the same schema, stability-reference, and validation-test pattern.

Merge Risk: Minimal · up to fb68c

No actionable merge-blocking risk remains for this compatibility change.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the issue, uses the required [`#20367`][fix] format, and accurately summarizes the primary change.
Description check		The description explains the problem and solution, identifies the API change, documents relevant test coverage, and includes the repository checklist. It is sufficiently complete for review.
Linked Issues check		Issue `#20367` is active and directly linked. `ChatCompletionRequest` adds `prompt_cache_key: Optional[str] = None` and retains `extra="forbid"`. The change accepts and ignores the OpenAI field while pr…
Out of Scope Changes check		The source change directly addresses issue `#20367`. The tests verify the required compatibility and validation behavior. The CPU test registration and stable API reference entry support the same public…

Full details: Linked Issues check

Explanation

Issue #20367 is active and directly linked. ChatCompletionRequest adds prompt_cache_key: Optional[str] = None and retains extra="forbid". The change accepts and ignores the OpenAI field while preserving rejection of other unknown fields. The added tests cover keyword and JSON input, the default, non-string rejection, unchanged fields, and unknown-field rejection. The API reference entry and CPU test registration are included.

Full details: Out of Scope Changes check

Explanation

The source change directly addresses issue #20367. The tests verify the required compatibility and validation behavior. The CPU test registration and stable API reference entry support the same public API change. The supplied whole-PR summary shows no unrelated changes.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

lucifer1004 · 2026-10-15T09:52:06Z

/bot run

tensorrt-cicd · 2026-10-15T10:00:38Z

PR_Github #78107 [ run ] triggered by Bot. Commit: 5569215 Link to invocation

tensorrt-cicd · 2026-10-15T10:15:26Z

PR_Github #78107 [ run ] completed with state FAILURE. Commit: 5569215
/LLM/main/L0_MergeRequest_PR pipeline #64378 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-noncommitted-api-review-committee

lucifer1004 · 2026-10-16T00:12:51Z

/bot run

tensorrt-cicd · 2026-10-16T00:19:11Z

PR_Github #78261 [ run ] triggered by Bot. Commit: 5569215 Link to invocation

tensorrt-cicd · 2026-10-16T03:30:08Z

PR_Github #78261 [ run ] completed with state FAILURE. Commit: 5569215
/LLM/main/L0_MergeRequest_PR pipeline #64511 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

lucifer1004 · 2026-10-16T04:34:20Z

The last CI run failed only on tests that this PR does not touch. unittest/_torch/compilation/test_phase_selective_forward.py monkeypatches moe_a2a_steady_state_budget_for_capture, which does not exist at the main commit this branch was based on; #20362 has since fixed the test on main. The other failures were fail-fast aborts. I merged current main into the branch and am re-running CI.

lucifer1004 · 2026-10-16T04:34:21Z

/bot run

tensorrt-cicd · 2026-10-16T04:41:40Z

PR_Github #78355 [ run ] triggered by Bot. Commit: da36bed Link to invocation

tensorrt-cicd · 2026-10-16T08:06:51Z

PR_Github #78355 [ run ] completed with state FAILURE. Commit: da36bed
/LLM/main/L0_MergeRequest_PR pipeline #64585 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lucifer1004 · 2026-10-16T08:29:48Z

CI #78355 failed outside this PR. The CPU failure is test_trtllm_serve_config_schemas.py::test_checked_in_schemas_are_current: main's checked-in serve schemas do not include the kv_pool_rebalance_config fields added in #19077 (details there), so every PR based on current main hits it. This PR's own test passed. The B200 / B300 attention suites failed with a sticky CUDA driver error: 201 (invalid device context) from the KV cache manager in sparse / attention-backend tests, and the H100 failures are in test_e2e_capture.py and the cache-transceiver tests. None of these touch openai_protocol.py. I'll re-run once the schemas are refreshed on main; a re-run before that would fail on the same test.

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

lucifer1004 · 2026-10-19T13:30:15Z

CI #78860 failed outside this PR:

CPU-Generic-arm-1: five unittest/llmapi/test_mpi_session.py::test_remote_mpi_session[*] cases timed out (exit status 124). The same cases have failed this way on arm in other recent PR pipelines (e.g. [#20373][perf] DFlash TRTLLM: run non-causal unwindowed block attention as decode rows up to batch 32 #20374's last run); this PR does not touch the MPI session code.
DGX_B200-PyTorch-9: unittest/_torch/thop/parallel/test_cute_dsl_moe.py::test_nvfp4_gather_grouped_gemm_act_fusion_blackwell[128-1-32-256-relu2] failed with 5.4% of its elements outside the atol/rtol 1e-4 tolerance, over the 5% limit (see [Bug] test_nvfp4_gather_grouped_gemm_act_fusion_blackwell flaky on B300: ~0.05% mismatch vs rtol 1e-4, varying parametrizations #16281 (comment)), before fail-fast aborted the stage. It is a CuTe DSL MoE kernel test, unrelated to the OpenAI protocol; it passed in the other recent runs.
This PR's test_openai_protocol_prompt_cache_key.py passed on both CPU-Generic-x86-1 and CPU-Generic-arm-1.

Rerunning.

lucifer1004 · 2026-10-19T13:30:17Z

/bot run

tensorrt-cicd · 2026-10-19T13:36:43Z

PR_Github #78926 [ run ] triggered by Bot. Commit: d33a514 Link to invocation

github-actions · 2026-10-19T13:38:23Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

tensorrt-cicd · 2026-10-19T16:19:30Z

PR_Github #78926 [ run ] completed with state SUCCESS. Commit: d33a514
/LLM/main/L0_MergeRequest_PR pipeline #65025 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

lucifer1004 · 2026-10-25T11:39:51Z

/bot run

tensorrt-cicd · 2026-10-25T11:47:40Z

PR_Github #80461 [ run ] triggered by Bot. Commit: fb68c1e Link to invocation

tensorrt-cicd · 2026-10-25T14:10:58Z

PR_Github #80461 [ run ] completed with state SUCCESS. Commit: fb68c1e
/LLM/main/L0_MergeRequest_PR pipeline #66232 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lucifer1004 · 2026-10-25T15:29:01Z

/bot run

tensorrt-cicd · 2026-10-25T15:36:19Z

PR_Github #80474 [ run ] triggered by Bot. Commit: fb68c1e Link to invocation

tensorrt-cicd · 2026-10-25T20:27:49Z

PR_Github #80474 [ run ] completed with state SUCCESS. Commit: fb68c1e
/LLM/main/L0_MergeRequest_PR pipeline #66243 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

lucifer1004 · 2026-10-26T06:01:10Z

/bot run

tensorrt-cicd · 2026-10-26T06:08:46Z

PR_Github #80549 [ run ] triggered by Bot. Commit: 7dc1606 Link to invocation

tensorrt-cicd · 2026-10-26T10:18:16Z

PR_Github #80549 [ run ] completed with state FAILURE. Commit: 7dc1606
/LLM/main/L0_MergeRequest_PR pipeline #66313 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lucifer1004 · 2026-10-26T12:05:12Z

CI #80549 failed for reasons unrelated to this PR (it only adds prompt_cache_key to ChatCompletionRequest):

CPU-Generic-x86-1: unittest/_torch/modeling_v2/test_modeling_v2_deepseek_default_shards.py::test_load_reads_only_the_rank_share[ws8-tp-moetp4-moeep2] reported "Test terminated unexpectedly" and passed on rerun in the same pipeline. It also passes in the other recent pipelines.
DGX_H100-PyTorch-6: several unittest/disaggregated/test_kv_transfer.py cases hit CUDA driver error: 201 (invalid device context) in KvCacheManager teardown and passed on rerun.
B300-PyTorch-2 and CPU-Generic-arm-1 were aborted by fail-fast.

This PR's test_openai_protocol_prompt_cache_key.py did not fail.

lucifer1004 · 2026-10-26T12:05:16Z

/bot run

tensorrt-cicd · 2026-10-26T12:13:09Z

PR_Github #80607 [ run ] triggered by Bot. Commit: 7dc1606 Link to invocation

tensorrt-cicd · 2026-10-26T15:27:24Z

PR_Github #80607 [ run ] completed with state SUCCESS. Commit: 7dc1606
/LLM/main/L0_MergeRequest_PR pipeline #66365 completed with status: 'SUCCESS'

CI Report

Link to invocation

lucifer1004 requested review from a team as code owners October 15, 2026 04:28

lucifer1004 requested review from BowenFu, YihuiLu512, brnguyen2, dpitman-nvda, hanjingtian and zhaoyuanh-nvidia October 15, 2026 04:28

github-actions Bot assigned lucifer1004 Oct 15, 2026

lucifer1004 added the api-compatible Accepted LLM API contract change that is backwards-compatible label Oct 15, 2026

BowenFu approved these changes Oct 15, 2026

View reviewed changes

mikeiovine approved these changes Oct 15, 2026

View reviewed changes

Merge main into upstream/chat-prompt-cache-key

da36bed

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

chenfeiz0326 mentioned this pull request Oct 16, 2026

[https://nvbugs/6843484][fix] Keep multi-node aggregated perf-sanity jobs in one NVLink domain #20418

Merged

1 task done

Merge main into upstream/chat-prompt-cache-key

f8e0fbd

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

brnguyen2 approved these changes Oct 19, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 19, 2026

lucifer1004 mentioned this pull request Oct 19, 2026

[#20365][fix] Use the fused DSpark RMSNorm/RoPE kernels on SM107 #20366

Merged

5 tasks done

This was referenced Oct 19, 2026

[#20507][fix] Number Sm107a as 5 to match the prebuilt trtllm-gen archives #20518

Open

[Bug] test_nvfp4_gather_grouped_gemm_act_fusion_blackwell flaky on B300: ~0.05% mismatch vs rtol 1e-4, varying parametrizations #16281

Open

JunyiXu-nv mentioned this pull request Oct 23, 2026

[None][fix] Chat Completions: accept the OpenAI store field #20864

Merged

1 task done

This was referenced Oct 25, 2026

[Bug]: trtllm-serve /v1/chat/completions rejects the standard OpenAI field safety_identifier with 400 #20962

Open

[#20962][fix] Accept the OpenAI safety_identifier field in Chat Completions requests #20963

Open

Merge main into upstream/chat-prompt-cache-key

fb68c1e

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

Merge main into upstream/chat-prompt-cache-key

7dc1606

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

coderabbitai Bot mentioned this pull request Oct 28, 2026

[None][fix] Chat Completions: accept service_tier and client_metadata #21154

Merged

1 task done

JunyiXu-nv mentioned this pull request Nov 3, 2026

[TRTLLM-17265][refactor] Split openai_protocol into a package #21544

Draft

1 task done

Conversation

lucifer1004 commented Oct 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

lucifer1004 commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

mikeiovine left a comment

Choose a reason for hiding this comment

Uh oh!

lucifer1004 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

lucifer1004 commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucifer1004 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

lucifer1004 commented Oct 16, 2026

Uh oh!

lucifer1004 commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucifer1004 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

lucifer1004 commented Oct 25, 2026

Uh oh!

tensorrt-cicd commented Oct 25, 2026

Uh oh!

tensorrt-cicd commented Oct 25, 2026

Uh oh!

lucifer1004 commented Oct 25, 2026

Uh oh!

tensorrt-cicd commented Oct 25, 2026

Uh oh!

tensorrt-cicd commented Oct 25, 2026

Uh oh!

lucifer1004 commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

lucifer1004 commented Oct 26, 2026

Uh oh!

lucifer1004 commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

This branch has not been deployed

lucifer1004 commented Oct 15, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

lucifer1004 commented Oct 16, 2026 •

edited

Loading

lucifer1004 commented Oct 19, 2026 •

edited

Loading