fix(panw_prisma_airs): honor experimental_use_latest_role_message_only on every request shape

[devin-ai-integration]

TLDR

Problem this solves:

• experimental_use_latest_role_message_only: true on the PANW Prisma AIRS guardrail only took effect on Anthropic /v1/messages; on /v1/chat/completions and /v1/responses every turn was still sent to AIRS
• On /v1/responses, instructions, function_call_output and reasoning items break the alignment between the flattened texts and the normalized messages, so a naive fix falls back to scanning everything, or worse, scans the model's reasoning text instead of the user's prompt

How it solves it:

• Explicit true or false now wins on every endpoint, same semantics as the bedrock guardrail flag; unset keeps today's behavior (latest-only on Anthropic /v1/messages, full role-filter scan elsewhere)
• Latest-turn selection works off the normalized structured_messages and maps each flattened text back to the message it came from, skipping Responses-only entries; reasoning items are identified from the raw input so model-authored text is never taken for the latest human turn
• Anything that cannot be aligned falls back to the existing broader scan, so the failure direction is more scanning, never less

Supersedes #38977 by @scthornton (co-authored on the first commit), whose base branch was deleted before it could merge

User Flow

Before: an admin enables latest-only scanning to cut AIRS spend on long agent conversations, but only Anthropic traffic benefits

1. Set experimental_use_latest_role_message_only: true on the panw_prisma_airs guardrail and restart the proxy
2. POST /v1/chat/completions or /v1/responses with an 8-turn history: AIRS receives one scan per user and system text, 36 to 44 scans over the conversation
3. A request whose only flagged text sits in an old turn comes back 400 Prompt blocked by PANW Prisma AI Security policy, even though the latest turn is clean

After: the same setting applies to every endpoint

1. Same config
2. Same requests: AIRS receives exactly one scan per request, the latest user turn, 8 scans over the conversation, including when instructions, function_call_output or reasoning items are present
3. Flagged text only in history now comes back 200 with the model reply; a flagged latest turn still comes back 400

Relevant issues

Supersedes #38977

Affected release

Linear ticket

Resolves LIT-7388

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Against the live Prisma AIRS API (by @scthornton, on this tip 8be0af1)

Local proxy on the PR head, guardrail api_base pointed at a recording forwarder relaying every scan to the live AIRS Scan API, Gemini upstream. Same 8-turn conversation replayed with the flag unset and then true. All 96 scans were live AIRS calls, HTTP 200 with real scan_ids. Details in #42447 (comment)

Shape	Flag unset	Flag true
/v1/chat/completions	44 scans, 3,126 tokens	8 scans, 419 tokens
/v1/responses, instructions + history	36 scans, 1,950 tokens	8 scans, 419 tokens
/v1/responses, function_call + function_call_output in history	36 scans, 1,950 tokens	8 scans, 419 tokens

With the flag on, every call scanned exactly the current user turn; no assistant or tool-output text was scanned in any run

Base vs head on a local rig (this PR)

Two proxies, merge base b682278 and this tip 8be0af1, each proxy_cli.py --num_workers 4, same config, Postgres and Redis. Real OpenAI (openai/gpt-5.4-mini), Anthropic (anthropic/claude-haiku-4-5) and OpenRouter (openrouter/openai/gpt-oss-120b, the one model here that returns reasoning items with text content) upstreams. The scanner is a local stand-in speaking the AIRS /v1/scan/sync/request wire format that logs every scan; it blocks any text containing RIGBLOCKME and masks EMAIL@RIG.TEST under the mask profile. The scan log is the evidence for which texts PANW was asked to inspect

guardrails:
  - guardrail_name: panw-true
    litellm_params:
      guardrail: panw_prisma_airs
      mode: [pre_call, post_call]
      default_on: false
      api_key: os.environ/PANW_PRISMA_AIRS_API_KEY
      api_base: os.environ/PANW_PRISMA_AIRS_API_BASE
      profile_name: rig-true
      experimental_use_latest_role_message_only: true
  - guardrail_name: panw-unset   # same, flag omitted
  - guardrail_name: panw-false   # same, experimental_use_latest_role_message_only: false

History in every request: system: You are terse., user: first turn: remember the word pineapple, assistant: Noted., then user: latest turn: which word did I ask you to remember? answer with one word. Example request:

curl -s http://127.0.0.1:16402/v1/responses -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"gpt","guardrails":["panw-true"],"instructions":"answer briefly",
       "input":[{"role":"system","content":"You are terse."},
                {"role":"user","content":"first turn: remember the word pineapple"},
                {"role":"assistant","content":"Noted."},
                {"role":"user","content":"latest turn: which word did I ask you to remember? answer with one word"}]}'

Request-side texts sent to the scanner, flag true:

Shape	Before (b682278)	After (8be0af1)
/v1/chat/completions	system, first turn, latest turn	latest turn
/v1/responses, plain	system, first turn, latest turn	latest turn
/v1/responses + instructions	system, first turn, Noted., latest turn	latest turn
/v1/responses + function_call_output	system, first turn, Noted., latest turn	latest turn
/v1/responses + reasoning item (summary only, replayed from a real gpt-5.4-mini response)	first turn, latest turn	latest turn
/v1/responses tool loop, reasoning item with text content after the latest turn (real gpt-oss-120b output)	first turn, Noted., latest turn, reasoning text	latest turn
/v1/messages	latest turn	latest turn

Outcomes, flag true, /v1/responses with instructions:

Request	Before	After
RIGBLOCKME only in the first user turn, clean latest turn	400 Prompt blocked by PANW Prisma AI Security policy (Category: malicious)	200 "output_text": "Hi", scan log shows only the latest turn
RIGBLOCKME in the latest turn	400 blocked	400 blocked
EMAIL@RIG.TEST in the latest turn, mask profile	200, email masked, every turn scanned	200 "output_text": "my email is [MASKED_EMAIL]", only the email turn scanned
Latest turn is an input_image only, RIGBLOCKME in the first turn	400 blocked	200, no request scan, response scan runs, proxy logs PANW Prisma AIRS: latest user message has no text, so experimental_use_latest_role_message_only leaves nothing to scan for call_id=...
Tool loop with a text reasoning item after a RIGBLOCKME latest turn	400 blocked	400 blocked (the previous tip e8b444f scanned only the reasoning text and returned 200, the Bugbot finding fixed in 8be0af1)

Flag unset and flag false: scans identical to Before on every endpoint and shape, with one exception, /v1/responses with instructions or function_call_output no longer scans the assistant turn (Noted.), see REVIEWER MUST KNOW. Anthropic unset stays latest-only, Anthropic false stays full scan. On the final re-run the Anthropic upstream answered 400 Your credit balance is too low; the pre-call scan had already run with the same scope, and the 819a9e1 run completed the same cells end to end

Regression coverage: tests/integration/observability/test_guardrail_effects.py::test_panw_latest_role_message_only_scans_only_latest_turn_on_responses_input boots a live proxy with a scripted upstream and recording scanner and fails on the merge base with plain: latest-only scanned ['first turn', 'latest turn ...']. tests/test_litellm/proxy/guardrails/guardrail_hooks/test_panw_prisma_airs.py (226 tests) covers plain, instructions, function_call_output, summary-only and text-content reasoning, tool loops, developer role, image-only latest turn, duplicate text collisions, unalignable input, masking, and explicit true, false and unset. Eleven mutations of the selection logic (ignore the flag, pick the first human turn, drop the alignment or count guards, stop excluding reasoning items, and so on) each turn at least one test red

Full audit at 819a9e1 vs b682278: 138 cells on both proxies (three endpoints, unset/true/false, streaming and non-streaming, curl plus openai and anthropic SDKs sync and async, edge shapes, blocked and masked outcomes, bad model, bad key, scanner 500 and slow scanner with fallback_on_error block and allow, during_call and logging_only modes, 30-request burst with scanner outage, worker kill -9). Every flag cell reproduces the tables above; unset and false are identical on both legs in every cell. Two findings identical on both legs and untouched here: a chat content of 5 returns 500 from the managed files hook, and a logging_only guardrail with default_on: false runs for every request. Dashboard Playground before/after recording: https://berriaillm.slack.com/archives/D0BEC4CBJKT/p1790067473347679

Not in this PR: on /v1/responses the instructions field is never scanned, flag on or off (that is why the Responses baseline above is 36 scans, not 44). It lives in the Responses text extraction, predates this change, and @scthornton offered to open a separate issue

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Latest-only trusts caller-supplied history: earlier turns are not rescanned, and a latest turn with no text (image only) sends nothing to the request scan, so that request has no request-side scan_id. The config description says so and the proxy logs a debug line with the call id

Low

• A function_call_output whose text equals the latest user text cannot be attributed unambiguously; the request falls back to scanning every extracted text, the same as Before, which still includes the latest turn
• The local rig used a stand-in scanner speaking the AIRS wire format; the live AIRS run above is @scthornton's

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and found no regressions/backward incompatible risks

REVIEWER MUST KNOW BEFORE APPROVING

1. Explicit experimental_use_latest_role_message_only: true on /v1/chat/completions and /v1/responses now sends only the latest user or developer turn to AIRS. Before: every user, system and developer text was scanned and a flagged old turn returned 400. After: one scan; flagged history only returns 200, a flagged latest turn still returns 400, and an image-only latest turn sends no request scan. This is the change LIT-7388 asks for, approved by @yucheng-berri in the Devin session that took fix(panw_prisma_airs): apply experimental_use_latest_role_message_only to every request shape #38977 over into this PR
2. Flag unset or false on /v1/responses with instructions or a function_call_output item: the assistant turn is no longer scanned. Before: system, user and assistant texts were all scanned because the role filter could not align the texts and scanned everything. After: system, user and developer texts only, the documented role filter that chat completions already applied. Not requested by the ticket, fail direction is fewer scans of model-authored text; surfaced here for @yucheng-berri to accept at approval

Link to Devin session: https://app.devin.ai/sessions/9bc67ae5d93b4e59959263bdd9efedbf
Open in Devin Desktop: https://app.devin.ai/desktop/session/9bc67ae5d93b4e59959263bdd9efedbf?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[mateo-berri]

bugbot run

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule findings

Summary

This PR makes PANW Prisma AIRS honor explicit latest-role-only configuration across chat completions, Anthropic Messages, and Responses requests

• Maps flattened scan text back to normalized messages while conservatively falling back when alignment is uncertain
• Excludes Responses reasoning items from human-turn selection and preserves image-only latest turns as zero-text scans
• Expands configuration guidance and regression coverage for Responses request shapes, reasoning, tools, masking, and flag semantics

_{Reviews (7) · Last reviewed commit: "fix(panw_prisma_airs): keep Responses re..."}

[cursor]

Stale Bugbot comment from a previous run.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_panw_latest_role_message_all_shapes (6788d0a) with main (6d4ccf7)¹}

Footnotes

1. No successful run was found on main (b9ba36c) during the generation of this report, so 6d4ccf7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[devin-ai-integration]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 98.75000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ardrail_hooks/panw_prisma_airs/panw_prisma_airs.py	98.75%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[scthornton]

The bidirectional walk in _text_source_message_indices is a nice solve. Requiring forward and reverse to agree kills the attribution ambiguity, and it generalizes past the three Responses item types instead of special-casing them.

One edge worth a look before this goes in.

When the flag is on and the latest user turn carries no text (image-only content), _get_latest_user_text_indices returns an empty frozenset. That's intentional per the docstring and I agree it beats promoting an earlier turn. But downstream, empty isn't distinguishable from "scan these zero indices": scannable_indices is None is False so there's no fallback to the role filter, and the loop at L1801 skips every text. Net effect is the request produces zero scan calls, no scan_id, nothing in the AIRS activity log.

The warning at L1786 doesn't cover it, since it's gated on effective_scan_only_tool_results_for_guardrail(self) and supports_scan_only_tool_results() returns False for this guardrail.

I don't think the scan decision is wrong. Latest-only means earlier turns aren't rescanned, and there's nothing in the latest turn to scan. It's the silence I'd flag. When I ran the 8-turn before/after through a local proxy against the live AIRS endpoint, what made the behavior legible was every turn producing a scan_id I could reconcile in the activity log. A turn that produces none looks identical to a guardrail that isn't firing, which is painful to debug from the operator side. A debug line when an image-only latest turn yields an empty set would close that gap.

Smaller and separate: _message_texts and the latest_human scan now call .get() on entries directly, where the prior versions guarded with isinstance(msg, dict). AllMessageValues is a TypedDict so these are dicts by contract, but if anything upstream ever hands this a pydantic message object it would now raise inside the guardrail rather than falling back to the broader scan. Is that path definitely normalized before it gets here?

Happy to re-run the live before/after against the real AIRS endpoint on this branch if that's useful. I still have the 8-turn harness and the e2e artifacts from the original measurement.

[devin-ai-integration]

@scthornton good catch on the silent zero-scan: 03cff16 adds a debug line with the call_id. Messages reach the guardrail as dicts, the handler already calls .get() upstream

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai please re-review the tip 03cff16, which adds a debug log line and the matching regression assertion

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai review 25a6891 is a merge of current main with no PANW changes, re-review for a fresh verdict on the tip please

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

Hi @scthornton , thanks for the review. Could you please rerun the live e2e tests against real AIRS endpoint to help validate? Thanks ahead

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai please review 4b8cebc, a merge of current main that resolves a test file conflict and changes nothing in this diff

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai please review e8b444f, a merge of current main that drops the legacy covers marker after main removed contracts.json

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8be0af1. Configure here.}

[scthornton]

Reran the live e2e against 8be0af1. Local proxy on the PR head, guardrail api_base pointed at a recording forwarder that relays every scan to the live AIRS Scan API, Gemini behind it. Same 8-turn conversation as before, replayed with the flag unset and then true, across three request shapes:

Shape	Flag unset	Flag true
/v1/chat/completions	44 scans, 3,126 tokens	8 scans, 419 tokens (7.46x)
/v1/responses, instructions + history	36 scans, 1,950 tokens	8 scans, 419 tokens (4.65x)
/v1/responses, function_call + function_call_output in history	36 scans, 1,950 tokens	8 scans, 419 tokens (4.65x)

All 96 scans were live AIRS calls, HTTP 200 with real scan_ids. With the flag on, every call scanned exactly the current user turn, and no assistant or tool-output text was scanned in any run. Chat numbers match the original validation on #38977. test_panw_prisma_airs.py passes locally on the same head (226).

One thing I noticed that's separate from this PR: on /v1/responses the instructions field isn't scanned at all, even with the flag unset, which is why the Responses baseline is 36 scans instead of 44. That looks like the Responses text extraction rather than anything here, so I'd treat it as its own issue. Happy to open one.

LGTM from my side.

[yucheng-berri]

@scthornton Thanks for the validation. We will keep track of the findings for instruction.