Skip to content

fix noma v2 deepcopy crashing in build scan payload - new PR - #26605

Merged
krrish-berri-2 merged 21 commits into
BerriAI:litellm_internal_stagingfrom
Noma-Security:omri/noma-v2-deepcopy
Apr 27, 2026
Merged

fix noma v2 deepcopy crashing in build scan payload - new PR#26605
krrish-berri-2 merged 21 commits into
BerriAI:litellm_internal_stagingfrom
Noma-Security:omri/noma-v2-deepcopy

Conversation

@omriShukrun08

Copy link
Copy Markdown
Contributor

Relevant issues

Noma v2 integration experiences a 500 error during post_call/during_call/during_mcp_call due to deepcopy(request_data) failing on unserializable objects (uvloop.Loop) in _build_scan_payload. A JSON-safe copy method is needed, similar to the approach in spend_tracking_utils.py and its _sanitize_payload_for_transport() helper, which currently runs after the deepcopy crash.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

Type

🐛 Bug Fix

Changes

idola9 and others added 21 commits November 19, 2025 16:59
…noma-app-id

Feature/litellm key name to noma app
…noma-app-id

Add tests to Use LiteLLM key alias as fallback Noma applicationId in NomaGuardrail
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro

Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:

- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
  input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
  per 1M input/output/cached input

Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.

No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.

Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields

* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants

gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.

Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.

Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
…s) (BerriAI#26361)

* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)

Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.

Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
  $60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
  but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro

reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once BerriAI#26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.

Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.

* test: register supports_low_reasoning_effort in cost-map JSON schema

azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.

Note: the runtime side of this flag (code that reads it) lands in
BerriAI#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
@CLAassistant

CLAassistant commented Apr 27, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 5 committers have signed the CLA.

✅ yuneng-berri
✅ mateo-berri
✅ TomAlon
✅ omriShukrun08
❌ idola9
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a 500 error in the Noma v2 guardrail by replacing deepcopy(request_data) with _sanitize_payload_for_transport(request_data) in _build_scan_payload, avoiding crashes when request_data contains C-extension objects (e.g. uvloop.Loop) whose __reduce__ raises on pickling. A regression test is included that verifies both the non-crash behavior and the non-mutation of the original dict.

Confidence Score: 4/5

Safe to merge; fix is correct and well-tested with only minor P2 concerns remaining.

The change is minimal and targeted, directly addresses the root cause (deepcopy crashing on unserializable objects), and does not affect the data that ultimately reaches Noma since _call_noma_scan always sanitizes the full payload. The regression test is solid. Two P2 issues exist: a missing RecursionError catch and a documentation gap around the double-sanitization pattern.

litellm/proxy/guardrails/guardrail_hooks/noma/noma_v2.py — specifically the exception handler in _sanitize_payload_for_transport

Important Files Changed

Filename Overview
litellm/proxy/guardrails/guardrail_hooks/noma/noma_v2.py Replaces deepcopy(request_data) with _sanitize_payload_for_transport(request_data) in _build_scan_payload to avoid crashes on C-extension objects like uvloop.Loop; two minor P2 issues noted (missing RecursionError in exception handler, post-sanitization injection of raw model_call_details).
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_noma_v2.py Adds a regression test for the deepcopy crash using a fake unpicklable object; no real network calls, correctly asserts non-mutation of the original dict and proper string fallback for unserializable values.

Sequence Diagram

sequenceDiagram
    participant Caller as apply_guardrail
    participant BSP as _build_scan_payload
    participant SPFT as _sanitize_payload_for_transport
    participant CNS as _call_noma_scan
    participant Noma as Noma API

    Caller->>BSP: request_data (may contain uvloop.Loop)
    BSP->>SPFT: request_data
    Note over SPFT: json.dumps with default=str fallback<br/>→ JSON round-trip → safe dict
    SPFT-->>BSP: payload_request_data (JSON-safe copy)
    BSP->>BSP: inject litellm_logging_obj (raw model_call_details)
    BSP-->>Caller: payload dict
    Caller->>CNS: payload
    CNS->>SPFT: payload (2nd sanitization pass)
    SPFT-->>CNS: sanitized_payload (litellm_logging_obj now safe)
    CNS->>Noma: POST /litellm/guardrail
    Noma-->>CNS: response JSON
    CNS-->>Caller: response_json
Loading

Comments Outside Diff (2)

  1. litellm/proxy/guardrails/guardrail_hooks/noma/noma_v2.py, line 167-170 (link)

    P2 RecursionError not caught in serialization fallback

    json.dumps will raise RecursionError (not ValueError/TypeError) on deeply-nested or self-referential structures in request_data. The current except (ValueError, TypeError) clause won't catch it, and the exception will propagate to the caller — the same 500 path this PR fixes.

  2. litellm/proxy/guardrails/guardrail_hooks/noma/noma_v2.py, line 141-145 (link)

    P2 model_call_details bypasses first sanitization pass

    _sanitize_payload_for_transport produces a clean JSON-safe dict at line 141, but model_call_details (a complex object) is injected back onto line 144 after the fact. The value is only sanitized later when _call_noma_scan calls _sanitize_payload_for_transport a second time on the full payload. This double-pass is harmless but the asymmetry is worth noting — if the call site ever changes to skip the second pass, litellm_logging_obj would contain unserialized data. Consider documenting that the second pass in _call_noma_scan is relied upon here.

Reviews (1): Last reviewed commit: "Fix spaces" | Re-trigger Greptile

@codecov

codecov Bot commented Apr 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@krrish-berri-2
krrish-berri-2 merged commit 0304fe0 into BerriAI:litellm_internal_staging Apr 27, 2026
43 checks passed
yugborana pushed a commit to yugborana/litellm that referenced this pull request Jun 2, 2026
…#26605)

* Use auth key name if there are no app id in in headers or in extra_data

* use key alias instead of key name

* Fix

* last priority key alias

* Fix

* Add tests

* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (BerriAI#26449)

* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro

Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:

- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
  input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
  per 1M input/output/cached input

Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.

No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.

Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields

* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants

gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.

Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.

Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.

* [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (BerriAI#26361)

* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)

Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.

Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
  $60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
  but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro

reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once BerriAI#26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.

Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.

* test: register supports_low_reasoning_effort in cost-map JSON schema

azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.

Note: the runtime side of this flag (code that reads it) lands in
BerriAI#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.

* Use sanitize deep copy style to replace deepcopy usage

* Added test checking error is not happening anymore

* Added warning log when json copy failed

* Reduce to one change

* Fix spaces

---------

Co-authored-by: Ido Lavi <ido@noma.security>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: TomAlon <tom@noma.security>
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…#26605)

* Use auth key name if there are no app id in in headers or in extra_data

* use key alias instead of key name

* Fix

* last priority key alias

* Fix

* Add tests

* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (BerriAI#26449)

* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro

Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:

- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
  input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
  per 1M input/output/cached input

Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.

No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.

Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields

* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants

gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.

Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.

Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.

* [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (BerriAI#26361)

* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)

Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.

Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
  $60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
  but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro

reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once BerriAI#26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.

Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.

* test: register supports_low_reasoning_effort in cost-map JSON schema

azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.

Note: the runtime side of this flag (code that reads it) lands in
BerriAI#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.

* Use sanitize deep copy style to replace deepcopy usage

* Added test checking error is not happening anymore

* Added warning log when json copy failed

* Reduce to one change

* Fix spaces

---------

Co-authored-by: Ido Lavi <ido@noma.security>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: TomAlon <tom@noma.security>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants