Skip to content

test(e2e): make the bedrock guardrail test match the guardrail it points at - #34568

Merged
mubashir1osmani merged 1 commit into
litellm_internal_stagingfrom
litellm_e2e_bedrock_guardrail_fixture
Jul 25, 2026
Merged

test(e2e): make the bedrock guardrail test match the guardrail it points at#34568
mubashir1osmani merged 1 commit into
litellm_internal_stagingfrom
litellm_e2e_bedrock_guardrail_fixture

Conversation

@mubashir1osmani

@mubashir1osmani mubashir1osmani commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

TLDR

The bedrock guardrail e2e test could never pass on stage, for two independent reasons.

Changes

It sent a bomb-making prompt expecting "stock hate/violence filters" to block it, but the guardrail the suite points at (wk4ijrsk7ska, named "husky") has no contentPolicy at all; it denies the topic and words "bread"/"cake" plus profanity. ApplyGuardrail returns action: NONE for the old prompt, so the request passes and the test reports "default-on guardrail did not block". The prompt now matches what the configured policy actually denies

It also registered the guardrail with aws_access_key_id / aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those env vars are deliberately absent from the gateway (static AWS keys hijack RDS IAM auth), and guardrail litellm_params do not expand os.environ/ indirection, so the literal string reached boto3 and failed with Invalid AWS region format: 'os.environ/AWS_REGION'. All three are dropped so the gateway signs ApplyGuardrail with its own pod-identity role, which is how the standard stack is meant to reach Bedrock

The underlying os.environ/ expansion gap is fixed separately in #34570.

Proof

Guardrail policy, straight from AWS:

name: husky | status: READY
contentPolicy: NOT CONFIGURED
topicPolicy: DENY topic "bread" ("DO NOT TALK ABOUT BREAD")
wordPolicy: blocks "bread", "cake", + PROFANITY managed list
boto ApplyGuardrail, old prompt   : action=NONE          (cannot block)
boto ApplyGuardrail, new prompt   : GUARDRAIL_INTERVENED
via proxy, identifier+version only: 400 "Violated guardrail policy"  (satisfies both assertions)
with the old os.environ/ params   : 500 Invalid AWS region format: 'os.environ/AWS_REGION'

Pod-identity credentials were confirmed working via STS (assumed-role/berrie-litellm-stage-litellm-gateway/...) and a successful direct ApplyGuardrail call.

Type

✅ Test

…nts at

The bedrock guardrail e2e test could never pass on stage. Two reasons.

It sent a bomb-making prompt expecting "stock hate/violence filters" to block,
but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no
contentPolicy at all; it denies the topic and words "bread"/"cake" plus
profanity. ApplyGuardrail returns action=NONE for the old prompt, so the
request passes and the test reports "default-on guardrail did not block".
Send a prompt the configured policy actually denies instead.

It also registered the guardrail with aws_access_key_id /
aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those
env vars are deliberately absent from the gateway (static AWS keys hijack RDS
IAM auth), and guardrail litellm_params do not expand os.environ/ indirection,
so the literal string reached boto and failed with "Invalid AWS region format:
'os.environ/AWS_REGION'". Drop all three and let the gateway sign
ApplyGuardrail with its own pod-identity role, which is how the standard stack
is meant to reach Bedrock.

Verified against the live stage proxy: registering the guardrail with only
identifier/version and sending the new prompt returns 400 "Violated guardrail
policy", satisfying both assertions.
@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Updates the live Bedrock guardrail test to match its configured policy and runtime authentication model.

  • Removes explicit static AWS credential references so ApplyGuardrail uses the gateway pod identity.
  • Replaces the unsupported violence-policy prompt with a prompt denied by the target guardrail’s word/topic policy.
  • Documents the guardrail policy and credential assumptions directly in the test.

Confidence Score: 5/5

The PR appears safe to merge because the test now exercises the configured Bedrock policy using the gateway’s intended AWS identity.

The omitted credential fields are optional and resolve through the existing ambient AWS credential chain, while the revised prompt corrects a mismatch between the test input and the live guardrail policy without weakening the rejection assertions.

Important Files Changed

Filename Overview
tests/e2e/guardrails/guardrails_client.py Removes optional AWS credential fields from Bedrock guardrail registration, allowing the supported ambient credential chain to authenticate through pod identity.
tests/e2e/guardrails/test_bedrock_guardrail_e2e.py Aligns the blocked prompt with the live guardrail’s configured policy while retaining assertions that require a guardrail-shaped rejection.

Reviews (1): Last reviewed commit: "test(e2e): make the bedrock guardrail te..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mubashir1osmani
mubashir1osmani enabled auto-merge (squash) July 25, 2026 03:43
@mubashir1osmani
mubashir1osmani merged commit fa9e0f1 into litellm_internal_staging Jul 25, 2026
76 of 77 checks passed
@mubashir1osmani
mubashir1osmani deleted the litellm_e2e_bedrock_guardrail_fixture branch July 25, 2026 17:32
Ericcwang23 pushed a commit to Ericcwang23/litellm that referenced this pull request Jul 27, 2026
…nts at (BerriAI#34568)

The bedrock guardrail e2e test could never pass on stage. Two reasons.

It sent a bomb-making prompt expecting "stock hate/violence filters" to block,
but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no
contentPolicy at all; it denies the topic and words "bread"/"cake" plus
profanity. ApplyGuardrail returns action=NONE for the old prompt, so the
request passes and the test reports "default-on guardrail did not block".
Send a prompt the configured policy actually denies instead.

It also registered the guardrail with aws_access_key_id /
aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those
env vars are deliberately absent from the gateway (static AWS keys hijack RDS
IAM auth), and guardrail litellm_params do not expand os.environ/ indirection,
so the literal string reached boto and failed with "Invalid AWS region format:
'os.environ/AWS_REGION'". Drop all three and let the gateway sign
ApplyGuardrail with its own pod-identity role, which is how the standard stack
is meant to reach Bedrock.

Verified against the live stage proxy: registering the guardrail with only
identifier/version and sending the new prompt returns 400 "Violated guardrail
policy", satisfying both assertions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants