test(e2e): make the bedrock guardrail test match the guardrail it points at - #34568
Merged
mubashir1osmani merged 1 commit intoJul 25, 2026
Merged
Conversation
…nts at The bedrock guardrail e2e test could never pass on stage. Two reasons. It sent a bomb-making prompt expecting "stock hate/violence filters" to block, but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no contentPolicy at all; it denies the topic and words "bread"/"cake" plus profanity. ApplyGuardrail returns action=NONE for the old prompt, so the request passes and the test reports "default-on guardrail did not block". Send a prompt the configured policy actually denies instead. It also registered the guardrail with aws_access_key_id / aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those env vars are deliberately absent from the gateway (static AWS keys hijack RDS IAM auth), and guardrail litellm_params do not expand os.environ/ indirection, so the literal string reached boto and failed with "Invalid AWS region format: 'os.environ/AWS_REGION'". Drop all three and let the gateway sign ApplyGuardrail with its own pod-identity role, which is how the standard stack is meant to reach Bedrock. Verified against the live stage proxy: registering the guardrail with only identifier/version and sending the new prompt returns 400 "Violated guardrail policy", satisfying both assertions.
Contributor
Greptile SummaryUpdates the live Bedrock guardrail test to match its configured policy and runtime authentication model.
Confidence Score: 5/5The PR appears safe to merge because the test now exercises the configured Bedrock policy using the gateway’s intended AWS identity. The omitted credential fields are optional and resolve through the existing ambient AWS credential chain, while the revised prompt corrects a mismatch between the test input and the live guardrail policy without weakening the rejection assertions.
|
| Filename | Overview |
|---|---|
| tests/e2e/guardrails/guardrails_client.py | Removes optional AWS credential fields from Bedrock guardrail registration, allowing the supported ambient credential chain to authenticate through pod identity. |
| tests/e2e/guardrails/test_bedrock_guardrail_e2e.py | Aligns the blocked prompt with the live guardrail’s configured policy while retaining assertions that require a guardrail-shaped rejection. |
Reviews (1): Last reviewed commit: "test(e2e): make the bedrock guardrail te..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
mubashir1osmani
enabled auto-merge (squash)
July 25, 2026 03:43
yucheng-berri
approved these changes
Jul 25, 2026
mubashir1osmani
merged commit Jul 25, 2026
fa9e0f1
into
litellm_internal_staging
76 of 77 checks passed
Ericcwang23
pushed a commit
to Ericcwang23/litellm
that referenced
this pull request
Jul 27, 2026
…nts at (BerriAI#34568) The bedrock guardrail e2e test could never pass on stage. Two reasons. It sent a bomb-making prompt expecting "stock hate/violence filters" to block, but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no contentPolicy at all; it denies the topic and words "bread"/"cake" plus profanity. ApplyGuardrail returns action=NONE for the old prompt, so the request passes and the test reports "default-on guardrail did not block". Send a prompt the configured policy actually denies instead. It also registered the guardrail with aws_access_key_id / aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those env vars are deliberately absent from the gateway (static AWS keys hijack RDS IAM auth), and guardrail litellm_params do not expand os.environ/ indirection, so the literal string reached boto and failed with "Invalid AWS region format: 'os.environ/AWS_REGION'". Drop all three and let the gateway sign ApplyGuardrail with its own pod-identity role, which is how the standard stack is meant to reach Bedrock. Verified against the live stage proxy: registering the guardrail with only identifier/version and sending the new prompt returns 400 "Violated guardrail policy", satisfying both assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
The bedrock guardrail e2e test could never pass on stage, for two independent reasons.
Changes
It sent a bomb-making prompt expecting "stock hate/violence filters" to block it, but the guardrail the suite points at (
wk4ijrsk7ska, named "husky") has nocontentPolicyat all; it denies the topic and words "bread"/"cake" plus profanity.ApplyGuardrailreturnsaction: NONEfor the old prompt, so the request passes and the test reports "default-on guardrail did not block". The prompt now matches what the configured policy actually deniesIt also registered the guardrail with
aws_access_key_id/aws_secret_access_key/aws_region_nameset to"os.environ/..."strings. Those env vars are deliberately absent from the gateway (static AWS keys hijack RDS IAM auth), and guardraillitellm_paramsdo not expandos.environ/indirection, so the literal string reached boto3 and failed withInvalid AWS region format: 'os.environ/AWS_REGION'. All three are dropped so the gateway signsApplyGuardrailwith its own pod-identity role, which is how the standard stack is meant to reach BedrockThe underlying
os.environ/expansion gap is fixed separately in #34570.Proof
Guardrail policy, straight from AWS:
Pod-identity credentials were confirmed working via STS (
assumed-role/berrie-litellm-stage-litellm-gateway/...) and a successful directApplyGuardrailcall.Type
✅ Test