Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/pass_through/bedrock.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,15 @@ curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-claude-model/converse-stream'
}'
```

#### post_call guardrails on `/converse-stream`

A `post_call` guardrail on the caller's key scans `/converse-stream` output while it streams, the same way it scans `/v1/chat/completions` streams:

- With `streaming_buffer_until_moderated: false` (the default for most guardrails) the proxy forwards frames as Bedrock sends them and scans the text every `streaming_sampling_rate` frames (default 5) and at the end. Text reaches the client before the guardrail's verdict, and masks are not applied. Each scan pauses the stream until the guardrail answers, so a slow guardrail stretches a long answer (a 700-word answer took about 47 s instead of 12 s with the Bedrock guardrail); add `streaming_end_of_stream_only: true` to scan once at the end instead.
- With `streaming_buffer_until_moderated: true` (the default for the [Bedrock guardrail](../proxy/guardrails/bedrock#streaming)) the proxy holds the stream until it is scanned, then sends it with masks applied per content block.
- A block after the first frame ends the stream with an AWS `validationException` event, which the AWS SDK raises as `EventStreamError` with the guardrail's message. A block before the first frame returns an HTTP error.
- Guardrails that only implement `async_post_call_success_hook`, guardrails with `mask_response_content: true`, and post_call policy pipelines still hold the whole stream and scan it once.

### Supported Bedrock Endpoints with config.yaml

When using models from config.yaml, you can call any Bedrock endpoint:
Expand Down
2 changes: 1 addition & 1 deletion docs/proxy/guardrails/bedrock.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ Chunks now stream to the client as they arrive, and one OUTPUT scan runs over th

With `streaming_buffer_until_moderated: false` alone, the guardrail scans the accumulated response every `streaming_sampling_rate` chunks while streaming. Each sampled scan is a separate ApplyGuardrail call over all text so far, so it adds mid-stream latency and repeated Bedrock text-unit charges. Pair it with `streaming_end_of_stream_only: true` unless you need mid-stream blocking.

These settings apply to both `/v1/chat/completions` and native `/v1/messages` streams.
These settings apply to `/v1/chat/completions`, native `/v1/messages` streams, and the Bedrock `/bedrock/model/<model>/converse-stream` pass-through, where a mid-stream block ends the stream with an AWS `validationException` event that the AWS SDK raises with the guardrail's message.

## Contextual Grounding

Expand Down
Loading