test: add Postman e2e collection and runner for virtual key expiry validation and enforcement - #4888
Merged
Merged
Conversation
|
|
Collaborator
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jul 3, 2026
Contributor
📝 WalkthroughWalkthroughAdds a new Postman e2e collection for virtual-key expiry behavior covering creation validation, update semantics, runtime enforcement, and teardown, plus a companion Bash/Newman runner script that resolves environments, builds globals, executes the collection, and reports results. ChangesVirtual Key Expiry E2E Tests
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Newman
participant BifrostAPI
participant MCPTool
Newman->>BifrostAPI: chat completion with unexpired VK
BifrostAPI-->>Newman: 200 success
Newman->>MCPTool: execute MCP tool with unexpired VK
MCPTool-->>Newman: 200 success
Newman->>BifrostAPI: set short expiry on VK
Newman->>BifrostAPI: chat completion with expired VK
BifrostAPI-->>Newman: 403 virtual_key_blocked (expired)
Newman->>BifrostAPI: extend expiry into future
Newman->>BifrostAPI: chat completion with extended VK
BifrostAPI-->>Newman: 200 success
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/e2e/api/collections/bifrost-v1-vk-expiry.postman_collection.json`:
- Around line 1171-1178: The prerequest delay in the Postman collection is a
no-op, so the expiry wait before the 403 checks is unreliable. Update the
prerequest script in the affected collection item to use a
blocking/runner-supported delay instead of setTimeout, and make sure the
subsequent request assertions for the expired virtual key (including the MCP
follow-up) still run after the wait.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 173f46c3-6236-46d9-9867-3244f07bf0a8
📒 Files selected for processing (2)
tests/e2e/api/collections/bifrost-v1-vk-expiry.postman_collection.jsontests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh
Pratham-Mishra04
force-pushed
the
07-03-tests_adds_harness_for_vk_expiry
branch
from
July 3, 2026 14:53
cee233a to
729e296
Compare
Pratham-Mishra04
force-pushed
the
07-03-feat_add_expiration_support_to_virtual_keys
branch
from
July 3, 2026 14:53
21eac0c to
041a823
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh (2)
195-195: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMinor: prefer
find/statoverlsfor report listing.Purely cosmetic output (not parsed), so this is very low priority.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh` at line 195, The report listing in the Newman expiry test script uses ls, which should be replaced with find/stat for more robust and predictable output. Update the report listing logic in the shell script where the current ls -lh pipeline appears so it enumerates files via find and formats sizes with stat while preserving the same human-readable, non-parsed output.Source: Linters/SAST tools
87-91: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueUnguarded
jqfailure could hard-exit underset -ewith a confusing error.If
jqfails (e.g., malformedprovider-capabilities.json), the script exits immediately without the friendlier error messaging used elsewhere in the script (e.g., Lines 70-74, 77-80).🛡️ Optional: guard the jq invocation
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then GLOBALS_TMP=$(mktemp) trap 'rm -f "$GLOBALS_TMP"' EXIT - jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP" + if ! jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"; then + echo -e "${RED}Error: Failed to build globals from $PROVIDER_CAPABILITIES_JSON${NC}" + exit 1 + fi fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh` around lines 87 - 91, The jq invocation in run-newman-vk-expiry-tests.sh is unguarded, so a malformed provider-capabilities.json can cause an immediate set -e exit without the script’s usual friendly error handling. Update the existing block around GLOBALS_TMP and the jq command to check jq’s exit status, then emit a clear error message consistent with the surrounding setup/validation logic before exiting; use the same style as the nearby capability-file checks so failures are easier to diagnose.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh`:
- Line 195: The report listing in the Newman expiry test script uses ls, which
should be replaced with find/stat for more robust and predictable output. Update
the report listing logic in the shell script where the current ls -lh pipeline
appears so it enumerates files via find and formats sizes with stat while
preserving the same human-readable, non-parsed output.
- Around line 87-91: The jq invocation in run-newman-vk-expiry-tests.sh is
unguarded, so a malformed provider-capabilities.json can cause an immediate set
-e exit without the script’s usual friendly error handling. Update the existing
block around GLOBALS_TMP and the jq command to check jq’s exit status, then emit
a clear error message consistent with the surrounding setup/validation logic
before exiting; use the same style as the nearby capability-file checks so
failures are easier to diagnose.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: bc6772df-4abc-4b5c-b560-65daafbdf617
📒 Files selected for processing (2)
tests/e2e/api/collections/bifrost-v1-vk-expiry.postman_collection.jsontests/e2e/api/runners/individual/run-newman-vk-expiry-tests.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/e2e/api/collections/bifrost-v1-vk-expiry.postman_collection.json
Contributor
Merge activity
|
akshaydeo
changed the base branch from
07-03-feat_add_expiration_support_to_virtual_keys
to
graphite-base/4888
July 3, 2026 15:17
akshaydeo
dismissed
coderabbitai[bot]’s stale review
July 3, 2026 15:17
The base branch was changed.
yangtuooc
added a commit
to yangtuooc/bifrost
that referenced
this pull request
Jul 4, 2026
* 'dev' of https://github.com/maximhq/bifrost: ipv6 support (maximhq#4895) docs: add virtual key expiry support docs (maximhq#4889) test: add Postman e2e collection and runner for virtual key expiry validation and enforcement (maximhq#4888) feat: add expiry field to virtual keys (maximhq#4887) fix: converts thinking to disabled if tool choice is required for deepseek (maximhq#4861) chore: adds docs for deepseek provider (maximhq#4854) chore: adds tests for deepseek provider (maximhq#4853) feat: adds deepseek provider (maximhq#4852) fix: cost for image generation or image edit streaming (maximhq#4802) feat: add `BedrockMantleKeyConfig` support to key hashing, schema/table mapping, and sensitive field clearing (maximhq#4886) fix: skip O(N) reference refresh on request-time rate-limit/budget reset (maximhq#4883) refactor: simplify Responses lifecycle permissions to require explicit per-verb flags and expose them in UI (maximhq#4880) fix: append datasheet models for incomplete list models call (maximhq#4879) # Conflicts: # ui/app/workspace/providers/fragments/allowedRequestsFields.tsx # ui/app/workspace/virtual-keys/views/virtualKeyDetailsSheet.tsx # ui/app/workspace/virtual-keys/views/virtualKeySheet.tsx # ui/app/workspace/virtual-keys/views/virtualKeysTable.tsx # ui/components/ui/datePickerWithRange.tsx
akhsaul
pushed a commit
to akhsaul/bifrost
that referenced
this pull request
Aug 27, 2026
…lidation and enforcement (maximhq#4888) ## Summary Adds an end-to-end Postman collection and Newman runner script for virtual key expiry functionality. This covers the full lifecycle of `expires_at` on virtual keys: create/update validation, persistence and retrieval semantics, and runtime enforcement at the governance layer for both inference and MCP tool execution. ## Changes - Added `bifrost-v1-vk-expiry.postman_collection.json` with four test groups: - **Setup**: Creates a VK without expiry, discovers connected MCP clients, grants MCP access, and probes candidates to select a working tool for enforcement tests. - **Create Validation**: Asserts that past expiry timestamps are rejected with 400 and a "future" error message, and that future timestamps are accepted and echoed back correctly. - **Update Semantics**: Verifies setting, extending, and clearing `expires_at` via PUT; confirms that omitting `expires_at` in an update leaves the existing value unchanged; validates RFC3339 parsing including timezone offsets; and asserts that invalid timestamps return 400 with an RFC3339 error. - **Enforcement**: Confirms unexpired VKs pass governance, expired VKs return 403 with `virtual_key_blocked` and an "expired" reason, inactive takes precedence over expired in the rejection reason, reactivation alone does not un-expire a key, and clearing or extending expiry restores access. All enforcement checks cover both `/v1/chat/completions` and `/v1/mcp/tool/execute`. - **Teardown**: Deletes both VKs created during the run. - Added `run-newman-vk-expiry-tests.sh` to run the collection via Newman with support for `--env`, `--verbose`, `--html`, `--json`, and `--bail` flags, provider environment file resolution, and `BIFROST_BASE_URL` override. ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh # Install Newman if not already installed npm install -g newman # Run with the default OpenAI environment cd tests/e2e/api bash runners/individual/run-newman-vk-expiry-tests.sh # Run with a specific provider environment bash runners/individual/run-newman-vk-expiry-tests.sh --env openai # Run with HTML and JSON reports bash runners/individual/run-newman-vk-expiry-tests.sh --env openai --html --json # Override the base URL BIFROST_BASE_URL=http://localhost:9090 bash runners/individual/run-newman-vk-expiry-tests.sh ``` Expected outcome: all tests pass. Enforcement tests that depend on a connected MCP client with a safe probe tool are automatically skipped when no suitable client is available. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations The collection self-provisions and cleans up its own virtual keys. No credentials or secrets are embedded; provider API keys are supplied via the existing Postman environment files. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end Postman collection and Newman runner script for virtual key expiry functionality. This covers the full lifecycle of
expires_aton virtual keys: create/update validation, persistence and retrieval semantics, and runtime enforcement at the governance layer for both inference and MCP tool execution.Changes
bifrost-v1-vk-expiry.postman_collection.jsonwith four test groups:expires_atvia PUT; confirms that omittingexpires_atin an update leaves the existing value unchanged; validates RFC3339 parsing including timezone offsets; and asserts that invalid timestamps return 400 with an RFC3339 error.virtual_key_blockedand an "expired" reason, inactive takes precedence over expired in the rejection reason, reactivation alone does not un-expire a key, and clearing or extending expiry restores access. All enforcement checks cover both/v1/chat/completionsand/v1/mcp/tool/execute.run-newman-vk-expiry-tests.shto run the collection via Newman with support for--env,--verbose,--html,--json, and--bailflags, provider environment file resolution, andBIFROST_BASE_URLoverride.Type of change
Affected areas
How to test
Expected outcome: all tests pass. Enforcement tests that depend on a connected MCP client with a safe probe tool are automatically skipped when no suitable client is available.
Screenshots/Recordings
N/A
Breaking changes
Related issues
Security considerations
The collection self-provisions and cleans up its own virtual keys. No credentials or secrets are embedded; provider API keys are supplied via the existing Postman environment files.
Checklist
docs/contributing/README.mdand followed the guidelines