test(policy): add TPP denial contract proof by stranske · Pull Request #1684 · stranske/trip-planner

stranske · 2026-08-11T17:31:50Z

Follow-up for #1676

Addresses the verifier completion-evidence gap on merged #1682.

Adds the required tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant contract test.
Exercises a current TPP policy_status=fail snapshot through the workspace import API and asserts the preserved BUD-001 blocking reason.
Deliberate-break proof: with the policy_status == fail branch temporarily disabled, this exact test fails (compliant instead of non_compliant); restoring the mapping makes the focused suite pass.

Validation

uv run ruff check tests/integrations/test_tpp_policy_import.py
uv run ruff format --check tests/integrations/test_tpp_policy_import.py
uv run --extra dev pytest -q tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant tests/app/test_policy.py::test_workspace_policy_import_maps_tpp_failure_to_blocking_reasons tests/app/test_policy.py::test_workspace_policy_import_surfaces_live_tpp_unavailable_errors

Summary by CodeRabbit

Tests
- Added end-to-end coverage for importing policies with failing verdicts.
- Verifies that blocked budget conditions produce a non-compliant evaluation with the expected failure details.

cursor · 2026-08-11T17:31:55Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

netlify · 2026-08-11T17:32:06Z

✅ Deploy Preview for stranske-trip-planner canceled.

Name	Link
🔨 Latest commit	`33c9357`
🔍 Latest deploy log	https://app.netlify.com/projects/stranske-trip-planner/deploys/6a7b5e21ed400d0008311f46

coderabbitai · 2026-08-11T17:32:20Z

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 53 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9446ac71-d963-43fc-9f97-24038af908a4

📥 Commits

Reviewing files that changed from the base of the PR and between e90e67c and 33c9357.

📒 Files selected for processing (1)

tests/integrations/test_tpp_policy_import.py

📝 Walkthrough

Walkthrough

This change adds end-to-end coverage for importing a failing TPP policy verdict. The test creates an isolated workspace, imports a modified policy fixture, and verifies the non-compliant result and its failure details.

Changes

TPP policy import

Layer / File(s)	Summary
Failing policy import evaluation `tests/integrations/test_tpp_policy_import.py`	The integration test creates an isolated workspace and authenticated client, imports a policy with a blocking budget issue, and verifies the non-compliant evaluation, issue code, message, severity, and category.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

stranske/trip-planner#1676 — The test covers the issue acceptance criteria for preserving failure details in a non-compliant TPP policy result.

Possibly related PRs

stranske/trip-planner#1682 — This PR adds integration coverage for the same failing-policy verdict behavior.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the added policy test and its purpose of proving TPP denial handling.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch closer/followup-1676-verifier-evidence

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integrations/test_tpp_policy_import.py`:
- Around line 37-44: Add status_code == 201 assertions with response.text to the
signup and trip creation responses in the test setup before yielding the client
or extracting trip_id. Update the relevant setup flow around the client fixture
and trip payload while preserving the existing successful-response behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3e08c23c-3dae-42f2-9c8d-7bf1e4085f1e

📥 Commits

Reviewing files that changed from the base of the PR and between 70db565 and e90e67c.

📒 Files selected for processing (1)

tests/integrations/test_tpp_policy_import.py

Address CodeRabbit review: include response.text on setup status checks before extracting trip_id in the TPP denial contract proof test. Co-authored-by: Cursor <cursoragent@cursor.com>

stranske · 2026-08-11T17:38:46Z

Closer review repair (cursor)

Addressed the active CodeRabbit thread on setup response assertions:

workspace_client fixture now asserts signup.status_code == 201 with signup.text
test_failing_policy_status_is_not_compliant now asserts trip.status_code == 201 with trip.text before extracting trip_id

Validation: uv run ruff check + uv run pytest -q tests/integrations/test_tpp_policy_import.py — 1 passed.

Pushed 33c93574. Post-push seven-minute review/check window active; merge deferred to next round after fresh Gate green + zero active non-outdated threads on this exact head.

github-actions · 2026-08-11T18:28:55Z

Provider Comparison Report

Provider Summary

Provider	Model	Verdict	Confidence	Summary
openai	gpt-5.6-terra	PASS	90%	The change adds the required `tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant` contract coverage. The added integration test exercises a TPP policy snapsho...
anthropic	claude-sonnet-5	CONCERNS	N/A	Review the PR manually or re-run once LLM credentials are available.

📋 Full Provider Details (click to expand)

openai

Model: gpt-5.6-terra
Verdict: PASS
Confidence: 90%
Scores:
- Correctness: 9.0/10
- Completeness: 9.0/10
- Quality: 8.0/10
- Testing: 9.0/10
- Risks: 9.0/10
Summary: The change adds the required tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant contract coverage. The added integration test exercises a TPP policy snapshot with policy_status=fail through the workspace-import path and verifies the result remains non-compliant with the BUD-001 blocking reason preserved. This directly protects the failure-to-blocking-reason mapping covered by the related application-policy tests. The test-focused implementation is readable and appropriately targets the documented regression. The accompanying compatibility/style edits appear mechanical and do not introduce an evident behavioral or security risk.

anthropic

Model: claude-sonnet-5
Verdict: CONCERNS
Confidence: N/A
Summary: Review the PR manually or re-run once LLM credentials are available.
Concerns:
- LLM evaluation could not run.
Error: LLM invocation failed: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.'}, 'request_id': 'req_011CdwSBvEcC99qmxWHUtVcR'}

Agreement

No clear areas of agreement.

Disagreement

Dimension	openai	anthropic
Verdict	PASS	CONCERNS

Unique Insights

openai: The change adds the required tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant contract coverage. The added integration test exercises a TPP policy snapshot with policy_status=fail through the workspace-import path and verifies the result remains non-co...
anthropic: LLM evaluation could not run.

🔍 LangSmith Traces

openai

github-actions · 2026-08-11T18:29:13Z

Workflow state fingerprint for Agents Verifier. Do not edit.

stranske · 2026-08-11T18:43:57Z

Closer verifier disposition (cursor)

Audited the durable provider-comparison report on merged #1684 against current main and the #1676 acceptance criteria.

OpenAI (gpt-5.6-terra): PASS (90%). The report confirms the issue-named contract test tests/integrations/test_tpp_policy_import.py::test_failing_policy_status_is_not_compliant exists, exercises a policy_status=fail TPP snapshot through the workspace-import path, and verifies non_compliant with the BUD-001 blocking reason preserved.

Anthropic (claude-sonnet-5): CONCERNS is an infrastructure false positive. The only concern is LLM evaluation could not run with error Your credit balance is too low to access the Anthropic API — not a product/code defect. Same class of false positive dispositioned on prior fleet PRs (e.g. Travel-Plan-Permission #1430, trip-planner #1682).

Review threads: 0 active non-outdated threads on merged head 33c93574.

Disposition: Follow-up chain for #1676 is complete. No additional bounded follow-up PR is warranted. Closer emits followup_completed + reset-chain.

test(policy): prove failing TPP verdict remains non-compliant

e90e67c

stranske added agent:codex Assign to Codex agent agents:keepalive Enable keepalive monitoring on PR autofix Let bots format/lint automatically follow-up labels Aug 11, 2026

stranske temporarily deployed to agent-standard August 11, 2026 17:32 — with GitHub Actions Inactive

coderabbitai Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread tests/integrations/test_tpp_policy_import.py Outdated

test(policy): assert signup and trip setup return 201

33c9357

Address CodeRabbit review: include response.text on setup status checks before extracting trip_id in the TPP denial contract proof test. Co-authored-by: Cursor <cursoragent@cursor.com>

stranske merged commit 51645d1 into main Aug 11, 2026
28 checks passed

stranske deleted the closer/followup-1676-verifier-evidence branch August 11, 2026 18:24

stranske added the verify:compare Runs verifier comparison mode after merge label Aug 11, 2026

stranske temporarily deployed to agent-standard August 11, 2026 18:24 — with GitHub Actions Inactive

Navigation Menu

test(policy): add TPP denial contract proof - #1684

test(policy): add TPP denial contract proof#1684
stranske merged 2 commits into
mainfrom
closer/followup-1676-verifier-evidence

stranske commented Aug 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

cursor Bot commented Aug 11, 2026

Uh oh!

netlify Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading

Review limit reached

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

stranske commented Aug 11, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

openai

anthropic

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

stranske commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

stranske commented Aug 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Follow-up for #1676

Validation

Summary by CodeRabbit

Uh oh!

cursor Bot commented Aug 11, 2026

Uh oh!

netlify Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for stranske-trip-planner canceled.

Uh oh!

coderabbitai Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stranske commented Aug 11, 2026

Closer review repair (cursor)

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Provider Comparison Report

Provider Summary

openai

anthropic

Agreement

Disagreement

Unique Insights

🔍 LangSmith Traces

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

stranske commented Aug 11, 2026

Closer verifier disposition (cursor)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

stranske commented Aug 11, 2026 •

edited by coderabbitai Bot

Loading

netlify Bot commented Aug 11, 2026 •

edited

Loading

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading