Skip to content

test(e2e): cover Google provider mutations - #6555

Merged
serrrfirat merged 3 commits into
mainfrom
codex/google-provider-operation-batch
Jul 23, 2026
Merged

serrrfirat merged 3 commits into
mainfrom
codex/google-provider-operation-batch

Conversation

@serrrfirat

Copy link
Copy Markdown
Collaborator

Summary

  • Add full-path typed operation cases for Gmail reply, Calendar create/delete, and Drive folder/upload.
  • Prove each mutation through exact Emulate provider readback, including unique-result counts, Gmail thread identity, calendar fields, Drive metadata, and uploaded bytes.
  • Split shared operation types and Google-owned cases from the small cross-provider registry so later provider batches do not grow one monolithic file.
  • Move five capabilities from waiver to tested, raising hermetic provider coverage from 31/123 to 36/123 without adding a CI lane, binary build, or Reborn process.

Change Type

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • CI/Infrastructure
  • Security
  • Dependencies

Linked Issue

Part of #6524

Validation

  • cargo fmt --all -- --check — Not applicable: no Rust changed.
  • cargo clippy --all --benches --tests --examples --all-features -- -D warnings — Not applicable: no Rust changed.
  • cargo build — Not applicable: no product code changed; the existing debug binary drove the full black-box lane.
  • Relevant tests pass: 107-test combined provider inventory, provider contract, recorded replay, and full-path Emulate lane.
  • Manual testing: focused all five new operations against pinned serrrfirat/emulate@92bd9c2157a74613c47549eb5cfcccc7c0740adf.
  • If a coding agent was used and supports it, review-pr or pr-shepherd --fix was run before requesting review — Not run; final diff, duplicate case IDs/capabilities, and complete caller-path lane were checked locally.

Test Strategy

User behavior: No product response contract changes. The harness now detects regressions where a real Reborn turn fails to reply in the requested Gmail thread, create/delete a calendar event, create a Drive folder, or upload exact Drive contents.

Risk areas:

  • Model behavior
  • Browser
  • Side effect
  • Persistence
  • Security or permissions
  • External provider
  • Cross-component behavior

Tests added or updated:

  • Unit or contract: Updated the closed capability inventory; the gate first failed on exactly the five unclassified operation cases, then passed after they moved from waiver to tested.
  • Reborn integration: Added five cases to the existing standalone Reborn → first-party extension → network mediation → Emulate runner, with provider-side baseline and outcome assertions.
  • Recorded fixture: Existing harvested fixtures all replayed successfully; no recorded model output was added or changed because typed operation cases provide deterministic model responses inline.
  • Browser E2E: Not applicable: no browser surface or behavior changed.
  • Backend or runtime: Not applicable: no product/runtime implementation changed.
  • Live canary: Not applicable: this PR expands hermetic coverage; the existing supplemental live-canary strategy is unchanged.

What the tests prove: Every new case executes the real capability boundary exactly once, starts from reset seeded state, completes successfully, and produces one provider-observable result with the requested semantic fields. The full combined lane proves the new mutable cases remain isolated from all existing provider contracts and harvested journeys.

Commands run:

tests/e2e/.venv/bin/python -m pytest tests/e2e/scenarios/test_provider_capability_inventory.py -q
# Expected red phase: failed only for the five newly added, not-yet-classified operation cases.

tests/e2e/.venv/bin/python -m pytest tests/e2e/scenarios/test_provider_capability_inventory.py -q
# 2 passed

IRONCLAW_EMULATE_CLI=<pinned-emulate>/packages/emulate/dist/index.js \
  tests/e2e/.venv/bin/python -m pytest \
  tests/e2e/scenarios/test_reborn_qa_trace_full_path.py \
  -k 'gmail_reply_to_message or google_calendar_create_event or google_calendar_delete_event or google_drive_create_folder or google_drive_upload_file' \
  -v --timeout=120
# 5 passed

IRONCLAW_EMULATE_CLI=<pinned-emulate>/packages/emulate/dist/index.js \
  tests/e2e/.venv/bin/python -m pytest \
  tests/e2e/scenarios/test_provider_capability_inventory.py \
  tests/e2e/scenarios/test_emulate_reborn_provider_contracts.py \
  tests/e2e/scenarios/test_reborn_qa_trace_replay.py \
  tests/e2e/scenarios/test_reborn_qa_trace_full_path.py \
  -v --timeout=120
# 107 passed

tests/e2e/.venv/bin/python -m compileall -q \
  tests/e2e/provider_operation_types.py \
  tests/e2e/provider_operation_google_cases.py \
  tests/e2e/provider_operation_cases.py

git diff --check

Security Impact

None. This changes only hermetic E2E fixtures, case definitions, and assertions. It adds no permissions, production network calls, secrets, file access, tool execution path, or sandbox-policy change.

Reborn Trust-Boundary Checklist

N/A: the PR exercises the existing Reborn trust boundaries but does not change runtime, policy, evidence, serialization, queues, errors, or host interfaces.

Database Impact

None.

Blast Radius

Limited to the combined hermetic provider E2E lane and its deterministic Google seed. A faulty oracle or seed could fail that lane; product binaries and live provider behavior are unchanged.

Rollback Plan

Revert this commit to restore the four-case registry and return the five capabilities to their owned waiver.

Review Follow-Through

No known unaddressed feedback. Reviewer focus: whether each provider readback is the right durable semantic evidence and whether the provider-owned module boundary is appropriate for subsequent batches.


Review track: A (tests/infrastructure only)

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@ironloopai

ironloopai Bot commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

🔎 IronLoop Review Status

Head: 8ff441aa4652c226e8ff640f153c60f71b71e954
Result: 1/1 reviewers completed without blocking findings.
Next: Ready for normal human review and CI checks.
Updated: 2026-07-23T08:56:19.375Z

Current reviewers:

Reviewer State Verdict Findings Last update
ironloop/common-reviewer (reviewer) Completed Approved 0 blocking findings / 0 notes 2026-07-23T08:56:19.366Z
Reviewer summaries
Reviewer Detail
ironloop/common-reviewer (reviewer) Approved; 0 blocking findings; Reviewed the complete normal, test-only diff (6 files; 411 additions/162 deletions). The added Google mutation cases are wired through the existing full-path runner with provider-…
Recent activity
Time Reviewer State Detail
2026-07-23T08:52:41.291Z ironloop/common-reviewer (reviewer) Superseded A newer PR head replaced this review (8ff441a).
2026-07-23T08:52:41.291Z ironloop/common-reviewer (reviewer) Superseded A newer PR head replaced this review (8ff441a).
2026-07-23T08:53:11.777Z ironloop/common-reviewer (reviewer) Queued Accepted review request for head 8ff441a.
2026-07-23T08:53:11.777Z ironloop/common-reviewer (reviewer) Queued Waiting for this reviewer lane to become available.
2026-07-23T08:53:12.048Z ironloop/common-reviewer (reviewer) Started Reviewer worker started.
2026-07-23T08:53:14.596Z ironloop/common-reviewer (reviewer) Workspace ready Prepared isolated checkout (merge_ref) at 36893f5.
2026-07-23T08:56:19.366Z ironloop/common-reviewer (reviewer) Result captured Approved; 0 blocking findings.
2026-07-23T08:56:19.366Z ironloop/common-reviewer (reviewer) Completed Review completed and terminal status was persisted.
Available commands
  • @ironloopai help
  • @ironloopai agents
  • @ironloopai review
  • @ironloopai review --agent <agent>
Run metadata

Admission: webhook accepted the request and IronLoop persisted reviewer state before this projection.

@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6555 July 23, 2026 08:21 Destroyed
@github-actions github-actions Bot added size: XS < 10 changed lines (excluding docs) risk: low Changes to docs, tests, or low-risk modules labels Jul 23, 2026
@coderabbitai

coderabbitai Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Google provider e2e operation cases were extracted into shared types and a dedicated Google module. Drive, Gmail, and Calendar state assertions now back registered operations, with MIME, fixture, and capability-coverage updates for the added cases.

Changes

Google provider operation coverage

Layer / File(s) Summary
Shared operation case types
tests/e2e/provider_operation_types.py
Defines provider identifiers, async assertion callback types, and the frozen ProviderOperationCase data structure.
Google operation cases and assertions
tests/e2e/provider_operation_google_cases.py, tests/e2e/provider_operation_cases.py, tests/e2e/emulate_provider.py
Adds Drive, Gmail, and Calendar baseline/outcome assertions, registers operation arguments and capability IDs, and extends MIME generation with reply headers.
Fixture and capability coverage updates
tests/e2e/fixtures/emulate/google_gmail.yaml, tests/e2e/fixtures/provider_capability_coverage.toml
Associates the seeded unread Gmail message with a thread and marks additional Google capabilities as tested rather than waived.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title follows conventional-commits style and accurately summarizes the added Google provider mutation coverage.
Description check ✅ Passed The description matches the template closely, covering summary, change type, linked issue, validation, test strategy, safety, rollback, and review notes.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/e2e/provider_operation_google_cases.py`:
- Around line 18-33: Extract a shared async _get(emulate_url, path, params=None)
-> httpx.Response helper that creates the configured httpx.AsyncClient, performs
the GET, calls raise_for_status(), and returns the response. Update _drive_file,
_drive_files_named, and the other identified call sites to use _get while
preserving their existing paths, query parameters, and JSON handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: cc680319-fa3b-4346-bf0b-fabfdd3a92d4

📥 Commits

Reviewing files that changed from the base of the PR and between 495d814 and 07b6424.

📒 Files selected for processing (5)
  • tests/e2e/fixtures/emulate/google_gmail.yaml
  • tests/e2e/fixtures/provider_capability_coverage.toml
  • tests/e2e/provider_operation_cases.py
  • tests/e2e/provider_operation_google_cases.py
  • tests/e2e/provider_operation_types.py

Comment thread tests/e2e/provider_operation_google_cases.py Outdated

@ironloopai ironloopai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ IronLoop Review: reviewer

Review at a glance

Verdict Blocking Notes Inline Head
❌ Changes requested 1 0 1 07b642487705

Head: 07b64248770517e70b4173df453af81dd2fa4df9
Next: Fix the blocking findings, push the PR branch, then re-run this reviewer.

Run details

Status: Current
Needs human: no
Needs validation: no

Summary

The focused 5-file E2E change has one blocking coverage gap: the new Gmail reply case cannot form a valid Gmail thread, so it should not move that capability from waived to tested.

Findings

Blocking: 1 / Notes: 0

Blocking findings

1. ❌ [MEDIUM] Send a Gmail-valid reply fixture

Location: tests/e2e/provider_operation_google_cases.py:271-274
This raw message uses a different subject from the seeded message (Emulate seeded unread) and raw_mime supplies neither In-Reply-To nor References. Gmail requires matching subject plus RFC-compliant In-Reply-To and References headers, in addition to threadId, to place a sent message in an existing thread (Gmail threading requirements). Emulate accepts this and makes the readback pass, but live Gmail need not, leaving the newly classified reply capability effectively untested. Seed a Message-ID and send a matching-subject RFC-compliant reply before moving this capability from the waiver.

Developer follow-up

After fixing this feedback:

  1. Push the fix to this PR branch.
  2. Re-run this reviewer with @ironloopai review --agent reviewer if you only changed this reviewer's findings.
  3. Re-run all reviewers with @ironloopai review when the fix may affect multiple areas.

"message": {
"raw": raw_mime(
to="qa-sender@example.com",
subject=GMAIL_REPLY_MARKER,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cannot be a valid Gmail reply: its subject differs from the seeded message and raw_mime adds no In-Reply-To/References headers. Gmail requires those headers and a matching subject with threadId to join an existing thread. Emulate will pass this, but production Gmail may not; seed a Message-ID and construct a compliant reply before classifying the capability as tested.

@railway-app

railway-app Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

🚅 Deployed to the ironclaw-pr-6555 environment in ironclaw-ci-preview

Service Status Web Updated (UTC)
ironclaw ✅ Success (View Logs) Web Jul 23, 2026 at 9:01 am

@serrrfirat

Copy link
Copy Markdown
Collaborator Author

@ironloopai review --agent reviewer

@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6555 July 23, 2026 08:51 Destroyed
@serrrfirat

Copy link
Copy Markdown
Collaborator Author

@ironloopai review --agent reviewer

@ironloopai ironloopai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ IronLoop Review: reviewer

Review at a glance

Verdict Blocking Notes Inline Head
✅ Approved 0 0 0 8ff441aa4652

Head: 8ff441aa4652c226e8ff640f153c60f71b71e954
Next: No reviewer action needed.

Run details

Status: Current
Needs human: no
Needs validation: no

Summary

Reviewed the complete normal, test-only diff (6 files; 411 additions/162 deletions). The added Google mutation cases are wired through the existing full-path runner with provider-side readback assertions, and the inventory classifications match those cases. No actionable regression found.

Findings

None.

Developer follow-up

After fixing this feedback:

  1. Push the fix to this PR branch.
  2. Re-run this reviewer with @ironloopai review --agent reviewer if you only changed this reviewer's findings.
  3. Re-run all reviewers with @ironloopai review when the fix may affect multiple areas.

@serrrfirat
serrrfirat merged commit 346db3c into main Jul 23, 2026
42 of 43 checks passed
@serrrfirat
serrrfirat deleted the codex/google-provider-operation-batch branch July 23, 2026 10:27

This branch was successfully deployed

No deployments
ironclaw-ci-preview / ironclaw-pr-6555 — 8ff441aa Deployed Jul 23, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: low Changes to docs, tests, or low-risk modules size: XS < 10 changed lines (excluding docs)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant