Skip to content

fix(cloud-agent-next): stabilize queue e2e scenarios after #6660 - #6726

Merged
eshurakov merged 3 commits into
mainfrom
eshurakov/cloud-agent-e2e-queue-stability
Sep 28, 2026
Merged

eshurakov merged 3 commits into
mainfrom
eshurakov/cloud-agent-e2e-queue-stability

Conversation

@eshurakov

Copy link
Copy Markdown
Contributor

Summary

Fix flakiness in the deployed e2e queue scenarios introduced by the delivery semantics of #6660.

headQueuedMessageId now returns the oldest queued id even while another message is accepted, so a ready runtime receives queued follow-ups without waiting for the running turn. Two scenarios assumed the previous semantics:

  • queue-overflow: sent fills sequentially, so the DO drained the pending queue between sends and the 10-message limit was often never reached — no 429.
  • queue-while-busy: snapshotted the event buffer at the third message's terminal and required all three completion frames to already be present. fix(cloud-agent-next): deliver follow-ups without waiting for the current turn #6660 settles batched follow-ups through concurrent per-message deliveries, so completion frames can arrive out of order or late.

Changes (test-only, services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts):

  • queue-overflow fills the pending queue with bounded concurrent waves. Admissions are serialized inside the DO, so a burst closes the capacity window before delivery can drain it, and the server-enforced 429 is reliably reached.
  • queue-while-busy and queue-rapid-fire-no-gate wait for every expected message terminal, then assert all completed and FIFO on cloud.message.sent. In-order delivery is the documented contract (SESSION-CONTINUITY.md E2, e2e README); completion-frame arrival order is not.

Adds regression unit tests in scenarios-shared.test.ts; both fail on the previous implementation.

Verification

  • Local real stack (pnpm dev:start cloud-agent, worktree offset 1000): queue-overflow 4/4 runs reach TOO_MANY_REQUESTS: Pending message queue is full (10), cleanup cleared; queue-while-busy 2/2; queue-rapid-fire-no-gate 1/1; queue-interrupt-clears 1/1.
  • New regression unit tests fail against the stashed previous source and pass with the fix.
  • pnpm --filter cloud-agent-next typecheck, lint, and the full unit suite (7665 passed, 3 skipped).

Visual Changes

N/A

Reviewer Notes

  • Test-only change: no Worker deploy is needed to re-run cloud-agent-e2e-tests.yml.
  • Not reproduced locally before the fix — the original failures appeared only under the deployed parallel matrix (E2E_PARALLEL: 4), so the mechanism is inferred from fix(cloud-agent-next): deliver follow-ups without waiting for the current turn #6660, the queue integration test, and the failure signatures rather than a measured before/after flake rate.
  • The FIFO assertion was moved from completion-frame order to delivery order (cloud.message.sent). If reviewers consider serial completion a contract, the wrapper's concurrent per-message settlement is the place to enforce it.

#6660 dispatches queued follow-ups to a ready runtime without waiting for the running turn, so the pending queue drains between sequential sends and batched follow-ups settle through concurrent per-message deliveries.

queue-overflow now fills the pending queue with bounded concurrent waves so the server-enforced 429 is reliably reached. queue-while-busy and queue-rapid-fire-no-gate now wait for every expected terminal and assert FIFO on cloud.message.sent (in-order delivery) instead of requiring completion frames to arrive in order.

Adds regression unit tests; both fail on the previous implementation.
Comment thread services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts Outdated
Comment thread services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts
@kilo-code-bot

kilo-code-bot Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change at 4242bc0e resolves the prior cleanup-barrier finding: the queue-overflow wave loop now scans every admitted result and guards the success message, and no new issues were found in the changed lines.

Files Reviewed (1 file)
  • services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts
Previous Review Summaries (2 snapshots, latest commit 408500a)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 408500a)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The two prior suggestions were verified fixed at 408500ae7e; one new test-quality gap remains in the queue-overflow wave loop, where the early break excludes some admitted messages from the cleanup barrier.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts 549 break on overflowOk skips admitted results later in the wave, so they are not awaited by the cleanup barrier
Files Reviewed (1 file)
  • services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts - 1 issue

Fix these issues in Kilo Cloud

Previous review (commit 65b38df)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

Test-only stabilization PR; both findings are low-severity diagnostic gaps in the new bounded concurrent fill loop of queue-overflow, not runtime or logic defects.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 2
Issue Details (click to expand)

SUGGESTION

File Line Issue
services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts 552 Failure detail prints the post-increment fillAttempt, not the failing attempt
services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts 563 A later after-budget 429 can overwrite overflowMessage after overflowOk is set
Files Reviewed (2 files)
  • services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts - 2 issues
  • services/cloud-agent-next/test/unit/e2e/scenarios-shared.test.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0

Review guidance: REVIEW.md from base branch main

Report the failing attempt index (not the post-wave counter) and stop scanning a wave once a within-budget 429 is found, so a later after-budget rejection cannot overwrite the success message while ok is true.
Comment thread services/cloud-agent-next/test/e2e/scenarios-shared-queue.ts Outdated
Finish scanning each wave after a rejection so every admitted message enters queuedIds and the failed-event barrier waits for it. Guard the success assignment with overflowOk so a later after-budget rejection cannot overwrite the message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants