fix(kiloclaw) smoke prod parity - #4901
Merged
Merged
Conversation
…on a paid route The live smoke asserted the candidate in one shape only, an instance upgraded onto a persisted root. Nothing covered a brand new instance, which is what a customer provisioned after a release actually gets. Add a third leg that runs the same 29 assertions against the candidate on a fresh root, and assert each leg's root shape before boot so a leg that is not in its claimed shape fails loudly instead of silently duplicating another leg. Demote the baseline leg to seeding that root, with no live turn. It asserts the version being replaced, proves nothing about the candidate, and was the source of repeated failures that were mistaken for candidate regressions. Run on a paid route by default (kilocode/kilo-auto/balanced, what production instances use) and reject free routes. Verified against a live instance: openclaw 2026.6.11 on kilocode/kilo-auto/free fails every turn with 'provider rejected the request schema or tool payload' while the same image on a paid route reaches the provider normally, so a free-route run tests provider free-tier behaviour rather than the image. Tag a 402 as NO_CREDITS and report it as a billing state. Boot containers with AUTO_APPROVE_DEVICES=true, which every production instance sets unconditionally, and settle before the live turn to clear the device warm-up window that local Docker exposes and Fly does not.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryThe incremental commit resolves both previously reported findings — the mid-run Resolved since last review
Notes
Files Reviewed (2 files)
Previous Review Summary (commit 3e4b4dc)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 3e4b4dc)Status: 2 Issues Found | Recommendation: Address before merge Executive SummaryThe new fresh-root leg deletes a container-root-owned bind mount with a bare Overview
Issue Details (click to expand)WARNING
SUGGESTION
Notes
Files Reviewed (4 files)
Reviewed by claude-opus-5 · Input: 26 · Output: 4.6K · Cached: 604K Review guidance: REVIEW.md from base branch |
The fresh-root leg deleted the baseline's bind mount with a bare rm -rf under set -euo pipefail. The container writes /root as uid 0, so on native Linux Docker that tree is root-owned on the host and the delete fails with EACCES for a non-root user, aborting the run after two legs had passed and before the results summary. Allocate a second root instead and remove every allocated root at EXIT, where failures cannot abort the run. Add a default branch to assert_root_shape so an unrecognized shape argument fails loudly instead of falling through and asserting nothing, which was the silent no-coverage the function exists to prevent. Update DEVELOPMENT.md, which still documented the live smoke as running on kilocode/kilo-auto/free.
jeanduplessis
approved these changes
Jul 30, 2026
St0rmz1
added a commit
that referenced
this pull request
Jul 30, 2026
Brings in the reworked live smoke (#4901): paid-route default, new-install and upgraded shapes, root-shape assertions, and the AUTO_APPROVE_DEVICES production parity fix. Lockfile conflict resolved by regenerating against the merged package.json files rather than hand-merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The live smoke asserted the candidate image in one shape only, an instance that
upgraded onto a persisted root. Nothing covered a brand new instance, which is
what a customer provisioned after a release actually gets. It also ran on a free
model route and booted a container configuration that production never uses,
which produced failures that looked like image regressions but were not.
This adds the missing shape, moves the run onto a paid route, and closes two
gaps between the local Docker harness and how instances actually run on Fly.
New install coverage. The upgrade run now asserts the candidate twice: once
on the baseline's persisted root (an upgrading instance) and once on a brand new
root (a new signup). Both legs run the same 29 assertions, so neither is a token
pass.
Each leg proves its own shape.
assert_root_shapechecks the root is emptyor seeded as expected before the container boots. Without it, a silent failure of
the fresh root swap would turn the new instance leg into a second upgrade run
that still reported green while covering nothing new.
The baseline leg no longer runs a live turn. It exists to generate the
persisted root. A turn there asserts the version being replaced, proves nothing
about the candidate, and was the source of repeated failures that were mistaken
for candidate regressions.
Paid route by default.
kilocode/kilo-auto/balanced, matching whatproduction instances use. Free routes are rejected, with
ALLOW_FREE_SMOKE_MODEL=trueas a deliberate override. This was confirmed against a live instance: openclaw
2026.6.11 on
kilocode/kilo-auto/freefails every turn with "provider rejectedthe request schema or tool payload", while the same image on a paid route reaches
the provider normally. A free route run therefore tests provider free tier
behavior rather than the image under test.
Production parity in the container. Containers now boot with
AUTO_APPROVE_DEVICES=true, which every real instance sets unconditionally inservices/kiloclaw/src/gateway/env.tsas a reserved system var. Without it thesmoke booted a configuration that does not exist in production: the controller's
gateway client device auto approval was disabled and
openclaw.jsonnever gotgateway.controlUi.allowInsecureAuth.Warm up handling. A settle before the live turn, with a short opportunistic
poll for an already approved device. On Fly a device is granted
operator.writewithin seconds. In local Docker a turn landing too early dies with
1008 pairing required, which reads as a product failure but is warm up.Billing is named as billing. A 402 is tagged
NO_CREDITSand reported as abilling state with the fix, rather than leaving a credits problem looking like a
regression.
Verification
Run locally against prebuilt images. Nothing here runs in CI.
legs green, including both live turns and all three root shape assertions.
Phase 1 keyless 17 passed, 0 failed, Phase 2 smoke 72 passed, 0 failed.
KILOCODE_SMOKE_MODEL=kilocode/kilo-auto/freerefuses before any image is built.
report
empty, the upgraded leg reportsseeded.Visual Changes
N/A
Reviewer Notes
The device scope poll in
wait_for_device_scopesusually cannot succeed, and thecomment says so. Verified against a live instance:
/root/.openclaw/devicesdoesnot exist before the first client connects, and
paired.jsonis created by thatconnection, so there is normally no signal to wait for and the settle is what does
the work. The poll is kept short for the case where a previous leg on the same
persisted root already left an approved device. It is never fatal, because absence
is the normal state.
Run time goes up. There is one more container boot and live turn, plus the settle
before each turn.
DEVICE_SCOPE_POLL_SECSandDEVICE_SCOPE_SETTLE_SECSareoverridable if that becomes annoying.
Rejecting free routes is a behavior change for anyone who currently runs the smoke
with a free key. The error names the override.
openclaw-upgrade-image-checks.shhere carries documentation edits only. TheactionRequiresTargetcheck fix lives on the 2026.7.1 bump branch and isdeliberately not duplicated here.