Skip to content

cmux-host: report the machine's workload so a Cloud idle pause never freezes a dev server, build or cron job - #36511

Merged
teamleaderleo merged 4 commits into
feat-cmux-nextfrom
feat/cloud-idle-busy-signals
Nov 5, 2026
Merged

teamleaderleo merged 4 commits into
feat-cmux-nextfrom
feat/cloud-idle-busy-signals

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Leo, on Cloud work: "If I tell you to do something, don't stick it into dev. Put it into prod. There's nobody else using prod."

The Cloud's idle decision looked only at input and running agent turns. A machine with a dev server, a build or a cron job an agent left running read as idle, and an idle pause would freeze it. This PR makes the guest report what it runs, so the platform can call a machine idle only when nothing is running (the platform side follows in manaflow-ai/cmux-platform).

What changed

cmux-host (the Cloud agent, cloud/workload.rs) probes the machine every 30 s and adds activity.workload to cloud.vm.status.report, with the daemon capability activity-workload while the probe reads:

  • busy_processes: processes above 2% of one CPU between two probes. Reaped children count toward their parent, so a make whose compilers each live a second still reads busy. A process new since the last probe counts all its ticks; pid reuse is caught by the start time.
  • listening_ports: TCP sockets in LISTEN (IPv4 and IPv6, loopback included). A socket with no owner found counts.
  • timers: user crontabs, /etc/cron.d entries, at jobs, transient (systemd-run) timers, and timer units under /etc/systemd/system or a user's ~/.config/systemd/user (masked units skipped).
  • Each count carries at most 8 names: a process's 15-character comm, a port number, or a timer name. Never a command line, so no secrets travel. last_busy_at is the last probe that found the machine busy, so the idle period starts no earlier.
  • System work never counts, by documented allowlists (SYSTEM_PROCESSES, SYSTEM_PROCESS_PREFIXES, SYSTEM_TIMERS, SYSTEM_CRON_D): kernel threads, systemd and its daemons, sshd, cron and atd, journald, dbus, cmux-host, the terminal daemon and its hosts, acpmux, machine health, the image's desktop and the distribution's housekeeping timers.
  • A probe that cannot read /proc makes the workload unknown. The report then drops activity-workload, and the platform treats the machine as busy.
  • A report goes out when the machine turns busy or quiet, when its ports or timers change, or when a quiet period starts again. A change in which process is using CPU while the machine is busy waits for the next report, so a busy machine does not report on every probe.

The block is an extra key in activity (whose schema allows additional properties), like phases and last_person_prompt_at, so the op catalog and the generated clients are unchanged.

Changelog

  • Fixed: a Cloud machine running a dev server, a build or a scheduled job is no longer read as idle.

Tests

cargo test -p cmux-host --lib cloud:: (fleet): new workload_tests.rs (stat and /proc/net/tcp parsing, the CPU threshold, pid reuse, the allowlists, the timers reader under a fake root with crontabs, systemd units, masked units and at jobs) and a session.rs test for the report and capability rules.

Leaned on

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Stops the Cloud's idle pause from freezing a machine that's actually working: cmux-host now probes the machine every 30 s and reports busy processes, listening ports, and user timers as activity.workload under the activity-workload capability, so a dev server, build, or cron job keeps the machine awake.

  • Busy processes are those using more than 2% of one CPU between probes (reaped children count toward their parent), plus TCP LISTEN sockets and user timers (cron, cron.d, at, systemd timers, including systemd-run --user). System work is excluded via documented allowlists.
  • Only names are reported (process comm, port number, timer name), never command lines; anything unreadable in /proc — a listener table or a process stat — marks the workload unknown and drops the capability, so the platform treats the machine as busy. Only an exited process or a missing IPv6 table is skipped; IPv4's listener table is required.
  • last_busy_at is monotonic, so a stepped-back clock never lengthens the idle period. Reports go out when the machine turns busy or quiet or when ports or timers change, not on every probe.

Written for commit ee4996d. Summary will update on new commits.

View guided diff Turn on auto-fix

… the Cloud idle decision

The Cloud agent probes every 30 s and adds activity.workload to
cloud.vm.status.report under the capability activity-workload: processes
above 2% of one CPU, TCP listeners and user timers (cron, cron.d, at,
systemd), excluding a documented allowlist of system processes and timers.
Names only (comm, port, timer name), never command lines. A probe that
cannot read /proc drops the capability, so the Cloud reads the machine busy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
@teamleaderleo
teamleaderleo temporarily deployed to cmux-next-cloud-journey November 5, 2026 11:53 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 5b54546: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr36511 -f ref=feat/cloud-idle-busy-signals -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 5, 2026
…sient timers

Review F1/F2 on #36511: a listener table or a process stat that exists but
does not read now makes the workload unknown (only an exited process or a
missing IPv6 table is skipped), and `systemd-run --user` timers under
/run/user/<uid>/systemd/transient count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
@blacksmith-sh

blacksmith-sh Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Found 11 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp057ib58n/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpopy3pzff/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs

...and 1 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 6efd5b2

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 3. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 6efd5b2

Visual verdict: no images; Leo tries the build

  1. [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] read_snapshot treats a missing /proc/net/tcp the same as an optional missing tcp6 table. If IPv4 listener data is unavailable while tcp6 remains readable, an active IPv4 server can be reported absent and the VM may be paused. Only missing tcp6 should be tolerated (workload.rs:350).

  2. [P1] [follow-up] [category:reliability] [finding:F2] [outcome:pending] PID enumeration silently drops ReadDir errors via e.ok()?, allowing unreadable /proc entries to be omitted while still producing a “known” snapshot. Propagate iterator errors as unknown and skip only non-PID directory names (workload.rs:331).

  3. [P2] [follow-up] [category:reliability] [finding:F3] [outcome:pending] last_busy_at is overwritten with the current wall clock on every busy probe, so a backward realtime correction can move the timestamp earlier and make the idle period appear longer than it is. Preserve the maximum observed timestamp (workload.rs:309).

Follow-ups:

  • [finding:F1] Require only tcp6 to be optional.
  • [finding:F2] Treat /proc directory iteration errors as unknown.
  • [finding:F3] Keep last_busy_at monotonic across wall-clock corrections.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 6efd5b2: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

…onotonic last_busy_at

- to_json inserts last_busy_at without indexing (clippy::indexing_slicing,
  cmux-host is at deny).
- Review follow-ups on #36511: /proc/net/tcp is required (only tcp6 may be
  absent), a /proc entry that does not read makes the workload unknown, and
  last_busy_at never moves earlier when the clock steps back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
@cursor

cursor Bot commented Nov 5, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at ee4996d

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 3. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at ee4996d

Visual verdict: no images; Leo tries the build

The requested cargo test could not run because cargo is unavailable in this box.

  • [P2] [follow-up] [category:correctness] [finding:F1] [outcome:pending] cmux-tui/crates/cmux-host/src/cloud/session.rs:253-256 returns before storing a newer workload when its key is unchanged. A busy node sample followed by a busy cargo sample therefore leaves stale process names and counts in the reporter, so later reports do not carry the latest probe as promised.
  • [P2] [follow-up] [category:performance] [finding:F2] [outcome:pending] cmux-tui/crates/cmux-host/src/cloud/workload.rs:239-242 defines the key’s busy bit from busy_processes alone. With a persistent listener or timer, crossing the CPU threshold on and off changes the key and emits reports even though the machine remains busy, contrary to the stated suppression of per-probe busy reports.
  • [P2] [follow-up] [category:correctness] [finding:F3] [outcome:pending] cmux-tui/crates/cmux-host/src/cloud/workload.rs:282-293 stores busy processes in a BTreeSet<String>, while to_json reports its length as the process count at lines 249-253. Multiple busy processes with the same comm are therefore reported as one process.

Follow-ups:

  • F1 — retain the latest workload payload even when no immediate report is scheduled.
  • F2 — base the busy key on the full workload busy state.
  • F3 — count busy process instances separately from the deduplicated display names.

@teamleaderleo
teamleaderleo merged commit e514b42 into feat-cmux-next Nov 5, 2026
215 of 217 checks passed
@teamleaderleo
teamleaderleo deleted the feat/cloud-idle-busy-signals branch November 5, 2026 14:47

This branch had an error being deployed

1 failed and 3 inactive deployments
ci-routing-agent — ee4996dd Deployed Nov 5, 2026 by github-actions[bot] via cmux-next macOS placement #73246
artifacts — ee4996dd Deployed Nov 5, 2026 by teamleaderleo via publish tree-addressed cmux-tui #116496
sentry-dsyms — ee4996dd Deployed Nov 5, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #116496
cmux-next-cloud-journey — ee4996dd Deployed Nov 5, 2026 by teamleaderleo via journey #53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant