Skip to content

Kanban workers boot on store-Python installs; unknown assignee, gh identity and dead-credential exits fixed (#122299 #122487 #122500 #122422 #122689, salvage #122420 #122567 #122714 #122716) - #126153

Merged
teknium1 merged 7 commits into
mainfrom
fix/s6-kanban-workers
Sep 28, 2026
Merged

teknium1 merged 7 commits into
mainfrom
fix/s6-kanban-workers

Conversation

@teknium1

@teknium1 teknium1 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Kanban workers boot on store-Python / PM installs, unknown assignees leave a board trace, PR acceptance reads GitHub as the card's assignee profile, and a dead credential at worker startup exits terminal instead of burning the retry budget.

Per issue

WHEN × WHERE (profile-scope table)

  • L5 (child processes) × T1/T2/T3/T4/T6: kanban worker + Bot Chat delivery runner — both spawn through build_subprocess_env/served_profile_child_env, never os.environ.copy(); the pin edits only the built child env.
  • L5 × T2/T3/T7: gh acceptance child — served_profile_child_env(target_home=B); B-only .env GH_TOKEN resolves, launch GH_TOKEN/GH_CONFIG_DIR residue absent.
  • L3 (dispatcher ticker) × T2/T10: the skip event is idempotent across ticks and across homes sharing one kanban.db.

Validation

Item Test origin/main this head
#122299/#122487/#122500 tests/hermes_cli/test_kanban_db.py::test_default_spawn_pins_repo_root_on_module_worker_pythonpath (FakePopen; module argv → PYTHONPATH[0] == install root; shim argv → untouched) 1 failed (KeyError: 'PYTHONPATH') 3 passed (with the two existing _resolve_hermes_argv tests)
#122422 / #101015 tests/hermes_cli/test_kanban_dispatch_unknown_assignee_event.py (event named after one tick; exactly one after three ticks; dry-run writes none) 2 failed 2 passed
#122689 tests/hermes_cli/test_kanban_pr_acceptance.py::test_acceptance_runs_gh_as_the_assignee_profile (ambient GH_TOKEN=launch-token + GH_CONFIG_DIR residue vs B's .env GH_TOKEN=b-token; fake gh dumps its env and returns repository: null) 1 failed 1 passed (file: 3 passed)
#122703 tests/hermes_cli/test_kanban_credential_startup_exit.py (revoked→78 / quota→75 / unknown→1 × quiet × worker; turn result precedence) 6 failed 15 passed

ruff check clean on touched files; check_no_tmp_literals.py clean; audit_pr_attribution.py --fix: all contributor emails mapped; git diff --check clean. Subset suite (kanban + single-query + cron bot-chat delivery files) run result is recorded in the lane report.

Infographic

DEGRADED — image_generate refused (fal.ai balance exhausted: "User is locked. Reason: Exhausted balance"); style picked was holo-tcg. No image embedded.

Review fixes

  • MAJOR kanban_pr_acceptance.py::_gh_env (assignee with no own GH_TOKEN/GH_CONFIG_DIR fell through to the launch user's ~/.config/gh because HOME/XDG stay the launch process's) → for a routed assignee home whose child env has none of GH_TOKEN/GITHUB_TOKEN/GH_CONFIG_DIR, GH_CONFIG_DIR is pinned to <profile_home>/gh; gh exits 4 ("authentication required"), which _api now classifies auth naming the profile. Popping XDG_CONFIG_HOME was not needed: gh consults GH_CONFIG_DIR first. Test: test_kanban_pr_acceptance.py::test_assignee_without_own_gh_login_never_falls_through_to_ambient_login (red → green).
  • MINOR (a) _assignee_profile_home fail-open → profile resolution moved inside collect_acceptance (assignee= replaces profile_home=); an assigned card whose profile cannot be resolved raises _GateAuthError → classification=auth naming the profile, gh never spawned. Only unassigned cards keep the ambient env. Test: test_assigned_card_with_unresolvable_profile_is_auth_not_ambient.
  • MINOR (b) docs → kanban.md: GH_TOKEN must live in the profile's .env/secret source; a shell/systemd export is scrubbed and never re-added.
  • MINOR (c) Codex 200/non-JSON refresh body → invalid_json_relogin=False in hermes_cli/auth_codex.py (same as xAI), so the exit-78 startup gate cannot turn an edge misfire into a sticky terminal block. The 401–403 edge path (_codex_refresh_failure_error) is unchanged.
  • MINOR (d) docs → kanban.md worker telemetry table gains the skipped_nonspawnable {assignee} row; the gave_up row notes that a startup relogin-required credential failure exits 78 (other startup failures 1).

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins area/auth Authentication, OAuth, credential pools area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Sep 28, 2026
@teknium1
teknium1 force-pushed the fix/s6-kanban-workers branch from 3271025 to afdefdd Compare September 28, 2026 08:26
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

૮ >ﻌ< ა ci review

ran on 37beb48 — fix(review): PR acceptance never falls through to the launch

⚠️ Warnings

CI timings · View report · View job

Wall time 58m39s vs 1m29s (+3853.9%). 2 faster,

  • Detect affected areas: -52.0s
  • All required checks pass: -1.0s

liuhao1024 and others added 7 commits September 28, 2026 02:07
The dispatcher's _resolve_hermes_argv proves hermes_cli importable in the
gateway process, where a store-python shim has the repo root on sys.path
in-process; the spawned bare 'sys.executable -m hermes_cli.main' child
runs from the task workspace with a scrubbed PYTHONPATH and cannot import
the package, dies before any work, and the board auto-blocks (#122299).

Keep the module-over-PATH ordering (da18c20, #111569) and instead
propagate the running install's package root onto the worker's
PYTHONPATH; a resolved shim path keeps owning its own imports.

Fixes #122299
#122299, #122487, #122500, salvage #122420)

_resolve_hermes_argv proves hermes_cli importable in the GATEWAY, where the
store-python shim / launcher bootstrap put the repo root on sys.path in-process.
The worker env scrub (build_subprocess_env / served_profile_child_env) strips
Hermes-owned PYTHONPATH entries, so the bare `sys.executable -m hermes_cli.main`
child could not import the package the parent just proved importable and every
worker died at birth (ModuleNotFoundError: hermes_cli) -> crashed x2 -> gave_up.

Reuse cron's pin_hermes_tree_on_pythonpath (#112729) on the module argv only:
a resolved shim path owns its imports. hermes_cli.main's bootstrap then
activates the committed PM dependency generation as usual, so a store Python
with an empty site-packages still boots. Same class in the Bot Chat delivery
runner (cron/scheduler_delivery.py), which builds the module argv after
served_profile_child_env.
Cards assigned to a nonexistent profile landed in the aggregate
skipped_nonspawnable bucket with no per-task event, so the card sat in
ready forever with zero board diagnostics (show/tail silent; the bucket
prints only on manual dispatch). Follow the respawn_guarded pattern:
append a skipped_nonspawnable event naming the assignee on every
non-dry-run skip.

Closes #122422
…nce per tick (#122422, salvage #122567)

The per-task `skipped_nonspawnable` event names the missing profile so
`hermes kanban show`/`tail` explain why a card sits in `ready`. Unlike a
respawn guard the condition never expires, so the event is written only when
the card's latest event is not already this skip: one row per card, not one per
dispatcher tick forever, and not one per foreign home per tick on a shared board
(#101015).
collect_acceptance's gh api subprocess inherited the calling process's
environment, so on a multi-profile host the gate read the contract repo as
the ambient (launch/default) gh login: a private repo in another org
returned 404 and the broad except classified it infra with 'check gh
authentication and retry', blocking green cards and holding them via the
blocker_auth respawn guard (#122689).

- Thread the assignee's profile home into every _api call and build the
  child env with served_profile_child_env(inherit_credentials=True): the
  profile's own GH_TOKEN/GH_CONFIG_DIR overlay, launch credential residue
  scrubbed (GH_CONFIG_DIR is a path, not a credential, so no scrub list
  saw it — drop it from the base for routed targets).
- Classify a refused read (gh HTTP 401/403/404, or GraphQL resolving the
  repository to null) as 'auth' naming the repository, distinct from
  retryable infra; persist only the status code + endpoint, never stderr.
- Unassigned/uninstalled cards and single-profile hosts keep the ambient
  env, unchanged.

Fixes #122689
…h login

Review findings on the assignee-login gate (#122689):

- MAJOR: served_profile_child_env(inherit_credentials=True) overlays only the
  assignee's own GH_TOKEN/GH_CONFIG_DIR but HOME/XDG_CONFIG_HOME stay the
  launch process's, so a profile with no login of its own fell through to
  ~/.config/gh/hosts.yml - the ambient login the gate promises never to use.
  For a routed assignee home with neither token nor config dir, pin
  GH_CONFIG_DIR to <profile_home>/gh; gh then exits 4 (authentication
  required), which _api classifies as auth naming the profile.
- MINOR (a): an ASSIGNED card whose profile cannot be resolved was fail-open
  (env=None -> completing process's full ambient login). Resolution now
  happens inside collect_acceptance and raises _GateAuthError naming the
  profile; only genuinely unassigned cards keep the ambient path.
- MINOR (c): Codex refresh 200/non-JSON body no longer sets relogin_required,
  so the new exit-78 startup gate cannot turn an edge misfire into a sticky
  terminal block (invalid_json_relogin=False, as xAI already does).
- MINOR (b)+(d) docs: GH_TOKEN must live in the profile's .env/secret source
  (a shell/systemd export is scrubbed); skipped_nonspawnable {assignee} row
  in the worker telemetry table; startup relogin-required failure exits 78.
@teknium1
teknium1 force-pushed the fix/s6-kanban-workers branch from afdefdd to 37beb48 Compare September 28, 2026 09:34
@teknium1 teknium1 added the ci-reviewed applied to manually approve dangerous changes label Sep 28, 2026
@teknium1
teknium1 merged commit 37daf85 into main Sep 28, 2026
100 of 102 checks passed
@teknium1
teknium1 deleted the fix/s6-kanban-workers branch September 28, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment