Repository navigation
Kanban workers boot on store-Python installs; unknown assignee, gh identity and dead-credential exits fixed (#122299 #122487 #122500 #122422 #122689, salvage #122420 #122567 #122714 #122716) - #126153
Merged
Conversation
teknium1
force-pushed
the
fix/s6-kanban-workers
branch
from
September 28, 2026 08:26
3271025 to
afdefdd
Compare
૮ >ﻌ< ა ci reviewran on 37beb48 — fix(review): PR acceptance never falls through to the launch
|
The dispatcher's _resolve_hermes_argv proves hermes_cli importable in the gateway process, where a store-python shim has the repo root on sys.path in-process; the spawned bare 'sys.executable -m hermes_cli.main' child runs from the task workspace with a scrubbed PYTHONPATH and cannot import the package, dies before any work, and the board auto-blocks (#122299). Keep the module-over-PATH ordering (da18c20, #111569) and instead propagate the running install's package root onto the worker's PYTHONPATH; a resolved shim path keeps owning its own imports. Fixes #122299
#122299, #122487, #122500, salvage #122420) _resolve_hermes_argv proves hermes_cli importable in the GATEWAY, where the store-python shim / launcher bootstrap put the repo root on sys.path in-process. The worker env scrub (build_subprocess_env / served_profile_child_env) strips Hermes-owned PYTHONPATH entries, so the bare `sys.executable -m hermes_cli.main` child could not import the package the parent just proved importable and every worker died at birth (ModuleNotFoundError: hermes_cli) -> crashed x2 -> gave_up. Reuse cron's pin_hermes_tree_on_pythonpath (#112729) on the module argv only: a resolved shim path owns its imports. hermes_cli.main's bootstrap then activates the committed PM dependency generation as usual, so a store Python with an empty site-packages still boots. Same class in the Bot Chat delivery runner (cron/scheduler_delivery.py), which builds the module argv after served_profile_child_env.
Cards assigned to a nonexistent profile landed in the aggregate skipped_nonspawnable bucket with no per-task event, so the card sat in ready forever with zero board diagnostics (show/tail silent; the bucket prints only on manual dispatch). Follow the respawn_guarded pattern: append a skipped_nonspawnable event naming the assignee on every non-dry-run skip. Closes #122422
…nce per tick (#122422, salvage #122567) The per-task `skipped_nonspawnable` event names the missing profile so `hermes kanban show`/`tail` explain why a card sits in `ready`. Unlike a respawn guard the condition never expires, so the event is written only when the card's latest event is not already this skip: one row per card, not one per dispatcher tick forever, and not one per foreign home per tick on a shared board (#101015).
collect_acceptance's gh api subprocess inherited the calling process's environment, so on a multi-profile host the gate read the contract repo as the ambient (launch/default) gh login: a private repo in another org returned 404 and the broad except classified it infra with 'check gh authentication and retry', blocking green cards and holding them via the blocker_auth respawn guard (#122689). - Thread the assignee's profile home into every _api call and build the child env with served_profile_child_env(inherit_credentials=True): the profile's own GH_TOKEN/GH_CONFIG_DIR overlay, launch credential residue scrubbed (GH_CONFIG_DIR is a path, not a credential, so no scrub list saw it — drop it from the base for routed targets). - Classify a refused read (gh HTTP 401/403/404, or GraphQL resolving the repository to null) as 'auth' naming the repository, distinct from retryable infra; persist only the status code + endpoint, never stderr. - Unassigned/uninstalled cards and single-profile hosts keep the ambient env, unchanged. Fixes #122689
…h login Review findings on the assignee-login gate (#122689): - MAJOR: served_profile_child_env(inherit_credentials=True) overlays only the assignee's own GH_TOKEN/GH_CONFIG_DIR but HOME/XDG_CONFIG_HOME stay the launch process's, so a profile with no login of its own fell through to ~/.config/gh/hosts.yml - the ambient login the gate promises never to use. For a routed assignee home with neither token nor config dir, pin GH_CONFIG_DIR to <profile_home>/gh; gh then exits 4 (authentication required), which _api classifies as auth naming the profile. - MINOR (a): an ASSIGNED card whose profile cannot be resolved was fail-open (env=None -> completing process's full ambient login). Resolution now happens inside collect_acceptance and raises _GateAuthError naming the profile; only genuinely unassigned cards keep the ambient path. - MINOR (c): Codex refresh 200/non-JSON body no longer sets relogin_required, so the new exit-78 startup gate cannot turn an edge misfire into a sticky terminal block (invalid_json_relogin=False, as xAI already does). - MINOR (b)+(d) docs: GH_TOKEN must live in the profile's .env/secret source (a shell/systemd export is scrubbed); skipped_nonspawnable {assignee} row in the worker telemetry table; startup relogin-required failure exits 78.
teknium1
force-pushed
the
fix/s6-kanban-workers
branch
from
September 28, 2026 09:34
afdefdd to
37beb48
Compare
11 of 13 tasks
6 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kanban workers boot on store-Python / PM installs, unknown assignees leave a board trace, PR acceptance reads GitHub as the card's assignee profile, and a dead credential at worker startup exits terminal instead of burning the retry budget.
Per issue
hermes_cli/kanban_db_dispatch.py::_default_spawnnow pins the running install's root on the worker env'sPYTHONPATHwhenever the module argv (sys.executable -m hermes_cli.main) is chosen, via cron's existingpin_hermes_tree_on_pythonpath([Bug]: Restart-safe external cron worker fails on dispatch due to symlink resolution in sys.executable #112729). Root cause in one sentence:_resolve_hermes_argv'sfind_spec("hermes_cli")answers a question about the GATEWAY (where the launcher bootstrap injected the repo root intosys.path), while the worker env scrub strips Hermes-ownedPYTHONPATHentries from the CHILD, so the bare interpreter died withModuleNotFoundError: hermes_cli.hermes_cli.main→hermes_bootstrap→activate_dependenciesthen selects the committed PM dependency generation in the child, so a store Python with an emptysite-packagesboots too. A resolved shim path ($HERMES_BIN,which hermes) owns its imports and is left alone. Same class fixed in the Bot Chat delivery runner (cron/scheduler_delivery.py, module argv afterserved_profile_child_env). Salvages fix(kanban): carry the module argv's import root to the worker env #122420 by @liuhao1024 (earliest; fix(kanban): spawn workers through the installation-bound launcher bootstrap #125088 by @intellectronica reaches the same outcome by switching the argv to-I -c <bootstrap>and fix(launch): publish launchers on a runtime that can spawn a child (kanban/cron workers) #123041 by @strokesws by republishing launchers — both superseded by the smaller pin, which keeps the argv shapegateway/status.py's cmdline matchers recognise).readyforever with zero board diagnostics #122422 —_dispatch_lane_taskwrites a per-taskskipped_nonspawnableevent{"assignee": <name>}when the assignee profile does not exist, sohermes kanban show/tailname the missing profile. Written once per card (only when the card's latest event is not already this skip), because unlike a respawn guard the condition never expires: one row per tick forever, or one per foreign home per tick on a shared board (Kanban dispatch requires assignee profiles to exist locally: one-agent-per-container fleets sharing a board silently drop every cross-agent card #101015), would have been the noise. Salvages fix(kanban): write per-task event when dispatcher skips unknown assignee #122567 by @Finn763.kanban.dispatch_profiles, Kanban dispatcher claims across homes on a shared board: profile_exists("default") is unconditionally True, so every home can claim default-assigned cards #110995/fix(kanban): per-home dispatch claim allowlist for shared boards (#110995) #111004) rather than zero-config identity mapping; not changed here.hermes_cli/kanban_pr_acceptance.py::_apirunsgh apiunderserved_profile_child_env(target_home=<assignee home>, inherit_credentials=True): the assignee profile's ownGH_TOKEN/GH_CONFIG_DIR(its.env+ external sources), launch residue scrubbed. A 401/403/404 or anullGraphQL repository is classifiedauthnaming the repository instead of a retryableinfra. The assignee home is resolved exactly as the dispatcher resolves the worker's home (resolve_profile_env). Salvages fix(kanban): PR acceptance runs gh as the assignee profile's login #122714 by @dskwe (trimmed: the second commit's operation×status refusal ladder is not taken).relogin_required(revoked key, model access disabled) now exitsKANBAN_TERMINAL_PROVIDER_EXIT_CODE(78) so the dispatcher blocks the card at once instead of charging the wholefailure_limit; unknown startup failures keep exit 1; a turn result still takes precedence. Salvages fix(kanban): preserve credential startup failure exit codes #122716 by @Wenfengcheng. The pre-claim model probe half (fix(kanban): probe model/provider pins before the worker lifecycle is spent #122893 by @strzhao, +735) is a new capability with a per-tick network cost and is left for a decision — not relabelled away, seenot_coveredin the lane report.WHEN × WHERE (profile-scope table)
build_subprocess_env/served_profile_child_env, neveros.environ.copy(); the pin edits only the built child env.ghacceptance child —served_profile_child_env(target_home=B); B-only.envGH_TOKENresolves, launchGH_TOKEN/GH_CONFIG_DIRresidue absent.kanban.db.Validation
tests/hermes_cli/test_kanban_db.py::test_default_spawn_pins_repo_root_on_module_worker_pythonpath(FakePopen; module argv →PYTHONPATH[0]== install root; shim argv → untouched)KeyError: 'PYTHONPATH')_resolve_hermes_argvtests)tests/hermes_cli/test_kanban_dispatch_unknown_assignee_event.py(event named after one tick; exactly one after three ticks; dry-run writes none)tests/hermes_cli/test_kanban_pr_acceptance.py::test_acceptance_runs_gh_as_the_assignee_profile(ambientGH_TOKEN=launch-token+GH_CONFIG_DIRresidue vs B's.envGH_TOKEN=b-token; fakeghdumps its env and returnsrepository: null)tests/hermes_cli/test_kanban_credential_startup_exit.py(revoked→78 / quota→75 / unknown→1 × quiet × worker; turn result precedence)ruff checkclean on touched files;check_no_tmp_literals.pyclean;audit_pr_attribution.py --fix: all contributor emails mapped;git diff --checkclean. Subset suite (kanban + single-query + cron bot-chat delivery files) run result is recorded in the lane report.Infographic
DEGRADED —
image_generaterefused (fal.ai balance exhausted: "User is locked. Reason: Exhausted balance"); style picked washolo-tcg. No image embedded.Review fixes
kanban_pr_acceptance.py::_gh_env(assignee with no ownGH_TOKEN/GH_CONFIG_DIRfell through to the launch user's~/.config/ghbecause HOME/XDG stay the launch process's) → for a routed assignee home whose child env has none ofGH_TOKEN/GITHUB_TOKEN/GH_CONFIG_DIR,GH_CONFIG_DIRis pinned to<profile_home>/gh; gh exits 4 ("authentication required"), which_apinow classifiesauthnaming the profile. PoppingXDG_CONFIG_HOMEwas not needed: gh consultsGH_CONFIG_DIRfirst. Test:test_kanban_pr_acceptance.py::test_assignee_without_own_gh_login_never_falls_through_to_ambient_login(red → green)._assignee_profile_homefail-open → profile resolution moved insidecollect_acceptance(assignee=replacesprofile_home=); an assigned card whose profile cannot be resolved raises_GateAuthError→classification=authnaming the profile, gh never spawned. Only unassigned cards keep the ambient env. Test:test_assigned_card_with_unresolvable_profile_is_auth_not_ambient.kanban.md:GH_TOKENmust live in the profile's.env/secret source; a shell/systemd export is scrubbed and never re-added.invalid_json_relogin=Falseinhermes_cli/auth_codex.py(same as xAI), so the exit-78 startup gate cannot turn an edge misfire into a sticky terminal block. The 401–403 edge path (_codex_refresh_failure_error) is unchanged.kanban.mdworker telemetry table gains theskipped_nonspawnable {assignee}row; thegave_uprow notes that a startup relogin-required credential failure exits78(other startup failures1).