fix(docker): probe the configured state dir before /workspace for UID/GID detection (#7027)

[jorgejiro]

Fixes #7027.

Implements the fix shape agreed in #7027 (comment).

Thinking Path

• Hermes WebUI's single-container Docker deploy is the setup users are pointed at first, so its startup path has to be forgiving of ordinary host layouts
• The container aligns its runtime user with the host by auto-detecting a UID/GID from mounted directories before usermod
• Detection is only as good as the paths it probes: a probe that resolves against something the image owns produces a confident answer with no host information in it
• The bug was that /workspace — present and owned 1024:1024 in every stock image — was probed while the configured state directory, which is a bind mount by definition, was not
• This PR probes the state directory first and stops treating an explicitly supplied 1024 as "unset"
• The result is that the documented single-container command comes up healthy on a host-owned state mount instead of restart-looping

What Changed

docker_init.bash

• The UID and GID probe loops now start with ${HERMES_WEBUI_STATE_DIR:-/app/data}, ahead of the hermes-home candidates.
• /workspace stays exactly where it was, as the priority-2 signal, for setups that genuinely bind-mount it. The hermes-home probes from Docker image: Permission denied writing to shared /opt/data volume when hermes-agent and hermes-webui run in separate containers #668 are untouched.
• Every detection branch is now gated on the value not having been supplied explicitly. Previously the guard was [ -z "${WANTED_UID+x}" ] || [ "${WANTED_UID}" = "1024" ], so the fallback sentinel and a legitimate UID were the same number and an operator's explicit 1024 was overwritten by detection.
• The origin of the value (explicit / detected) is persisted to $itdir next to the value itself. docker_init.bash runs twice — once as root, then exec su re-enters it as hermeswebui with the environment dropped — so without persisting the origin, the second pass reads 1024 back from the file and cannot tell an operator's explicit choice from the fallback. Re-detecting there would leave WANTED_UID disagreeing with the user the script is already running as, which is a hard startup failure. A detected 1024 stays re-detectable on later runs, which is what the sentinel was for.

tests/test_7027_state_dir_uid_probe.py (new) — 18 tests, described under Verification.

.github/workflows/docker-smoke.yml — new state-dir-uid job, described under Verification.

docs/docker.md — new troubleshooting entry #9 with the symptom, the cause, and the full probe order; #7027 added to Related issues.

Why It Matters

The failure is a restart loop with an error that names the consequence rather than the cause:

touch: cannot touch '/app/data/.testfile': Permission denied
!! ERROR: Failed to verify state directory at /app/data

Nothing in it points at UID detection, and because 1024 is also the fallback, the preceding Auto-detected workspace UID: 1024 line reads as though detection had found nothing — so the natural next step is to investigate why detection found nothing, when in fact it succeeded and picked the wrong source. The state directory is the one path that is a bind mount by definition in a single-container deploy, so its owner is always the host identity the container needs to match.

Verification

Behavioural tests — tests/test_7027_state_dir_uid_probe.py extracts the real UID/GID resolution block from docker_init.bash and runs it under bash with stat stubbed, so ownership can be simulated without root. This follows the existing pattern in tests/test_docker_env_readonly_vars.py; source-level assertions alone cannot catch ordering or shell-quoting regressions inside the block.

The reproduction test is bound to the shape pinned in the issue — host directory owned 1001:1001 bind-mounted as the state dir, stock /workspace at 1024:1024, no explicit IDs — rather than a fixture rebuilt from my reading of it. Coverage:

Scenario	Expected
non-1024 state mount + stock /workspace (the issue)	1001, detected from the state dir
macOS-style state mount, no /workspace	501:20
HERMES_WEBUI_STATE_DIR unset → default path	probed
explicit WANTED_UID=1024 + 1001-owned mounts	stays 1024
explicit 1024 across the su privilege drop	stays 1024
persisted fallback 1024, state volume appears later	re-detects 1001
explicit non-default (1500)	stays 1500
no state mount, /workspace owned 501	501 (#569 path intact)
hermes-home owned 1001 vs /workspace 1024	1001 (#668 path intact)
$HERMES_HOME probe	intact
root-owned state dir (fresh named volume)	skipped, falls through
nothing mounted	1024 fallback

Reverting docker_init.bash alone turns 10 of the 18 red, including both behavioural anchors (state_mount_beats_stock_workspace, explicit_1024_is_not_overwritten). The 8 that still pass without the fix are the ones pinning previously shipped behaviour — that is the intended split.

$ ./scripts/test.sh tests/test_7027_state_dir_uid_probe.py -q
18 passed

$ git stash -- docker_init.bash && ./scripts/test.sh tests/test_7027_state_dir_uid_probe.py -q
10 failed, 8 passed

Full suite — ./scripts/test.sh on Python 3.11, run twice: once on this branch, once on master with an otherwise identical environment. The failure sets are byte-identical (319 failing ids on both), so this branch introduces no regressions; it adds exactly 18 passes (12903 vs 12885).

branch: 312 failed, 12903 passed, 1444 skipped, 7 errors
master: 312 failed, 12885 passed, 1444 skipped, 7 errors
comm -23 branch_failures master_failures  ->  (empty)

Those 312 are environmental on my machine, not pre-existing repo breakage: 224 of them are FileNotFoundError: [Errno 2] No such file or directory: 'node' — I have no Node installed, and the JS-evaluating tests need it. CI has it. tests/test_compression_phantom_barrier.py was excluded from both runs because playwright is not in my local venv; it is unrelated to this change.

Docker-adjacent suites specifically (test_issue569_579.py, test_docker_docs_and_readonly.py, test_issue2237_docker_chown_git_objects.py, test_issue1908_docker_hardening.py, test_docker_env_readonly_vars.py, and the rest that read docker_init.bash): 108 passed.

scripts/ruff_lint.py clean, scripts/critical_markdown_check.py clean on docs/docker.md, bash -n docker_init.bash clean, and every run: block in the modified workflow parses under bash -n.

Startup health — the product gate named in the issue comment is /health, which no source-level invariant can prove, so it is a new state-dir-uid job in docker-smoke.yml rather than a pytest assertion. It boots a real container the way the issue does — HERMES_WEBUI_STATE_DIR bind-mounted from a host directory chowned to 1001:1001, no WANTED_UID, and deliberately no /workspace mount so the image's own 1024-owned /workspace is the competing signal — then requires /health to return 200, the log to name the state dir as the detection source, id -u hermeswebui to actually be 1001, and the startup log to be free of the error signatures. A second step covers the explicit-1024 half; it additionally mounts a 1001-owned hermes-home, because without that the pre-fix script would land on 1024 anyway via /workspace and the step would pass for the wrong reason.

What I could not verify myself: I have no Docker daemon on this machine, so that job has never executed on my side — it is verified by syntax only. Docker is owned by the runner, not by this repo's test suite, so please treat the CI run on this PR as the authoritative evidence for it, and I will iterate on the job if it comes back red.

Contract Routing

Task type: bug fix in container startup, with a public-doc update
Touched areas: docker_init.bash UID/GID resolution, docs/docker.md, Docker smoke workflow
Relevant public docs:

• AGENTS.md
• CONTRIBUTING.md
• docs/CONTRACTS.md
• docs/docker.md
  Scope boundaries: probe order and explicit-value handling only. No change to usermod/groupmod, the chown walk, the read-only-root branch, or any compose file.
  Evidence needed before claiming done: bash-level regressions for the reported shape plus the pre-existing fallbacks, and a real container reaching /health on a host-owned state mount.

Contract Change

• Previous: WANTED_UID/WANTED_GID equal to 1024 were treated as unset and re-derived by auto-detection, regardless of whether an operator had set them; detection ranked /workspace above the configured state directory.
• New: an explicitly supplied value always wins, including 1024; the configured state directory ranks first, /workspace remains the next signal.
• Affected docs and tests: docs/docker.md (new §9 documenting the full order), tests/test_7027_state_dir_uid_probe.py. tests/test_issue569_579.py still passes unmodified — the /workspace and hermes-home probes it pins are unchanged.
• Reason: 1024 was simultaneously the sentinel and a valid identity, and the pre-fix order let an image-owned path outrank a host-owned one. Deployments that set IDs explicitly, or that rely on /workspace/hermes-home detection, keep their current behaviour.

Risks / Follow-ups

• Deploys where the state dir and /workspace are owned by different non-root UIDs now follow the state dir instead of /workspace. That is the intended change, and it is the ownership that startup actually validates, but it is a behaviour change for anyone who was relying on the old precedence.
• $itdir gains two small marker files (hermeswebui_user_{uid,gid}_source). They are covered by the existing chown -R "$itdir" and written through write_privtmpfile, so they inherit the same 600 mode and ownership as the value files. A container carrying an old $itdir with no marker files reads them as absent and behaves exactly as before.
• Not addressed here: the state dir is probed before it is known to be writable, so a mounted-but-unwritable state dir still surfaces as the same "Failed to verify state directory" error, just with the right UID in it. That is a separate diagnostics improvement.
• Not addressed here either (raised by the Greptile review): the UID and GID loops are independent and each skips 0, so a probe owned 1001:0 can hand the UID to one directory and the GID to a later one, yielding a pair that matches neither mount. That behaviour predates this PR — the two loops have always been separate with the same != "0" guard — and this change extends both lists identically, so it adds a candidate rather than the divergence itself. Coupling the two loops would change detection semantics for the bug: workspace file browser shows empty when volume is mounted in Docker #569/Docker image: Permission denied writing to shared /opt/data volume when hermes-agent and hermes-webui run in separate containers #668 paths as well, which is past the scope agreed on the issue; happy to open a sibling issue for it, or to fold it in here if you would rather have it in one go.
• The new job reuses the same mutable action tags as the existing jobs in this workflow (actions/checkout@v4, docker/setup-buildx-action@v3, docker/build-push-action@v6). Also flagged by Greptile. I kept them consistent with the rest of the file rather than SHA-pinning only my steps; pinning the workflow is a worthwhile change but belongs in its own PR.

Release notes

Docker: fix UID/GID auto-detection selecting the image's 1024 owner (#7027) — single-container deploys with a bind-mounted HERMES_WEBUI_STATE_DIR owned by a non-1024 host user no longer restart-loop with Failed to verify state directory. The configured state directory is now probed before /workspace, whose owner in a stock image is a build-time artifact rather than a host signal. An explicitly supplied WANTED_UID/WANTED_GID of 1024 is also no longer overwritten by auto-detection.

Model Used

AI-assisted.

• Provider: Anthropic
• Model: Claude Opus 5 (1M context) — claude-opus-5[1m], via Claude Code
• Notable mode/tool use: the assistant read the repo's Docker init, test, and contract docs, wrote the patch and tests, and ran the suites listed under Verification. The Docker smoke job is the one piece it could not execute locally, as noted above.

[greptile-apps]

Greptile Summary

The PR changes Docker startup identity detection so the configured state directory is checked before image-owned paths and explicit UID/GID values, including 1024, survive privilege re-entry.

• Persists whether UID/GID values were explicit or detected.
• Adds state-directory probe-order and explicit-value regression coverage.
• Adds a real-container Docker smoke job and updates troubleshooting documentation.

Confidence Score: 4/5

The PR appears safe to merge, with non-blocking follow-ups for coherent UID/GID source selection and immutable workflow action pins.

The primary state-directory precedence fix is covered across shell-level and container-level scenarios; remaining concerns require an asymmetric mount-ownership layout or an upstream action-tag compromise.

Files Needing Attention: docker_init.bash and .github/workflows/docker-smoke.yml

Security Review

The new smoke job uses mutable action tags; its read-only token and lack of secrets limit impact, but immutable commit pins would prevent upstream tag movement from changing executed CI code.

Important Files Changed

Filename	Overview
docker_init.bash	Prioritizes the state directory and persists value origins correctly, but independent UID/GID probing can combine owners from different mounts.
tests/test_7027_state_dir_uid_probe.py	Adds focused source and behavioral tests for probe precedence, explicit 1024 handling, privilege re-entry, and existing fallbacks.
.github/workflows/docker-smoke.yml	Adds end-to-end startup gates for state-dir detection and explicit 1024, while introducing additional mutable action references.
docs/docker.md	Documents the failure symptom, cause, corrected probe order, explicit-value behavior, and workaround for older images.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Root docker_init] --> B{WANTED UID/GID supplied?}
  B -->|Yes| C[Mark explicit]
  B -->|No| D[Read persisted values and sources]
  D --> E{Detection allowed?}
  E -->|Yes| F[Probe state dir]
  F --> G[Probe hermes-home candidates]
  G --> H[Probe workspace]
  H --> I[Fallback 1024]
  C --> J[Persist values and sources]
  F --> J
  G --> J
  H --> J
  I --> J
  J --> K[Apply groupmod and usermod]
  K --> L[Re-enter init as hermeswebui]
  L --> D

Loading

Comments Outside Diff (1)

1. docker_init.bash, line 122-130 (link)

   UID/GID probes can diverge

   If a separately mounted state directory has a non-root UID but root GID, the UID loop selects that directory while the GID loop skips it and selects a later probe. This creates an artificial UID:GID pair matching neither mount, producing incorrect host-visible ownership and potentially inaccessible shared-home files.

   Knowledge Base Used: Bootstrap and deployment

_{Reviews (1): Last reviewed commit: "fix(docker): probe the configured state ..." | Re-trigger Greptile}

[greptile-apps]

Workflow actions use mutable tags

The added job executes actions through mutable major-version tags, so an upstream tag change alters trusted CI code without a repository change. Pinning these references to full commit SHAs prevents tag movement from affecting the Docker build; this job's read-only token and lack of secrets limit the current exposure.

How this was verified: The new job references @v4, @v3, and @v6 while the workflow grants only contents: read and does not pass secrets.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[jorgejiro]

Thanks — two notes on the automated review, both now also reflected in the PR description.

UID/GID probes can diverge (P2). Real, but it predates this PR rather than being introduced by it: on master the UID and GID loops are already independent and each already skips 0, so a probe owned 1001:0 could always hand the UID to one directory and the GID to a later one. This change extends both probe lists identically, so it adds one more candidate to a divergence that the design already allowed. Coupling the two loops would change detection semantics for the #569 / #668 paths too, which is beyond the scope agreed in #7027 (comment) — I left it out deliberately rather than by oversight. Happy to open a sibling issue, or to fold it into this PR if you would rather have it in one go.

Mutable action tags. The new job reuses the same tags as the existing jobs in this workflow (actions/checkout@v4, docker/setup-buildx-action@v3, docker/build-push-action@v6). I kept it consistent with the rest of the file rather than SHA-pinning only my steps — pinning the whole workflow seems worth doing, but as its own change.

One thing I could not verify myself: the state-dir-uid job has never executed, since the Actions runs on this PR are sitting at action_required (fork PR) and I have no Docker daemon locally. Whenever you approve the workflows, that job is the authoritative check on the startup-health gate, and I will fix it promptly if it comes back red.

[nesquena-hermes]

🔬 Exact-head gate — SAFE TO SHIP (queued for maintainer merge)

Thanks @jorgejiro — well-reasoned fix for #7027. The entrypoint couldn't distinguish an operator's explicit WANTED_UID=1024 from "unset" (1024 is also the built-in default), so auto-detection would overwrite a deliberate choice; and /workspace (which ships owned by the build-time 1024) was probed too early to be a meaningful host-identity signal.

Full gate — SAFE TO SHIP (head ae05dc52):

• Explicit choice preserved across the su re-entry: Codex confirmed both the UID and GID auto-detect branches honor the persisted _source=explicit marker, and the marker files survive the root→runtime-user re-entry (the core docker: UID auto-detection probes /workspace before the configured HERMES_WEBUI_STATE_DIR, selecting the image's 1024 owner #7027 fix). Every auto-detect block is correctly gated.
• No regression: state-dir → hermes-home → workspace precedence behaves as documented; two-container hermes-home detection (Docker image: Permission denied writing to shared /opt/data volume when hermes-agent and hermes-webui run in separate containers #668) still works when HERMES_WEBUI_STATE_DIR is unset. Missing legacy marker files fall back safely; default stays 1024 (backward-compat with older images). GID logic mirrors UID.
• Real-container proof: the added docker-smoke state-dir job and all existing Docker smoke variants build and run the container green — the end-to-end proof a Docker entrypoint change needs. bash -n clean, 59 focused Docker/init tests + the new 18-case suite pass (they execute the real bash resolution block, covering the two-pass explicit-1024 scenario), full suite 14,597 passed.

Backend/entrypoint only, no visible surface, ships with a substantive test + extended CI smoke coverage. Queuing gate-pass for a maintainer merge nod (the branch is a bit behind master — a quick "Update branch" before merge will re-green required checks).

(Release-manager overnight gate — queued for maintainer merge, not autonomously tagged.)

[nesquena-hermes]

Shipped in experimental release exp-v0.52.230 (batched with the GLM-5.3 catalog add). Merged onto master via batch release PR #7090 — your commit content is in master and live on the experimental channel now.

Thanks @jorgejiro for the fix.

🤖 Closed by the release-manager agent.