Skip to content

fix(docker): clean up orphaned container when docker run fails - #7440

Closed
Tranquil-Flow wants to merge 2 commits into
NousResearch:mainfrom
Tranquil-Flow:fix/docker-container-leak-on-failed-start
Closed

fix(docker): clean up orphaned container when docker run fails#7440
Tranquil-Flow wants to merge 2 commits into
NousResearch:mainfrom
Tranquil-Flow:fix/docker-container-leak-on-failed-start

Conversation

@Tranquil-Flow

@Tranquil-Flow Tranquil-Flow commented Apr 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

When docker run -d fails (e.g. exit code 125, timeout), Docker may have already created the container object before returning the error. The current code raises before self._container_id is set, so cleanup() can never remove it — leaking a stale "Created" container on every failed attempt.

This PR wraps the docker run call in try/except and removes the container by its known name before re-raising, then adds two unit tests covering the CalledProcessError and TimeoutExpired failure paths.

Observed in production: 110+ orphaned containers accumulated over 3 days from hourly cron-scheduled gateway sessions hitting a Docker Desktop startup race.

Related Issue

Fixes #7439

Also related:

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/environments/docker.py
    • Wrap subprocess.run(run_cmd, ..., check=True) in try/except for CalledProcessError and TimeoutExpired.
    • On failure, run docker rm -f <container_name> (name is known before the run) to remove any partially-created container.
    • Re-raise the original exception so existing error handling is unaffected.
  • tests/tools/test_docker_environment.py
    • test_failed_docker_run_cleans_up_orphaned_container — mocks docker run to raise exit code 125, asserts docker rm -f is called with the container name.
    • test_docker_run_timeout_cleans_up_orphaned_container — same for TimeoutExpired.

How to Test

  1. pytest tests/tools/test_docker_environment.py -v — all 20 tests pass (18 existing + 2 new).
  2. Manual: stop Docker Desktop, attempt DockerEnvironment() creation, verify no orphaned container in docker ps -a.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15 (Darwin 24.6.0)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

$ pytest tests/tools/test_docker_environment.py -v
20 passed (18 existing + 2 new)

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists backend/docker Docker container execution labels Apr 29, 2026
@alt-glitch

alt-glitch commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator

Competing fix with #7498 — both address #7439 (container leak on docker run failure). Same approach: cleanup by container name before re-raising.

@Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails
844292e 
When `docker run -d` fails (e.g. exit code 125 because the daemon isn't
ready), Docker may have already created the container object. The
CalledProcessError was raised before self._container_id was set, so
cleanup() could never remove it. Over time, cron-scheduled sessions
accumulated hundreds of stale "Created" containers.

Wrap the docker run call in try/except and remove the container by its
known name before re-raising.

Fixes NousResearch#7439
@Tranquil-Flow
Tranquil-Flow force-pushed the fix/docker-container-leak-on-failed-start branch from 35f98f3 to 844292e Compare May 25, 2026 11:07
@benbarclay benbarclay mentioned this pull request Jun 5, 2026
Merged
pedrotecinf pushed a commit to pedrotecinf/hermes-agent that referenced this pull request Jun 5, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
82c157b 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
davidgut1982 pushed a commit to davidgut1982/hermes-agent that referenced this pull request Jun 5, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
84da5a3 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
changman pushed a commit to changman/hermes-agent that referenced this pull request Jun 10, 2026
@benbarclay @Tranquil-Flow @changman
fix(docker): clean up orphaned container when docker run fails (salvage
78b4810 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
alt-glitch pushed a commit that referenced this pull request Jun 14, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
30b1505 
#7440) (#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in #7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in #33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages #7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes #7440. Fixes #7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
kossteg pushed a commit to kossteg/hermes-agent that referenced this pull request Jun 16, 2026
@benbarclay @Tranquil-Flow @kossteg
fix(docker): clean up orphaned container when docker run fails (salvage
9a18448 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
T02200059 pushed a commit to T02200059/hermes-agent that referenced this pull request Jun 18, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
ae75d7c 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
43e2b3a 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
d36d872 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
donbowman pushed a commit to donbowman/hermes-agent that referenced this pull request Jul 13, 2026
@benbarclay @Tranquil-Flow @donbowman
fix(docker): clean up orphaned container when docker run fails (salvage
bcecaf1 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
2cff928 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
@zhenheco zhenheco mentioned this pull request Jul 21, 2026
Closed
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
@benbarclay @Tranquil-Flow
fix(docker): clean up orphaned container when docker run fails (salvage
2498484 
NousResearch#7440) (NousResearch#39412)

When `docker run -d` fails after Docker has already created the container
object (e.g. exit 125 when the daemon isn't ready, or a timeout mid image
pull), the code raised before `self._container_id` was set — so the
container leaked permanently in "Created" state. Reported in NousResearch#7439:
110+ orphaned containers accumulated over 3 days from hourly cron-
scheduled gateway sessions hitting a Docker Desktop startup race.

The orphan reaper added in NousResearch#33645 (reap_orphan_containers) does NOT cover
this case: it filters `status=exited`, but a failed-create container is in
`Created` state, so it slips through and is never reaped.

Wrap the `docker run -d` call in try/except and `docker rm -f` the
container by its known name before re-raising.

Salvages NousResearch#7440 by @Tranquil-Flow. Their branch predated the cross-process
reuse + labels rework on `main`, so a cherry-pick conflicted; reconstructed
the same intent (plus their two regression tests, adapted to mock the new
reuse `docker ps` probe) against current `main`.

Verified adversarially: reverted just the product change to origin/main's
`docker.py`, ran the two new tests -> both FAIL with
`assert 0 == 1 ("docker rm should be called once")`. With the fix applied,
both pass; full test_docker_environment.py is 65/65 green.

Closes NousResearch#7440. Fixes NousResearch#7439.

Co-authored-by: Evi Nova <66773372+Tranquil-Flow@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backend/docker Docker container execution P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Docker containers leak in Created state when docker run fails

2 participants

@Tranquil-Flow @alt-glitch