Skip to content

fix: wait for daemon health before reclaiming occupied port - #1858

Merged
nicoloboschi merged 1 commit into
vectorize-io:mainfrom
s09x:fix/daemon-port-health-grace
May 29, 2026
Merged

fix: wait for daemon health before reclaiming occupied port#1858
nicoloboschi merged 1 commit into
vectorize-io:mainfrom
s09x:fix/daemon-port-health-grace

Conversation

@s09x

@s09x s09x commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a bounded health grace window before reclaiming occupied daemon ports.
  • Reuse an already-starting daemon only after /health returns Hindsight's initialized health payload.
  • Preserve stale listener reclaim behavior after the grace window expires.

Reproduction

A concurrent daemon start can bind the profile port before /health is ready. The previous _clear_port behavior checked health once, saw the warming daemon as unhealthy/non-Hindsight, and moved into PID lookup/reclaim instead of waiting for the daemon to finish starting.

The regression coverage reproduces that race deterministically without real services or private data:

  1. _is_port_in_use(port) returns True.
  2. /health responses are simulated as 503, 503, then 200 with {"status": "healthy", "database": "connected"}.
  3. Previous behavior returns False after the first health probe and calls _find_pid_on_port.
  4. Patched behavior keeps polling within the grace window, returns True when the daemon becomes healthy, and does not call _find_pid_on_port or _kill_process.

Fix

  • Add configurable, bounded port-health grace settings:
    • HINDSIGHT_EMBED_PORT_HEALTH_GRACE_TIMEOUT
    • HINDSIGHT_EMBED_PORT_HEALTH_CHECK_INTERVAL
  • Add health helpers that require the initialized Hindsight health payload, not just any HTTP 200 response.
  • Use monotonic time for the grace deadline.
  • Treat listeners that disappear during the grace wait as already cleared.
  • Reclaim stale/unhealthy listeners only after the grace window expires.

Test Plan

  • uv run pytest tests/test_daemon_client.py::TestClearPort -q
  • uv run pytest tests/test_embed_manager.py -q
  • uv run pytest -q
  • uv run ruff check hindsight_embed/daemon_embed_manager.py tests/test_daemon_client.py
  • git diff --check --cached -- hindsight-embed/hindsight_embed/daemon_embed_manager.py hindsight-embed/tests/test_daemon_client.py

@s09x

s09x commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

The failing check appears to be a transient dependency download issue in verify-generated-files, not a code/test failure.

The job failed while Cargo was updating the crates.io registry:

failed to get `h2` as a dependency
unable to update registry `crates-io`
download of 2/h2 failed
curl failed
[16] Error in the HTTP2 framing layer

The local checks I ran before opening the PR completed successfully:

  • uv run pytest tests/test_daemon_client.py::TestClearPort -q
  • uv run pytest tests/test_embed_manager.py -q
  • uv run pytest -q
  • uv run ruff check hindsight_embed/daemon_embed_manager.py tests/test_daemon_client.py
  • git diff --check --cached -- hindsight-embed/hindsight_embed/daemon_embed_manager.py hindsight-embed/tests/test_daemon_client.py

@nicoloboschi
nicoloboschi merged commit 5e547f7 into vectorize-io:main May 29, 2026
59 of 60 checks passed
@s09x
s09x deleted the fix/daemon-port-health-grace branch May 29, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants