Skip to content

Add the readiness probe vocabulary for services: tcp, http, log - #403

Closed
ishandhanani wants to merge 1 commit into
idhanani/srt2-12-dynamo-sourcefrom
idhanani/srt2-13-readiness
Closed

ishandhanani wants to merge 1 commit into
idhanani/srt2-12-dynamo-sourcefrom
idhanani/srt2-13-readiness

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #400 (Track 2 step 8 of #385). Draft until the stack below it merges.

What

services[].readiness gains the probe vocabulary from the schema design: exactly one of

readiness:
  port: 9000                 # shorthand for tcp
readiness:
  tcp:
    port: 9000               # a TCP connection is accepted
readiness:
  http:
    port: 8000
    path: /ready             # GET http://<node>:8000/ready
    status: 200              # returns this status
readiness:
  log:
    pattern: 'Uvicorn running on .*:\d+'   # regex against service_<name>.out

plus timeout_seconds (default 120) and interval_seconds (default 2). The generic wait loop in core/readiness.py re-runs the probe until it passes, the deadline expires, or the process dies, so a crashed service fails the job at once with its exit code instead of after the full timeout. The port-collision check now keys on the port a tcp or http probe implies.

Per the design, the global health_check block and the worker and frontend health checks are untouched: this fixes the silent-dead-sidecar class (NVIDIA/InferenceMAX#271) without touching the downstream recipes that set health_check.

Also

  • examples/features/services.yaml now gates on an http probe against the log browser's directory listing.
  • tests/test_readiness.py covers the schema (shorthand, exactly-one, invalid regex/path/interval), each probe, and the wait loop with a fake clock (retries, fail-fast on death, clipped final sleep). tests/test_services.py updated for the probe API.
  • Docs: probe section in docs/services.md, config-reference row, regenerated schema reference.

Validation

  • Full suite green (1835 passed); lint, schema-docs drift check, every example validates.
  • sa-b200, job 12295: the services example gated on http://<node>:9911/ -> 200 came ready after 40s, the benchmark completed, and the job finished clean (3m25s).

services[].readiness names exactly one probe: tcp (a port accepts a
connection; `port:` alone is shorthand), http (GET a path on a port and
expect a status), or log (a regular expression matched against the
service's log file), plus timeout_seconds and interval_seconds. The
generic wait loop in core/readiness.py re-runs the probe until it passes,
the deadline expires, or the process dies, so a crashed service fails at
once instead of after the full timeout. The global health_check block and
the worker and frontend health checks are untouched, per the schema
design: this fixes the silent-dead-sidecar class without touching the
377 downstream recipes that set health_check.

The services example now gates on an http probe against the log browser's
directory listing.
@ishandhanani

Copy link
Copy Markdown
Collaborator Author

Consolidated into #407, which carries these same commits as one PR against main (this description is reproduced there as one of its parts). Closing to keep review in one place; the branch stays for per-step reference via stack #398.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant