Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docker/stage2-hook.sh
Original file line number Diff line number Diff line change
Expand Up @@ -431,6 +431,33 @@ seed_one ".env" ".env.example"
seed_one "config.yaml" "cli-config.yaml.example"
seed_one "SOUL.md" "docker/SOUL.md"

# --- Ensure a gateway api_server key exists (loopback control plane) ---
# The gateway's aiohttp api_server refuses to start without a strong
# API_SERVER_KEY (>=16 chars; startup guard in gateway/platforms/api_server.py).
# Hosted deployments need that listener on loopback so the dashboard — the
# container's only public HTTP door — can forward Chronos cron fires into the
# GATEWAY process, where the live platform adapters (relay, E2EE) live. The
# cron-fire route itself is NAS-JWT-authed, not key-authed; the key gates the
# rest of the api_server surface. Generate once, persist in .env (mounted
# volume), never overwrite an operator-provided value. Loopback-only: the
# default bind host is 127.0.0.1 and the Fly service only exposes the
# dashboard's port, so this listener is never publicly reachable.
if [ -f "$HERMES_HOME/.env" ] && ! grep -q '^API_SERVER_KEY=..*' "$HERMES_HOME/.env" 2>/dev/null; then
if refuse_symlinked_path "append" "$HERMES_HOME/.env"; then
:
else
_gen_key=$(head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n')
if [ -n "$_gen_key" ]; then
# Drop an empty assignment line if the seed left one behind, then
# append the generated key.
sed -i '/^API_SERVER_KEY=$/d' "$HERMES_HOME/.env" 2>/dev/null || true
printf 'API_SERVER_KEY=%s\n' "$_gen_key" >> "$HERMES_HOME/.env"
echo "[stage2] Generated API_SERVER_KEY for the loopback gateway api_server"
fi
unset _gen_key
fi
fi

# .env holds API keys and secrets — restrict to owner-only access. Applied
# unconditionally (not only on first-seed) so a host-mounted .env that was
# created with a permissive umask gets tightened on every container start.
Expand Down
26 changes: 20 additions & 6 deletions docs/chronos-managed-cron-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,12 +126,26 @@ Arm (or re-arm, idempotently) exactly one one-shot for a job.

## Inbound `POST /api/cron/fire` (NAS → agent) — agent side, already implemented

This is the agent endpoint NAS calls in Endpoint 3 step 3. Served by the
**dashboard app** (`hermes_cli/web_server.py`) — the agent's always-reachable
public HTTP surface on hosted deployments (the gateway may be idle/scaled down);
it is in `PUBLIC_API_PATHS` so the dashboard cookie gate lets the bearer-JWT
callback through to the verifier. (Also registered on the optional
`APIServerAdapter` for self-host API-server deployments.) The verifier is
This is the agent endpoint NAS calls in Endpoint 3 step 3. Two hops on hosted
deployments:

1. **Dashboard app** (`hermes_cli/web_server.py`) — the agent's only public
HTTP surface (the Fly proxy exposes exactly one port, the dashboard's). It
is in `PUBLIC_API_PATHS` so the dashboard cookie gate lets the bearer-JWT
callback through to the verifier. The dashboard verifies the JWT, resolves
the job's profile, then **forwards** the fire to hop 2 on loopback with the
NAS bearer preserved — it does NOT execute the job itself.
2. **Gateway `APIServerAdapter`** (`gateway/platforms/api_server.py`, loopback
bind, default port 8642) — re-verifies the JWT (defense in depth) and runs
the job with the gateway's **live platform adapters**, which is what makes
delivery work for relay-fronted logical platforms and E2EE rooms (the
standalone send path can serve neither). Self-host API-server deployments
that expose the api_server directly hit hop 2 without hop 1.

Gateway unreachable from hop 1 (scale-to-zero wake still booting, restart
window, api_server disabled) → the dashboard returns **503** and NAS retries
(non-2xx = retryable, below); the store CAS de-dupes the eventual double fire.
There is deliberately no in-dashboard execution fallback. The verifier is
`plugins/cron/chronos/verify.py`.

- **Auth:** `Authorization: Bearer <NAS-minted JWT>`. The agent verifies:
Expand Down
20 changes: 19 additions & 1 deletion gateway/platforms/api_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -5849,10 +5849,28 @@ async def _handle_cron_fire(self, request: "web.Request") -> "web.Response":
provider = resolve_cron_scheduler()

loop = asyncio.get_running_loop()
# Live adapters for delivery parity with the built-in ticker
# (gateway/run.py passes runner.adapters to the in-process
# scheduler). Without them, _deliver_result cannot resolve a live
# transport, so E2EE platforms and relay-fronted logical platforms
# (whose only send path IS the live relay adapter — no native
# credential exists) fail with "platform 'X' not
# configured/enabled" on every external-provider fire even though
# the same job delivers fine under the built-in ticker.
runner = self.gateway_runner or request.app.get("gateway_runner")
if runner is None:
try:
from gateway.run import _gateway_runner_ref

runner = _gateway_runner_ref()
except Exception:
runner = None
adapters = getattr(runner, "adapters", None) or None

# Fire in the background (202 immediately). fire_due claims via the
# store CAS, so a retry while this is in flight is de-duped.
task = asyncio.create_task(
asyncio.to_thread(provider.fire_due, job_id, adapters=None, loop=loop)
asyncio.to_thread(provider.fire_due, job_id, adapters=adapters, loop=loop)
)
reservation["detached"] = True
task.add_done_callback(
Expand Down
62 changes: 33 additions & 29 deletions gateway/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -7732,6 +7732,34 @@ def _restart_loop_guard_config(self) -> tuple:
pass
return max_restarts, window_seconds, max_gap_seconds

def _scale_to_zero_active_messaging_platforms(self) -> list:
"""ENABLED platforms that count for the relay-only arm gate (D1/F6).

Two filters, both load-bearing:
- enabled only: config.platforms is pre-seeded with disabled
placeholders for the full platform catalog (the F25 bug).
- MESSAGING only: non-messaging surfaces must not disarm scale-to-zero.
The api_server is a loopback listener force-enabled by the presence
of API_SERVER_KEY (which the Docker stage2 hook now generates for
every container, so hosted instances ALWAYS have it enabled) — it
holds no outbound socket and Chronos fires through it already reset
the idle clock. Counting it made messaging_is_relay_only_or_absent
False on every hosted instance, silently disarming the feature.
Mirrors the non-messaging exclusion set used for handoff eligibility
(see the `messaging_platforms` computation in _connect_platforms).
"""
if not self.config:
return []
non_messaging = {Platform.LOCAL, Platform.API_SERVER, Platform.WEBHOOK}
try:
return [
p
for p, pc in self.config.platforms.items()
if getattr(pc, "enabled", False) and p not in non_messaging
]
except Exception: # noqa: BLE001
return []

def _scale_to_zero_should_arm(self) -> bool:
"""Whether to start the idle watcher (D1/D11/§3.4(1))."""
from gateway.relay import relay_wake_url
Expand All @@ -7741,23 +7769,7 @@ def _scale_to_zero_should_arm(self) -> bool:
should_arm,
)

try:
# Only ENABLED platforms count. `config.platforms` is pre-seeded with a
# disabled placeholder PlatformConfig for every KNOWN platform (telegram,
# discord, slack, …), so `.keys()` is the full ~20-entry catalog regardless
# of what this instance actually runs. Passing the bare keys made
# `messaging_is_relay_only_or_absent` see those placeholders as live
# direct-socket platforms and return False, so scale-to-zero NEVER armed on
# a real relay-only instance. Mirror the connect loop, which already gates on
# `platform_config.enabled` (see the `if not platform_config.enabled: continue`
# in the adapter-connect loop) — arm off the same notion of "active platform."
platforms = (
[p for p, pc in self.config.platforms.items() if getattr(pc, "enabled", False)]
if self.config
else []
)
except Exception: # noqa: BLE001
platforms = []
platforms = self._scale_to_zero_active_messaging_platforms()
try:
wake_url = relay_wake_url()
except Exception: # noqa: BLE001
Expand Down Expand Up @@ -7786,18 +7798,10 @@ def _log_scale_to_zero_not_armed_reason(self) -> None:
enabled = scale_to_zero_enabled()
if not enabled:
return # not opted in — normal, stay quiet
try:
active = (
[
getattr(p, "value", p)
for p, pc in self.config.platforms.items()
if getattr(pc, "enabled", False)
]
if self.config
else []
)
except Exception: # noqa: BLE001
active = []
active = [
getattr(p, "value", p)
for p in self._scale_to_zero_active_messaging_platforms()
]
relay_only = messaging_is_relay_only_or_absent(active)
try:
wake_url = relay_wake_url()
Expand Down
51 changes: 33 additions & 18 deletions hermes_cli/web_routers/cron.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
_delete_cron_job_sync = late("_delete_cron_job_sync")
_find_cron_job_profile = late("_find_cron_job_profile")
_fire_cron_job_for_profile = late("_fire_cron_job_for_profile")
_forward_cron_fire_to_gateway = late("_forward_cron_fire_to_gateway")
_call_cron_for_profile = late("_call_cron_for_profile")
_raise_if_cron_registration_error = late("_raise_if_cron_registration_error")
load_config = late("load_config")
Expand Down Expand Up @@ -124,22 +125,28 @@ async def delete_cron_job(job_id: str, profile: Optional[str] = None):

@router.post("/api/cron/fire")
async def cron_fire_webhook(request: Request):
"""Chronos managed-cron fire webhook (NAS -> agent).
"""Chronos managed-cron fire webhook (NAS -> agent) — gateway forwarder.

Authenticated by a short-lived NAS-minted JWT (verified by the pluggable
Chronos fire-verifier), NOT the dashboard session cookie — so this path is
in ``PUBLIC_API_PATHS`` to bypass the dashboard auth gate, and the JWT is
the real gate. This is the inbound half of scale-to-zero managed cron: NAS
POSTs here at fire time, the agent verifies, claims the job (store CAS, so
at-most-once across replicas / on a NAS retry), runs it, and re-arms the
next one-shot.

Lives on the dashboard app (not the api_server adapter) because the
dashboard is the agent's always-reachable public HTTP surface on hosted
deployments; the gateway may be idle/scaled down.

Returns 202 immediately and runs the job in the background so a long agent
turn never trips NAS's HTTP timeout.
the real gate.

The dashboard is only the PUBLIC DOOR here (on hosted deployments the Fly
proxy exposes exactly one port, the dashboard's). Cron execution belongs
to the GATEWAY process, which owns the live platform adapters — required
for relay-fronted logical platforms (their only sender is the live relay
adapter) and E2EE rooms, neither of which the dashboard's standalone send
path can serve. So after verifying the JWT this handler FORWARDS the fire
to the gateway api_server's own ``/api/cron/fire`` on loopback and passes
the gateway's response through (the gateway re-verifies the JWT — defense
in depth, no new trust link).

Gateway unreachable (scale-to-zero wake still booting, restart window,
api_server disabled) → 503, so NAS retries per the Chronos contract
(non-2xx = retryable). The store CAS claim de-dupes the eventual double
fire. Deliberately NO local-execution fallback: delivering from the wrong
process is worse than a delayed retry.
"""
from plugins.cron_providers.chronos.verify import get_fire_verifier

Expand Down Expand Up @@ -173,12 +180,20 @@ async def cron_fire_webhook(request: Request):
# does not retry a fire that is intentionally absent.
return JSONResponse({"status": "gone", "job_id": job_id}, status_code=200)

# Run in the background; the store CAS claim inside fire_due de-dupes a
# NAS/scheduler retry that arrives while this is in flight.
asyncio.create_task(
asyncio.to_thread(_fire_cron_job_for_profile, profile, job_id)
)
return JSONResponse({"status": "accepted", "job_id": job_id}, status_code=202)
forwarded = await _forward_cron_fire_to_gateway(profile, job_id, auth)
if forwarded is None:
return JSONResponse(
{
"error": "gateway unreachable; retry",
"job_id": job_id,
"profile": profile,
},
status_code=503,
)
status_code, gateway_body = forwarded
if isinstance(gateway_body, dict):
gateway_body.setdefault("job_id", job_id)
return JSONResponse(gateway_body, status_code=status_code)


@router.get("/api/cron/blueprints")
Expand Down
Loading
Loading