fix(sidecar): restore opt-in inbound-liveness heartbeat file - #84996
Closed
ragnos-dev wants to merge 63 commits into
Closed
fix(sidecar): restore opt-in inbound-liveness heartbeat file#84996ragnos-dev wants to merge 63 commits into
ragnos-dev wants to merge 63 commits into
Conversation
Replays the RAGnos delta (5 ragnos/ shim files clean; 2 Photon files) onto the v2026.6.19 release. The 3 Photon conflict hunks resolved: keep the timeout-aware send error + record_sent_message; merge the per-call client (event-loop safety) with the env-configurable PHOTON_SIDECAR_TIMEOUT (90s default); keep both sidecar imports. adapter.py compiles. Merge + the live iMessage round-trip are operator-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…emetry) Observe-only by default; RAGNOS_GOVERNANCE_ENFORCE=1 blocks gated tools so they route through the Hermes Hub. Sprint 2 of the realignment. Additive, in the RAGnos-owned plugins/ragnos-governance/ surface; upstream core untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cooldown Closes NousResearch#50185 Two independent gaps let a transient Photon/Spectrum upstream overflow degrade message delivery and amplify gRPC pressure: 1. _is_retryable_error did not recognise Photon- or Envoy-specific error strings ("internal sidecar error", "upstream connect error", "reset reason: overflow"), so _send_with_retry fell through to the plain-text fallback immediately instead of backing off and retrying. 2. send_typing had no rate gate, so a burst of typing-indicator calls during an overflow event kept hitting the upstream gRPC connection and widened the failure window. Fix: - Add _PHOTON_RETRYABLE_PATTERNS with the three high-specificity Envoy / sidecar substrings and override _is_retryable_error on PhotonAdapter to check them after delegating to the base-class patterns. base.py and all other adapters are untouched. - Add a 5 s per-chat cooldown in send_typing backed by _typing_last_sent. stop_typing clears the entry so the next start after a completed turn fires immediately — only rapid consecutive starts without a stop are suppressed. - Reduce PhotonAdapter._send_with_retry default max_retries from 2 to 1 (single 2 s back-off check) — enough to confirm whether the Envoy circuit-breaker has opened, without adding unnecessary latency. All changes are scoped to plugins/platforms/photon/adapter.py.
When the Node spectrum-ts sidecar process exited mid-session (crash,
OOM, upstream overflow escalation), _supervise_sidecar returned
silently — readline hit EOF, the log-pump loop broke, and nothing
notified the gateway. _inbound_loop entered an infinite retry loop
against a dead port, _running stayed True, and the adapter remained
in self.adapters with no path to self-recovery short of a manual
gateway restart.
Add a death-detection tail to _supervise_sidecar: after the log-pump
exits (EOF or exception), guard on _inbound_running to distinguish
unexpected death from a deliberate disconnect(). On unexpected exit,
call _set_fatal_error("SIDECAR_CRASHED", retryable=True) followed by
_notify_fatal_error() so the reconnect watcher picks up the platform
within 30 s and retries with exponential backoff (30 s → 300 s cap)
until the sidecar comes back up. All other platforms remain unaffected.
The _inbound_running guard is safe against races: disconnect() sets
_inbound_running = False before _stop_sidecar() cancels the supervisor
task. CancelledError is BaseException, not Exception, so it bypasses
the except clause and propagates normally — the detection block never
runs during a clean shutdown.
…tection Follow-up for salvaged PR NousResearch#50256. Unit tests for the three behaviors: retryable classification of Envoy/sidecar overflow strings, per-chat typing cooldown with stop_typing reset, and the _supervise_sidecar crash-detection path that raises a retryable fatal (and the clean-shutdown no-op).
…alate
spectrum-ts routes stream telemetry through @photon-ai/otel's createLogger,
which sends severity>=ERROR to console.error and WARN/INFO to console.log.
The two lines the health monitor keys off land on different channels:
log.error("stream persistently failing") -> console.error (caught), but
log.warn("stream interrupted; reconnecting") -> console.log (was missed).
The original interception patched console.error only, so the recovering->
degraded escalation counter never saw the interrupt bursts that are the
primary silent-inbound symptom. Verified live against spectrum-ts 3.1.0 +
@photon-ai/otel: 3 real log.warn('stream interrupted') calls now escalate
to degraded -> process.exit(75) -> adapter reconnect.
Adds a shared classifyStreamLog() fed by both console.error and console.log,
plus a regression test asserting both channels are intercepted.
Update the Photon platform plugin's Node.js sidecar from spectrum-ts 3.1.0 to 7.0.0, which splits the SDK into scoped `@spectrum-ts/*` packages with `spectrum-ts` as the umbrella re-export. - Bump exact pin in package.json/package-lock.json to 7.0.0 - Update mixed-attachments patch script to target the new `@spectrum-ts/imessage/dist/index.js` path and tab-indented output - Rewrite test fixture to match v7.x mapper shape (tab-indented, `const ... = async` declarations, single-line builder calls) and point at `@spectrum-ts/imessage/dist/index.js` - Update README upgrade guide to document the v5 package split and the postinstall patch validation step - Update comments in cli.py and index.mjs to reference v5/v7 changes
v8 made `richlink` outbound-only; inbound rich links now arrive as plain `text`. Remove the `getBalloonBundleId`/`toRichlinkMessage` branches from the iMessage mapper patch and update the fixture, lockfile, and README accordingly.
Populate `reply_to_message_id`, `reply_to_text`, and `reply_to_is_own_message` on reaction events so the gateway injects `[Replying to your previous message: "..."]` when the agent receives a tapback. The sidecar now extracts a capped text preview from the hydrated reaction target (plain text and mixed group messages; null for attachment/voice-only targets), emitting it as `targetText` in the NDJSON reaction payload. The Python adapter reads this field and sets the reply correlation fields on the `MessageEvent`.
…06-28 fix(photon): pull upstream v8 recovery fixes
Restores gateway/platforms/base.py and gateway/run.py to upstream state so hermes-upstream-conformance passes. The reply_to_is_own_message field is removed from MessageEvent. Instead the Photon adapter prefixes reply_to_text with "your previous message: " when handling tapbacks on bot messages. The generic gateway handler then emits `[Replying to: "your previous message: ..."]` with no core edits required. Tests updated to assert the new reply_to_text encoding. 19 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reverts the two immediate-reconnect edits in gateway/run.py (next_retry without the +30 delay, and the early-break on _failed_platforms in the idle sleep loop) so gateway/run.py matches the upstream merge-base and the hermes-upstream-conformance gate passes. Operator decision (Option A): accept upstream's ~30s retry delay. Fast Photon sidecar-crash recovery is still provided by the adapter's own sidecar-death watcher (plugins/platforms/photon/adapter.py) plus the 30s->300s reconnect backoff, so recovery is preserved, just not instant on the first retry. Updates test_retryable_runtime_error_reconnects_immediately -> test_retryable_runtime_error_queued_with_retry_delay to assert the 30s queueing. 59 gateway/photon tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e-2026-06-29 fix(hermes): restore gateway core to upstream, clear conformance gate
…eceipts feat(photon): add stable delivery receipts
…6.7.20 Re-baseline fork onto upstream v2026.7.20 (Quicksilver)
…real PhotonAdapter send path PhotonAdapter dedupes inbound sidecar deliveries only; it has no outbound dedup, so the delivery ledger's crash-ambiguity contract is the only guard against a duplicate iMessage from a crash-recovered final response. No photon test exercised _redeliver_pending_obligations until now. Covers, with the sidecar transport stubbed at _sidecar_call so the real send/format_message/_sidecar_send chain runs: - ambiguous (attempting) rows carry RECOVERED_MARKER into the /send body; pending rows redeliver plainly and clear resume_pending - a delivered row is never claimed or sent again by a later boot's sweep - a sidecar failure leaves the row failed-but-claimed, spending exactly one attempt per boot until MAX_ATTEMPTS abandons it Test-only change under tests/plugins/platforms/photon/ (allowed overlay path); no upstream core files touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…livery-test test(photon): pin startup obligation-redelivery contract through the real PhotonAdapter send path
…2-20260721 fix(photon): qualify Spectrum 12 delivery contracts
…ributor-20260721 fix(governance): restore fork overlay conformance
…v12-20260721 fix(photon): canonicalize direct chat spaces
…-8-3-20260803 feat(photon): rebaseline fork on v2026.8.3
* feat(voice): add loopback local streaming provider * Revert "feat(voice): add loopback local streaming provider" This reverts commit faa65dc. * feat(voice): register local streaming overlay --------- Co-authored-by: Dr. RAGnos <doctor@ragnos.io>
Co-authored-by: Dr. RAGnos <doctor@ragnos.io>
Co-authored-by: Dr. RAGnos <doctor@ragnos.io>
The v2026-8-3 rebaseline (#15) dropped the heartbeat-file stamping that 639d041 added, while scripts/launchd/photon-sidecar.sh (exports PHOTON_SIDECAR_HEARTBEAT_PATH) and keez-reaper.sh (kickstarts the sidecar + inbound adapter pair when the file goes stale >90s) still assume it. Result: as soon as both processes are up, the reaper kickstarts the pair every tick forever, and Keez inbound can never hold a connection. Ports the original stamping back onto current main: stamp on /inbound consumer connect and on every 25s keepalive tick. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The v2026-8-3 rebaseline (#15) dropped the opt-in heartbeat-file stamping originally added in 639d041. The RAGnos workspace side still assumes it exists:
scripts/launchd/photon-sidecar.shexportsPHOTON_SIDECAR_HEARTBEAT_PATHscripts/launchd/keez-reaper.shkickstarts the sidecar + inbound adapter pair whenever the file is >90s staleWith the stamping gone, the file froze at its last write (Jul 23). The check is gated on both processes being up, so it stayed dormant while the inbound adapter was down — and the moment the adapter was restored on 2026-08-12, the reaper began kickstarting the pair on every tick (~30s), a permanent restart loop.
This ports the original stamping onto current main verbatim: stamp
PHOTON_SIDECAR_HEARTBEAT_PATHon each /inbound consumer connect and on every 25s keepalive tick. Opt-in via env var; upstream behavior unchanged when unset.Evidence
keez-reaper.logshowedphoton inbound heartbeat STALE (1758501s > 90s) -> kickstarton every tick; sidecar/adapter PIDs churned each cycle.node --checkpasses.🤖 Generated with Claude Code