fix(photon): recover degraded sidecar streams

[tusharaggarwa1]

Summary

• classify Photon sidecar control failures into safe retryable/non-retryable errors, including upstream transient failures, invalid credentials, and Photon project policy failures
• restart/reconnect Photon when the sidecar process crashes or reports degraded upstream stream health, and avoid reporting an idle stream as healthy before real traffic arrives
• make gateway runtime adapter failures queue immediate background reconnection instead of waiting for an inbound message or another startup cycle
• let slow-starting adapters declare a longer connect timeout so Photon sidecar startup is not killed by the generic platform timeout
• resolve restarted Photon DM spaces with space.get(any;-;+E164) before falling back to phone-based space.create, so established iMessage DMs remain sendable after sidecar restart
• retry standalone Photon sends when the sidecar reports a transient retryable send failure, and avoid blank standalone-send errors when Python/httpx raises an exception with an empty string
• upgrade the Photon sidecar to spectrum-ts@5.2.0 and skip the mixed-attachment patch when the newer inbound mapper is present

Related PRs / Expected Merge Path

• fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161 is now the likely maintainer-preferred stream-recovery base: it salvages fix(photon): recover degraded upstream stream #51105/fix(photon): label upstream CatchUpEvents failures #50071 and adds the important console.log interception fix for Spectrum WARN/INFO stream logs. If fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161 lands first, this PR should be rebased and duplicate stream-recovery hunks should be dropped.
• fix(photon): recover sidecar after transport drops #49653 is an older/narrower transport-drop recovery PR. It overlaps with outbound upstream_connection_dropped sidecar restart behavior, but does not cover the broader /healthz stream-state path, slow Photon startup timeout, DM rehydration, or the spectrum-ts upgrade/patch guard here.
• fix(photon): harden iMessage outbound recovery #51075 overlaps on outbound hardening and the spectrum-ts upgrade. This PR incorporates the DM-resolution and retry-classification pieces, but still does not include fix(photon): harden iMessage outbound recovery #51075's HEIC/HEIF image transcoding; that remains separable from the gateway/sidecar liveness issue.

After #51161, the intended remaining value of this PR is the live-repro follow-up surface: structured sidecar send classification, adapter-specific Photon startup timeout, DM-space rehydration before phone-based creation, standalone send retry/error clarity, and the spectrum-ts@5.2.0 upgrade/patch guard if those are not already merged elsewhere.

Happy to split those remaining pieces further if maintainers prefer a narrower review path.

Validation

• ./scripts/run_tests.sh tests/plugins/platforms/photon/test_overflow_recovery.py tests/plugins/platforms/photon/test_spectrum_patch.py tests/gateway/test_platform_reconnect.py
• node --check plugins/platforms/photon/sidecar/index.mjs
• node --check plugins/platforms/photon/sidecar/patch-spectrum-mixed-attachments.mjs
• /home/tushar/.hermes/hermes-agent/venv/bin/python -m py_compile plugins/platforms/photon/adapter.py gateway/run.py
• git diff --check
• /home/tushar/.hermes/hermes-agent/venv/bin/python scripts/check-windows-footguns.py --diff origin/main

Live Runtime Check

• restarted the local systemd gateway by SIGTERMing the main process and letting systemd revive it
• reproduced the old failure mode: Photon startup was cut off around the sidecar readiness timeout, leaving systemd green with no Photon sidecar child
• verified the patched gateway gives Photon the longer adapter-specific startup window; local Photon startup took about 35s and then connected
• observed a real inbound Photon text reach Hermes and trigger a gateway response
• upgraded the live sidecar to spectrum-ts@5.2.0; gateway restarted and connected with the newer Spectrum runtime
• verified direct sidecar /send can return a real Spectrum message id; also observed intermittent upstream send drops now surface as classified upstream_transient retryable errors

[huntsyea]

This PR matches a real live failure mode we hit today: the local sidecar process was healthy enough to mask the fact that the upstream Photon stream was degraded/dead.

Observed state:

Photon sidecar process: running
127.0.0.1:8789: listening
spectrum-ts: 5.2.0
Gateway health: ok

But the stream itself was failing repeatedly:

[spectrum.stream] ERROR stream persistently failing; still retrying
spectrum.stream.label: imessage.messages:shared
spectrum.stream.phase: catch-up
spectrum.stream.attempt: 16
spectrum.error.type: ConnectionError
spectrum.error.message: upstream connect error or disconnect/reset before headers. reset reason: connection termination
grpcCode: 14

Earlier in the same incident we also saw:

CatchUpEvents PERMISSION_DENIED: [upstream] Authentication failed. grpcCode: 7
SpectrumCloudError: Invalid credentials status=401

Best-practice confirmation: /healthz needs to include stream health, not just local HTTP/process liveness. A live index.mjs and open /inbound loop are not sufficient proof that iMessage inbound is actually flowing.

[tusharaggarwa1]

For reviewer context, this PR now intentionally overlaps with two related Photon PRs:

• fix(photon): recover degraded upstream stream #51105 (fix(photon): recover degraded upstream stream)
• fix(photon): harden iMessage outbound recovery #51075 (fix(photon): harden iMessage outbound recovery)

What this PR changes:

• Adds safe, machine-readable Photon sidecar error classification so Hermes can distinguish transient upstream failures from permanent auth/project-policy failures.
• Promotes degraded Photon upstream stream health into a retryable adapter fatal error, so the gateway reconnect path can restart Photon instead of leaving a live local sidecar with dead inbound delivery.
• Avoids marking the sidecar stream healthy before actual inbound traffic is observed, which prevents /healthz from reporting a false healthy state while the stream is only idle/starting.
• Queues retryable runtime adapter failures for immediate background reconnection.
• Lets slow-starting adapters declare a longer connect timeout; this fixes a reproduced local failure where systemd showed the gateway as active, but Photon had no sidecar child because startup was killed before Spectrum finished initializing.
• Resolves established Photon DM ids with space.get(any;-;+E164) before falling back to phone-based space.create, preserving sendability after sidecar restart.
• Retries standalone Photon sends when the sidecar marks a send failure as transient/retryable, and reports timeout/transport exception class names instead of returning blank errors.
• Upgrades the Photon sidecar to spectrum-ts@5.2.0 and skips the mixed-attachment patch when the newer inbound mapper is present.

Relationship to #51105:

• Same core problem area: a local Photon sidecar can be alive while the upstream Spectrum stream is degraded or recovering.
• This PR incorporates the important fix(photon): recover degraded upstream stream #51105 stream-health semantics, especially not declaring the stream healthy until real traffic is observed.
• This PR keeps the broader /healthz degradation-monitoring path and adapter fatal/reconnect integration from the original branch.

Relationship to #51075:

• Same outbound recovery problem area: restarted sidecars need to rehydrate existing DM spaces before trying phone-based creation, and sidecar send failures should carry retryability/classification.
• This PR now incorporates the relevant DM-resolution and retry-classification pieces, plus the spectrum-ts@5.2.0 sidecar upgrade and patch-script guard.
• This PR still does not take the broader HEIC/HEIF image transcoding from fix(photon): harden iMessage outbound recovery #51075. That looks useful, but it is separable from the gateway/sidecar liveness issue this PR is trying to fix.

Validation run on this branch:

• ./scripts/run_tests.sh tests/plugins/platforms/photon/test_overflow_recovery.py tests/plugins/platforms/photon/test_spectrum_patch.py tests/gateway/test_platform_reconnect.py → 60 passed
• node --check plugins/platforms/photon/sidecar/index.mjs
• node --check plugins/platforms/photon/sidecar/patch-spectrum-mixed-attachments.mjs
• /home/tushar/.hermes/hermes-agent/venv/bin/python -m py_compile plugins/platforms/photon/adapter.py gateway/run.py
• git diff --check
• /home/tushar/.hermes/hermes-agent/venv/bin/python scripts/check-windows-footguns.py --diff origin/main

Live verification:

• Reproduced the pre-fix state where the gateway service was active but Photon had no sidecar child.
• Verified Photon startup needed about 35s locally; the old generic timeout path could cut it off before startup completed.
• Observed a real inbound Photon text reach Hermes and trigger a gateway response.
• Verified direct sidecar /send can return a real Spectrum message id.
• Observed intermittent Photon upstream send drops now surface as classified upstream_transient retryable errors instead of generic/blank failures.

Happy to split this if maintainers prefer a narrower merge path. The natural split would be:

1. stream health + reconnect behavior (fix(photon): recover degraded upstream stream #51105-equivalent),
2. outbound DM/error classification behavior (fix(photon): harden iMessage outbound recovery #51075-overlap),
3. adapter-specific connect timeout for slow Photon startup,
4. spectrum-ts@5.2.0 sidecar upgrade.

[tusharaggarwa1]

Coordination update after reviewing the newer related Photon PRs, especially #51161 and #49653.

I agree that #51161 looks like the clean stream-recovery landing path if maintainers prefer it. It salvages #51105/#50071 and adds an important follow-up fix this branch should not ignore: Spectrum WARN/INFO stream lines can land on console.log, not only console.error, so the degraded-stream detector needs to observe both channels.

How I see the current split:

• fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161: inbound upstream stream-health recovery, CatchUpEvents upstream-error labeling, and the console.log/console.error stream-log interception fix. This is probably the best candidate to land first for the core silent-inbound outage.
• fix(photon): recover sidecar after transport drops #49653: older/narrower transport-drop recovery. It overlaps with outbound upstream_connection_dropped recovery, but does not appear to cover the broader /healthz stream-state path, slow Photon startup timeout, DM rehydration, or the spectrum-ts upgrade/patch guard in this PR.
• fix(photon): harden iMessage outbound recovery #51075: outbound hardening, including DM-space recovery and spectrum-ts upgrade work. The HEIC/HEIF transcoding part still seems separable from the gateway/sidecar liveness issue.
• fix(photon): recover degraded sidecar streams #50974: currently broader than fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161. The parts intended to remain useful after fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161 are structured sidecar send classification (retryable vs permanent auth/project-policy failures), adapter-specific Photon startup timeout, space.get(any;-;+E164) before phone-based DM creation, standalone send retry/error clarity, and the spectrum-ts@5.2.0 upgrade plus mixed-attachment patch guard.

I am not trying to supersede the maintainer salvage in #51161. The maintainer-friendly path I expect is:

1. Let fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161 land first if maintainers prefer it as the stream-recovery base.
2. Rebase this PR on current main.
3. Drop any duplicate stream-recovery hunks already covered by fix(photon): recover degraded upstream stream (salvage #51105 + #50071) #51161.
4. Keep only the remaining outbound/timeout/DM/runtime-classification fixes, with focused tests and the live repro evidence already captured here.

Happy to split those remaining pieces further if that is easier to review.

[teknium1]

Thanks for preserving the residual Photon recovery work and documenting the overlap with #51161.

Problems

• Current main already contains the stream-recovery path from 06cbc3bae and the console.log follow-up from 7f1c278db (plugins/platforms/photon/sidecar/index.mjs:171-204,526-559). Those hunks should not be salvaged again.
• The PR's spectrum-ts 5.2.0 pin conflicts with current main's v8 pin at plugins/platforms/photon/sidecar/package.json:16, introduced by 4345b3e. The v5 patch guard/layout needs a v8-targeted port rather than a downgrade.
• The newly adjustable timeout/health settings are environment-backed in the diff, but AGENTS.md:102-106,620-623 requires non-secret behavioral settings to be config.yaml-backed.

Suggested changes

• Preserve and re-target only the still-live behavior: current main still has the 15-second readiness deadline (plugins/platforms/photon/adapter.py:984-1005), phone-create-before-DM-get resolution (plugins/platforms/photon/sidecar/index.mjs:661-683), and unclassified sidecar reply handling (plugins/platforms/photon/adapter.py:1552-1561).
• Keep Spectrum v8 and add config-resolution coverage for any new knobs.

Automated hermes-sweeper review.

[teknium1]

Current main is now pinned to spectrum-ts 8.0.0 by 4345b3e. Please port the remaining recovery behavior to the v8 sidecar and retain that dependency rather than restoring this 5.2.0 pin and its incompatible lockfile/patch layout.

[teknium1]

This introduces a user-facing timeout/threshold through environment variables. AGENTS.md:102-106,620-623 requires non-secret behavioral settings to be config.yaml-backed; bridge to environment only internally if the sidecar requires it.

[teknium1]

Closing — the healthz stream-health and degraded→fatal→reconnect core of this landed earlier (adapter health monitoring on main), the spectrum 5.2.0 upgrade is obsolete against the 8.0.0 pin, and the structured-error/retry work landed via #73563. The DM get-before-create idea remains a good one — resubmit that narrowly if you're up for it.