Skip to content

fix(mesh): publish the estate so every transport can report it - #2309

Merged
igorls merged 1 commit into
developfrom
fix/mesh-estate-visible-to-all-clients
Aug 19, 2026
Merged

fix(mesh): publish the estate so every transport can report it#2309
igorls merged 1 commit into
developfrom
fix/mesh-estate-visible-to-all-clients

Conversation

@igorls

@igorls igorls commented Aug 19, 2026

Copy link
Copy Markdown
Member

The bug

_start_peer_sync_thread() is called from exactly one place — inside _serve_http(). The estate that loop builds (_PEER_SYNC_STATE, _KNOWN_PROFILES) is plain process memory. But mempalace_mesh_peers ships in every transport, including the stdio servers agents actually connect through, and those processes never run a sync round.

So they answered from a permanently empty estate. Observed from a stdio client against a live three-replica mesh:

"peers": [
  {"name": "windows", "url": "http://…:8765"},
  {"name": "blade",   "url": "https://…"}
],
"unnamed_origins": ["rep_6983fcd7a464", "rep_fac93be106f3"],
"origin_profiles": {"rep_537a9141eb94": {}}

No reachable, no last_success_at, no remote_version_vector, no profile — all of which the tool description promises. Both configured peers landed in unnamed_origins ("origins known only transitively") because a peer's replica_id is learned during a sync and nothing else supplies it. Meanwhile the hub for the same palace had the complete picture.

Reproduced against a fresh HTTP server, which shows it is purely about whether a sync round has run in this process:

peer keys origin_profiles
before first sync round name, url self only
after first sync round name, url, reachable, replica_id, remote_version_vector, profile, last_success_at, last_error, last_error_at, last_pulled_events, last_pulled_artifacts all 3 replicas

A stdio process never reaches the second row.

The fix

The sync loop publishes the estate to mesh_state.json in the per-palace server state directory — the same place the bearer token and serverinfo.json already live, for the same "the hub records something other local processes read" reason, with the same 0600 and the same pid-liveness trust model. Written to a temp name and renamed, so a reader in another process never observes a half-serialized estate; the file is rewritten every round.

_mesh_peers_payload merges the published estate underneath any in-process state, per peer, so the process that actually syncs keeps reporting its own fresher observation and every other process reports the hub's instead of nothing.

Publishing happens once per round rather than per peer — the estate is only coherent after every configured peer has been attempted.

New field

estate_source — additive, nothing renamed or removed (the desktop viewer consumes this payload):

"estate_source": {"in_process": false, "published_at": "2026-08-19T20:45:20Z", "writer_alive": true}

A crashed hub leaves a last-known-good estate. That is worth showing — "last seen as" beats a blank node — but must not be read as live, so writer_alive marks it rather than the file being deleted on shutdown.

Verified end to end

A process with no sync loop, reading an estate a hub published:

peer windows reachable= True replica= rep_fac93be106f3
peer blade   reachable= True replica= rep_6983fcd7a464
origin_profiles: ['rep_6983fcd7a464', 'rep_537a9141eb94', 'rep_fac93be106f3']
unnamed_origins: []
estate_source: {"in_process": false, "published_at": "…", "writer_alive": true}

Tests

Seven in TestPublishedEstate (tests/test_logstream_sync.py), all failing on develop:

  • non-syncing process surfaces the published estate — the regression itself
  • a peer named by the estate no longer reads as a transitive origin
  • in-process state wins over the file (hub's own fresh "peer went down" is not masked by a stale reachable)
  • peers.json tokens reach neither the file nor the payload
  • file is 0600 with no .tmp left behind
  • dead writer still shows the estate but reports writer_alive: false
  • missing and malformed files degrade quietly

Full suite: 4315 passed, 31 skipped. Lint and format clean.

@igorls
igorls requested a review from milla-jovovich as a code owner August 19, 2026 20:49
Copilot AI lite review requested due to automatic review settings August 19, 2026 20:49
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@igorls
igorls force-pushed the fix/mesh-estate-visible-to-all-clients branch from 3fa5707 to a779002 Compare August 19, 2026 20:49
The peer sync loop is started only by _serve_http, and the estate it
builds (_PEER_SYNC_STATE, _KNOWN_PROFILES) is process memory. But
mempalace_mesh_peers ships in every transport, including the stdio
servers agents connect through, and those processes never run a sync
round.

So they answered from a permanently empty estate: every peer reduced to
a bare name and url with no reachable, last_success_at,
remote_version_vector or profile; origin_profiles holding only this
node; and configured peers reported as unnamed_origins -- "known only
transitively" -- because their replica_id is learned during a sync and
nothing else supplies it. The hub next door had all of it.

The sync loop now publishes the estate to mesh_state.json in the
per-palace server state directory after every round, alongside the
token and serverinfo that already use that directory for exactly this
"hub records something other local processes read" purpose. 0600, and
written to a temp name then renamed so a reader in another process
never observes a half-serialized estate.

_mesh_peers_payload merges the published estate underneath any
in-process state, per peer, so the process that actually syncs keeps
reporting its own fresher observation and every other process reports
the hub's instead of nothing.

The new estate_source field says where the reading came from:
in_process, published_at, and whether the publishing hub is still
alive. A crashed hub leaves a last-known-good estate, which is worth
showing -- "last seen as" beats a blank node -- but must not be read as
live.

peers.json tokens are not in the estate and never reach the file; both
are asserted.
@igorls
igorls force-pushed the fix/mesh-estate-visible-to-all-clients branch from a779002 to 3ea3b26 Compare August 19, 2026 21:08
@igorls
igorls merged commit db836b7 into develop Aug 19, 2026
9 checks passed
@igorls
igorls deleted the fix/mesh-estate-visible-to-all-clients branch August 19, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants