Skip to content

fix(tui): handle unpaired surrogates in table markers - #1

Closed
tttboy123 wants to merge 8 commits into
mainfrom
fix/2001-tui-lone-surrogates
Closed

tttboy123 wants to merge 8 commits into
mainfrom
fix/2001-tui-lone-surrogates

Conversation

@tttboy123

@tttboy123 tttboy123 commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Context

This is a fork-local draft for the candidate fix discussed in upstream Discussion #2001. It is opened against this fork's main for review only; no upstream pull request will be opened without an explicit maintainer invitation.

No-Ticket: candidate fix for upstream Discussion PrimeIntellect-ai#2001

Changes

  • Replace unpaired UTF-16 surrogates with U+FFFD at the table-marker encoding boundary.
  • Preserve valid surrogate pairs such as emoji.
  • Keep the marker format and public APIs unchanged.
  • Add public-path unit coverage and an end-to-end Markdown.render() regression.
  • Add the required TUI changelog fragment; its PR attribution will be updated if an upstream PR is invited.

Validation

  • Red: both the marker-level and Markdown-render tests failed on upstream main with URIError: URI malformed.
  • Green: marker-level regression passed, 1 test and 0 failures.
  • Green: end-to-end Markdown regression passed, 1 test and 0 failures.
  • npm run check passed, including formatting, linting, type checking, installer checks, and browser smoke checks.

tttboy123 and others added 8 commits September 4, 2026 02:05
…rimeIntellect-ai#1986)

* fix: count busy descendants in the agents-view subagents indicator

The "N subagents running" indicator only tallied direct children whose
row classified as running, so a busy grandchild under an idle
intermediate child left every ancestor reading "N subagents" with a zero
count. The tally is now a bottom-up pass over the built rows - the live
per-descendant truth - replacing the direct-child increment.

Indicator only, per policy: a session is Running when it works itself;
idle ancestors of busy subtrees stay in Idle. Within Idle they now rank
above plain idle rows, and a collapsed group's summary row renders its
running count in the success color instead of dimmed so the busy subtree
is discoverable without expanding.

Diagnosis credit: Vincent Bailly (VincentBailly#9) traced
the stale hasRunningRlmChildren snapshot and the row-walk mechanism;
this adopts his walk for the indicator while rejecting the section
promotion.

RES-1253

* fix: tally busy descendants before heartbeat promotion inflates sections

The descendant tally read row sections after
propagateHeartbeatStateToAncestors had promoted idle ancestors to
running, so a heartbeat-active grandchild counted its promoted parent
too. The tally now runs before the promotion pass and counts
intrinsically busy rows only; once that propagation pass is deleted
(PrimeIntellect-ai#1967) the ordering is a no-op.

RES-1253

* fix: make the descendant tally iterative so deep chains cannot overflow

The recursive tally threw RangeError on a deep child chain just by
opening the agents view. The nesting loop assigns every row at most one
parent, so the tree is a forest: a reverse breadth-first pass over one
work list computes the same bottom-up counts iteratively, with the
pre-promotion ordering kept.

RES-1253

* fix: classify running from the session's own work, not delegated children

Second half of the RES-1253 policy, user-approved: Running means the
session you enter is doing work now. Delegated child work no longer
classifies a session as running - the busy-descendant badge, count, and
idle ranking from the first half carry the delegation signal.

hasRunningRlmChildren loses its section-classification role in both
owners: classifySessionRosterStatus composes busy from the session's own
activity/isSessionActive, and isActiveSessionBusy (the worker's activity
axis) no longer holds a settled parent at "working" for its children.
The "subagents running" status label dies with the section it had to
agree with. The field itself stays on the wire, and isSessionSummaryBusy
keeps it on purpose for its residency and shutdown-safety consumers
(worker eviction snapshots, empty-draft eviction, busy client-owned
session counts, daemon stop confirmation).

RES-1253

* chore: note the running-means-the-session policy in the changelog fragment

* fix: split display activity from live-work safety in the busy predicates

isActiveSessionBusy served two meanings after the classification change:
the display activity axis (session's own work only, correct) and worker
recovery plus draft-discard (where a running RLM child is live work that
dies with the worker). The recovery journal recorded busy:false for a
settled parent with a running child, so a worker death skipped its
interruption record and notice.

The shared predicate is deleted and each meaning gets one named owner:
activeActivityForSession reads the session's own isSessionActive
directly, and hasLiveSessionWork (own turn or running RLM child) backs
recordWorkerRecoveryState and isDiscardableDraft. No caller can grab the
wrong meaning blind.

RES-1253

* fix: gate destructive agent actions on live work, not display section

Since Running means the session's own work, an idle parent of a running
crew was branching into the delete path: killSubagent chose stop-vs-
delete from the display section and stopAgentForDeletion treated
activity==='working' as the only live work. Destructive actions are
safety consumers of the display/safety split: one row-level hasLiveWork
(own section running, busy descendants, or the wire running-children
flag) now drives the stop-first branch, the deletion flow, and the
confirmation verb.

Also trimmed the tally comment to current behavior.

RES-1253

* fix: stop live descendants when cancelling a settled child run

Stopping a subagent row whose own run already settled reported
'Subagent already finished' while a nested descendant kept running:
cancelRlmChildRun only cancelled the targeted run itself, and a fully
released child (run removed, session retained) was never matched by id
at all. Cancellation now descends where the tree lives: a settled target
stops every running or queued run in its retained session's subtree
(cancelRunningRlmDescendants, mirroring hasRunningRlmChildren's walk),
and the returned flag stays truthful so the UI says stopped only when
something stopped.

RES-1253

* chore: trim comments and fold overlapping pins on the indicator branch

Comment pass: multi-line narration cut to one-line invariant guards
(destructive live-work gate, residency-vs-section busy split, worker-
death live work) and code-readable notes deleted. Test pass: the direct-
child indicator pin folded into the grandchild pin (the general case),
which now also carries the idle-label and wire-flag assertions.

* fix: cancel and descend at every node of the rlm cancel walk

Cancelling a live child run aborts it, and abort cascades into the
child's ACTIVE runs - but running work retained under a settled
descendant of that child was the end of the line: neither the cascade
nor the cancel walk visited it. cancelRlmChildRun now descends into the
target's session after cancelling it, and cancelRunningRlmDescendants
descends at every node instead of treating running runs as leaves, so
cancellation is exhaustive over the subtree regardless of each node's
run state. The returned flag stays truthful.

RES-1253
…ubagent lineage (PrimeIntellect-ai#2021)

resolveRuntimeSessionOptions now whitelists semanticParentSessionId and semanticSpawnedByRequestId, so sessions created through the production runtime factory (daemon workers, runtime hosts) register spawn lineage; the factory is extracted as createDefaultRuntimeFactory so the pin exercises the real path instead of a forward-everything test factory. A split-turn summary slice resolving after a sibling already failed the compaction now settles as request_failed instead of pushing into the drained slice list and staying in-flight forever.
)

* fix: count busy descendants in the agents-view subagents indicator

The "N subagents running" indicator only tallied direct children whose
row classified as running, so a busy grandchild under an idle
intermediate child left every ancestor reading "N subagents" with a zero
count. The tally is now a bottom-up pass over the built rows - the live
per-descendant truth - replacing the direct-child increment.

Indicator only, per policy: a session is Running when it works itself;
idle ancestors of busy subtrees stay in Idle. Within Idle they now rank
above plain idle rows, and a collapsed group's summary row renders its
running count in the success color instead of dimmed so the busy subtree
is discoverable without expanding.

Diagnosis credit: Vincent Bailly (VincentBailly#9) traced
the stale hasRunningRlmChildren snapshot and the row-walk mechanism;
this adopts his walk for the indicator while rejecting the section
promotion.

RES-1253

* fix: tally busy descendants before heartbeat promotion inflates sections

The descendant tally read row sections after
propagateHeartbeatStateToAncestors had promoted idle ancestors to
running, so a heartbeat-active grandchild counted its promoted parent
too. The tally now runs before the promotion pass and counts
intrinsically busy rows only; once that propagation pass is deleted
(PrimeIntellect-ai#1967) the ordering is a no-op.

RES-1253

* fix: make the descendant tally iterative so deep chains cannot overflow

The recursive tally threw RangeError on a deep child chain just by
opening the agents view. The nesting loop assigns every row at most one
parent, so the tree is a forest: a reverse breadth-first pass over one
work list computes the same bottom-up counts iteratively, with the
pre-promotion ordering kept.

RES-1253

* fix: classify running from the session's own work, not delegated children

Second half of the RES-1253 policy, user-approved: Running means the
session you enter is doing work now. Delegated child work no longer
classifies a session as running - the busy-descendant badge, count, and
idle ranking from the first half carry the delegation signal.

hasRunningRlmChildren loses its section-classification role in both
owners: classifySessionRosterStatus composes busy from the session's own
activity/isSessionActive, and isActiveSessionBusy (the worker's activity
axis) no longer holds a settled parent at "working" for its children.
The "subagents running" status label dies with the section it had to
agree with. The field itself stays on the wire, and isSessionSummaryBusy
keeps it on purpose for its residency and shutdown-safety consumers
(worker eviction snapshots, empty-draft eviction, busy client-owned
session counts, daemon stop confirmation).

RES-1253

* chore: note the running-means-the-session policy in the changelog fragment

* fix: split display activity from live-work safety in the busy predicates

isActiveSessionBusy served two meanings after the classification change:
the display activity axis (session's own work only, correct) and worker
recovery plus draft-discard (where a running RLM child is live work that
dies with the worker). The recovery journal recorded busy:false for a
settled parent with a running child, so a worker death skipped its
interruption record and notice.

The shared predicate is deleted and each meaning gets one named owner:
activeActivityForSession reads the session's own isSessionActive
directly, and hasLiveSessionWork (own turn or running RLM child) backs
recordWorkerRecoveryState and isDiscardableDraft. No caller can grab the
wrong meaning blind.

RES-1253

* feat: show token and cost totals on agents view rows

Every agents view row with usage data now reads
`<input>/<output> | $<own> ($<recursive>)` in the details cell, replacing
the message-count detail. Own numbers are the session's whole-file spend
(every branch, forks included, attributed child usage subtracted) - the
money answer, deliberately broader than /usage's current-branch context
answer, and identical for resident and on-disk rows so nothing shifts at
passivation or revival. The recursive total rides the same reverse
breadth-first traversal as the running-subagent tally, summing each
descendant row's own cost exactly once.

Producers: resident summaries compute the total from the already-loaded
entries, memoized until entries change (0.27 ms measured cold at 10k
entries); saved rows accumulate it inside the existing mtime/size-cached
catalog scan, so old files get costs with no new IO and no new cache.
Entries over the scan's max line length keep their existing skip, so a
giant assistant message can undercount slightly. Schema revision 26
publishes the new optional summary and saved-row field.

RES-1258

* feat: final row usage format with arrows and conditional recursive cost

Details cell now reads `↑12k ↓1.2k · $0.42 ($1.10 w/ subagents)`; the
parenthesized total renders only when descendants added spend visible at
cent rounding, so leaf rows read `↑500 ↓50 · $0.68`. Arrow glyphs follow
the existing TUI vocabulary (keybinding hints, token status) and the
width pipeline already treats them as single cells.

RES-1258

* fix: one own-spend truth across scan, loader, and filters

Review round on the usage feature:

- The catalog scan subtracted attributed child usage from disk values
  that (in append-only files) still carry the assistant's ORIGINAL
  model-response usage, double-subtracting and shifting numbers at
  passivation. Full-file rewrites (migrations, forks) can also persist
  the already-folded aggregates, so raw summation is wrong for those
  files instead. The scan now mirrors the loader exactly: fold each
  attribution's aggregate onto its target, then subtract the child usage
  - both disk representations cancel to the same own spend the resident
  computation reports. The equality pin now goes through a real v3
  flushed-then-attributed fixture (the old fixture had no attribution,
  which is why it never caught this).
- Recursive cost was computed over search-filtered rows, so a matching
  parent showed an incomplete "w/ subagents" total. The rollup now comes
  from the unfiltered record hierarchy (computeRecursiveCosts over the
  unified index) and filtering can no longer change the number.
- Zero-spend sessions published 0/0/$0.00 live and nothing once
  passivated; sessionUsageSummaryFrom now returns undefined for zero
  totals, shared by both producers, so rows stay age-only until real
  spend exists.

RES-1258

* fix: gate destructive agent actions on live work, not display section

Since Running means the session's own work, an idle parent of a running
crew was branching into the delete path: killSubagent chose stop-vs-
delete from the display section and stopAgentForDeletion treated
activity==='working' as the only live work. Destructive actions are
safety consumers of the display/safety split: one row-level hasLiveWork
(own section running, busy descendants, or the wire running-children
flag) now drives the stop-first branch, the deletion flow, and the
confirmation verb.

Also trimmed the tally comment to current behavior.

RES-1253

* fix: stop live descendants when cancelling a settled child run

Stopping a subagent row whose own run already settled reported
'Subagent already finished' while a nested descendant kept running:
cancelRlmChildRun only cancelled the targeted run itself, and a fully
released child (run removed, session retained) was never matched by id
at all. Cancellation now descends where the tree lives: a settled target
stops every running or queued run in its retained session's subtree
(cancelRunningRlmDescendants, mirroring hasRunningRlmChildren's walk),
and the returned flag stays truthful so the UI says stopped only when
something stopped.

RES-1253

* feat: render the usage cell unconditionally, zeros included

Every agents view row now renders the full details cell
`↑<up> ↓<down> · $<own> ($<total> w/ subagents)` over four defaulted
numbers - no presence branching, no thresholds, no conditional paren. A
session that never spent reads `↑0 ↓0 · $0.00 ($0.00 w/ subagents)`, and
a zero-usage parent with billed subagents shows its subtree spend
instead of hiding it.

RES-1258

* chore: trim comments and fold overlapping pins on the indicator branch

Comment pass: multi-line narration cut to one-line invariant guards
(destructive live-work gate, residency-vs-section busy split, worker-
death live work) and code-readable notes deleted. Test pass: the direct-
child indicator pin folded into the grandchild pin (the general case),
which now also carries the idle-label and wire-flag assertions.

* chore: cut the usage feature's comments and collapse its pins

Comment pass (>80%): 24 added comment lines down to 4 one-line invariant
guards (loader-fold cancellation in the scan, passivation-invariance on
the resident getter, unfiltered rollup, the revision note). Test pass:
the scan-accumulation and resident-equality pins merged onto one forked-
and-attributed fixture; the descendant-rollup and filtered-total pins
merged with a grandchild; the format pin asserts all four row shapes
from one fixture; redundant zero-case and permutation tests deleted.

* fix: cancel and descend at every node of the rlm cancel walk

Cancelling a live child run aborts it, and abort cascades into the
child's ACTIVE runs - but running work retained under a settled
descendant of that child was the end of the line: neither the cascade
nor the cancel walk visited it. cancelRlmChildRun now descends into the
target's session after cancelling it, and cancelRunningRlmDescendants
descends at every node instead of treating running runs as leaves, so
cancellation is exhaustive over the subtree regardless of each node's
run state. The returned flag stays truthful.

RES-1253

* fix: passivated RLM descendants survive restarts as saved-catalog rows

An RLM child's transcript lives in session-artifacts, which the saved-
session scan never visits. While something resident remembered the child
its row survived; after a supervisor restart an inactive parent's
passivated descendants silently disappeared from the agents view, and
their spend vanished from the parent's subtree total.

Both list_saved_sessions composition points now append live spawn-ledger
children the scan missed, through one shared walk: readSessionInfo per
descendant, parentSessionPath/rlmDepth backfilled from the edge, deleted
edges excluded. The view already renders saved-only descendants, merges
resident duplicates by session identity, and rolls saved usage into
recursive totals, so restored rows restore the money with no view change.

Measured at 300 sessions with 36 passivated descendants: scan 33ms,
merge +9ms (cold ledger, ~0.25ms per descendant).

RES-1262
RES-1258

* feat: compaction and branch-summary calls bill the session they serve

Summarization was invisible money: the compaction model call(s) - two on
a split turn - and branch-summary calls returned usage that was dropped
on the floor, so a session that compacted often under-reported what it
actually cost. The summarizer responses' usage is now folded (both
split-turn slices) and persisted on the compaction/branch_summary entry,
and both own-spend producers add the same term: own spend =
fold(assistant usage) + fold(summarization usage) - attributions,
identical resident and scanned. Old entries without the field fold as
zero; a failed compaction persists nothing, so failed-call billing is
not recoverable - there is no committed entry to carry it.

RES-1258

* fix: harden the passive-descendant catalog merge (round 2)

Three review findings on the merge, all real: the transcript header's
parentSession can point at a forked-away ancestor, so the ledger edge is
now the authoritative topology (family() semantics) for parent and
depth; ledgers are per sessions-dir family, so each catalog request now
reads the ledger of the directory it lists instead of always the
default one; and a broken ledger (unreadable, over-quota) no longer
fails the whole list_saved_sessions response - the merge logs and
returns the scanned catalog.

RES-1262

* chore: consolidate the accumulated review-round pins

The worker-daemon catalog pins from three rounds folded onto one
fixture: restart rows, per-family ledger isolation, and broken-ledger
degradation now share the two-family scaffold. Redundant assertions and
narration comments cut; every behavior keeps exactly one pin.

* test(coding-agent): align unified view regression with usage rows

---------

Co-authored-by: Seth <seth@primeintellect.ai>
…walk (PrimeIntellect-ai#2027)

* fix(coding-agent): cancel RLM subtrees through one iterative visited walk

A finished RLM child lives in BOTH parent maps until passivation: its
run stays in _activeRlmChildRuns while its session is retained in
_rlmChildSessions. The cancel walk descended through both recursively,
so every done intermediate doubled the traversal - 2^k subtree walks on
a chain of finished intermediates, enough to freeze the worker around
depth 25 - and the recursion was depth-bound besides.

One private iterative subtree iterator now owns the traversal: explicit
stack, visited set, per node the union of active-run sessions and
retained sessions, each session yielded once. cancelRunningRlmDescendants
is a flat loop cancelling local runs at each visited session (two
recursive loops deleted); cancelRlmChildRun's miss path walks the same
iterator checking both maps per session (three recursive search
constructs deleted). The teardown cascade, hasLiveWork gates, and the
idempotent _cancelRlmChildRun primitive are untouched.

RES-1265

* fix: route every subtree walker through the one visited iterator

hasRunningRlmChildren shared the cancel walk's exponential dual-
membership hazard - and runs far more often, on every roster flush and
destructive-action gate. It, getRlmChildSession's search, and
deleteInactiveRlmSubagent's not-found fallback now all walk
_rlmSubtreeSessions(), leaving exactly one subtree enumeration in the
session. The counting pin also bounds hasRunningRlmChildren's map
iterations on the same 20-level dual-membership chain and checks the
boolean flips after the cancel. Changelog fragment added; comments cut
to invariant one-liners.

RES-1265

* fix: keep walking past a fruitless id match in the cancel search

Child node ids are only mkdir-unique among siblings (sub- plus eight
random hex chars), so the same id can exist in two subtrees. The
iterative rewrite returned early when a matched run or retained child
had nothing to cancel, making a colliding live run elsewhere
unreachable; the recursive code it replaced fell through and kept
searching. A fruitless match now continues the walk - same false
result when the id is truly finished, correct cancellation when it
collides.

RES-1265
@tttboy123

Copy link
Copy Markdown
Owner Author

Closing because the upstream fix has landed in PrimeIntellect-ai#2028, incorporating PrimeIntellect-ai#2033 with the lone-surrogate handling and regression test. This draft is no longer needed.

@tttboy123 tttboy123 closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants