fix(cli): use subprocess IPC for non-blocking native calls by junhoyeo · Pull Request #10 · junhoyeo/tokscale

junhoyeo · 2025-12-13T16:10:33Z

Summary

Fixes the TUI loader freezing during data loading by running heavy native Rust functions in a subprocess.

Problem

The CLI TUI loader/spinner would freeze for 100-500ms while native Rust functions executed, blocking the Node.js event loop.

Solution

Add native-runner.ts: subprocess that reads method/args from stdin, executes native function, writes result to stdout
Add *NativeAsync() wrappers that spawn subprocess and communicate via stdin/stdout pipes
Update useData.ts to use async subprocess versions

Benefits over temp file approach

Aspect	Temp Files	stdin/stdout
File I/O	2 writes + 2 reads	None
Cleanup	Manual unlinkSync	Automatic
Crash safety	Leftover files	Clean
Security	Data on disk	In memory

Testing

The spinner now animates smoothly throughout the entire data loading process.

Heavy native Rust functions now run in a subprocess to keep the main process event loop free for UI rendering. - Add native-runner.ts: reads method/args from stdin, writes result to stdout - Add *NativeAsync() wrappers that spawn subprocess and communicate via pipes - Update useData.ts to use async subprocess versions Benefits over AsyncTask approach: - Guaranteed non-blocking (separate process entirely) - No file I/O overhead (stdin/stdout pipes) - No cleanup needed (no temp files) - Standard Unix IPC pattern

- Use Buffer.concat instead of string concatenation for proper encoding - Use process.stdout.write instead of console.log to avoid extra newlines - Add stdout preview to error messages for debugging

junhoyeo · 2025-12-13T23:34:06Z

#11

…ined fix(cli): non-blocking native calls via subprocess (combines PR #9 & #10)

…arser tests Map the raw junhoyeo#19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, junhoyeo#9/junhoyeo#10==junhoyeo#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow

#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.#19` is the response model, `usage.#5`/`#9`/`#10` are cacheRead/output/thinking (verified by the invariant `#9 + #10 == #3`, the stored total output), `#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes #712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on #713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw #19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, #9/#10==#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the #707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>

junhoyeo#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.junhoyeo#19` is the response model, `usage.junhoyeo#5`/`junhoyeo#9`/`junhoyeo#10` are cacheRead/output/thinking (verified by the invariant `junhoyeo#9 + junhoyeo#10 == junhoyeo#3`, the stored total output), `junhoyeo#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `junhoyeo#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes junhoyeo#712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on junhoyeo#713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw junhoyeo#19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, junhoyeo#9/junhoyeo#10==junhoyeo#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>

junhoyeo#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.#19` is the response model, `usage.#5`/`#9`/`#10` are cacheRead/output/thinking (verified by the invariant `#9 + #10 == #3`, the stored total output), `#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes junhoyeo#712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on junhoyeo#713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw #19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, #9/#10==#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>

…sion window The agy 1.1.18 `chatModel.#9.#10` payload is 8 bytes whose encoding was never confirmed against a real database, so every candidate reading of it is a guess that has to earn acceptance. The only gate on those guesses was an absolute "is this a believable date" window running from 2020-01-01 to five years out. That is not a meaningful test for a raw integer: read as a nanosecond count the window alone covers ~2% of the u64 range, so trying both byte orders leaves an arbitrary payload — an id, a hash, a duration — a few percent chance of passing as a date. A false accept silently buckets a turn into the wrong day and feeds the server-side monotonic ratchet, which has no correction path, making it strictly worse than the known-wrong session-start dating it replaces. Require every inferred reading to land inside the containing session's own lifetime as well: at or after the session-created stamp less one hour, and at or before now plus one hour. A turn cannot predate its conversation nor happen after we read the file, and that pair of bounds is hours or days wide instead of a decade. When there is no positive anchor to corroborate against, decline inference entirely and let the caller fall back as before. The explicit `#9.#4` Timestamp is untouched: it is a confirmed representation read off real pre-1.1.18 databases, keeps its `ms > 0` filter, takes no session bound, and still outranks the inferred reading. Constraint: `#9.#10`'s encoding is inferred from a field dump, not observed Constraint: mis-dating is uncorrectable downstream; under-dating is not Rejected: tightening only the absolute window | no absolute date range is narrow enough to make a raw 8-byte integer a safe timestamp Rejected: day-wide tolerances | hands back the integer space the session window exists to remove Confidence: high Scope-risk: narrow Directive: the one-hour tolerances are load-bearing and pinned by tests; widening them re-opens the false-accept surface this closes Not-tested: a real agy 1.1.18 database — none was available, which is why the reading is inferred in the first place

…1.1.18 gen_metadata layout (#1196) * fix(antigravity-cli): read the per-generation timestamp from the agy 1.1.18 gen_metadata layout agy 1.1.18 dropped `chatModel.#9.#4`, the `{#1: seconds, #2: nanos}` Timestamp this parser used to date each turn. `#9` now carries `#2` = u64::MAX (an int64 -1 "unset" sentinel) plus a new `#10` holding 8 length-delimited bytes. With `#4` gone every row fell through to the session-created stamp, so on a long-running session every turn was bucketed to the session start date and `--today` reported zero. `#9.#4` is still read first and unchanged, so pre-1.1.18 databases and older installs keep their exact behaviour. When it is absent, `#9.#10` is decoded as a nested Timestamp, a nested message holding the scalar in field 1 (varint or fixed64), or the payload itself as 8 raw fixed64-style bytes in either byte order. Every one of those readings is unit-detected by magnitude and range-checked against 2020-01-01..now+5y before it is accepted; anything outside that window is discarded and the session-created fallback takes over. `#9.#2` is never consulted, and u64::MAX is rejected explicitly so no path can promote the sentinel into a date. Constraint: no agy 1.1.18 install or gen_metadata database available, so `#10`'s encoding is inferred from a field dump in the issue, not observed Rejected: read `#9.#2` as the new timestamp | the only value ever seen there is the u64::MAX unset sentinel Rejected: decode the 8 bytes as an IEEE-754 f64 | any double in the 2^30-ish exponent range reads as a plausible epoch-second count, so it is the one candidate with a non-trivial false-positive rate against a non-timestamp payload Confidence: high that pre-1.1.18 parsing is unchanged; medium that the 1.1.18 reading fires on real data Scope-risk: narrow Directive: keep every inferred reading behind `plausible_epoch_ms` — a wrong date silently corrupts day buckets and the monotonic ratchet, which is worse than the session-start fallback this degrades to Not-tested: a real agy 1.1.18 `gen_metadata` row; if `#10` is neither a timestamp nor decodes in range, behaviour is identical to today's * fix(antigravity-cli): bound inferred generation timestamps to the session window The agy 1.1.18 `chatModel.#9.#10` payload is 8 bytes whose encoding was never confirmed against a real database, so every candidate reading of it is a guess that has to earn acceptance. The only gate on those guesses was an absolute "is this a believable date" window running from 2020-01-01 to five years out. That is not a meaningful test for a raw integer: read as a nanosecond count the window alone covers ~2% of the u64 range, so trying both byte orders leaves an arbitrary payload — an id, a hash, a duration — a few percent chance of passing as a date. A false accept silently buckets a turn into the wrong day and feeds the server-side monotonic ratchet, which has no correction path, making it strictly worse than the known-wrong session-start dating it replaces. Require every inferred reading to land inside the containing session's own lifetime as well: at or after the session-created stamp less one hour, and at or before now plus one hour. A turn cannot predate its conversation nor happen after we read the file, and that pair of bounds is hours or days wide instead of a decade. When there is no positive anchor to corroborate against, decline inference entirely and let the caller fall back as before. The explicit `#9.#4` Timestamp is untouched: it is a confirmed representation read off real pre-1.1.18 databases, keeps its `ms > 0` filter, takes no session bound, and still outranks the inferred reading. Constraint: `#9.#10`'s encoding is inferred from a field dump, not observed Constraint: mis-dating is uncorrectable downstream; under-dating is not Rejected: tightening only the absolute window | no absolute date range is narrow enough to make a raw 8-byte integer a safe timestamp Rejected: day-wide tolerances | hands back the integer space the session window exists to remove Confidence: high Scope-risk: narrow Directive: the one-hour tolerances are load-bearing and pinned by tests; widening them re-opens the false-accept surface this closes Not-tested: a real agy 1.1.18 database — none was available, which is why the reading is inferred in the first place

…ded session anchor (#1202) `read_trajectory_meta` collapsed the decoded `trajectory_metadata_blob` created-at and the file mtime into one `i64`, and that number was used both as the per-row timestamp fallback and as the trust anchor bounding the inferred agy 1.1.18 `chatModel.#9.#10` reading. A file mtime is always positive, so the `session_timestamp <= 0` guard that was supposed to decline inference without an anchor almost never fired: databases with a missing or undecodable metadata blob ran inference against a window built on the last write to the file. An opaque payload decoding near the mtime was accepted, and a genuine older turn was rejected for sitting below it. Report the created-at as `Option<i64>` alongside the effective fallback and thread it through `parse_gen_metadata` into `generation_timestamp_ms` and `inferred_epoch_ms`. With no decoded created-at the `#9.#10` inference is skipped entirely and the row keeps the fallback dating it had before the 1.1.18 layout was handled. The confirmed `#9.#4` Timestamp path returns before any of this and is unchanged. The existing missing-anchor test only passed 0 and -1, neither of which a real database produces. Adds an end-to-end SQLite test covering both shapes that yield no created-at (table absent, and a blob carrying no `#2`) with a payload the old mtime-anchored window would have accepted. Constraint: `#9.#10`'s 8-byte layout is inferred from a field dump, not read off a real database, so an accepted reading must be corroborated Rejected: Keep one i64 and sentinel the mtime case | any in-band sentinel is a valid epoch-ms value Confidence: high Scope-risk: narrow Directive: `session_window_ms` must only ever be handed a decoded created-at; do not reintroduce the mtime as an anchor Not-tested: a real agy 1.1.18 database (none available; `#9.#10` remains inferred)

junhoyeo added 2 commits December 14, 2025 01:10

fix: improve stdin/stdout buffer handling in subprocess IPC

2d1aebd

- Use Buffer.concat instead of string concatenation for proper encoding - Use process.stdout.write instead of console.log to avoid extra newlines - Add stdout preview to error messages for debugging

junhoyeo mentioned this pull request Dec 13, 2025

fix(cli): non-blocking native calls via subprocess (combines PR #9 & #10) #11

Merged

5 tasks

junhoyeo closed this Dec 13, 2025

junhoyeo added a commit that referenced this pull request Dec 13, 2025

Merge pull request #11 from junhoyeo/fix/subprocess-native-async-comb…

d50c967

…ined fix(cli): non-blocking native calls via subprocess (combines PR #9 & #10)

junhoyeo deleted the fix/subprocess-native-runner branch December 14, 2025 02:40

junhoyeo restored the fix/subprocess-native-runner branch December 15, 2025 20:28

junhoyeo deleted the fix/subprocess-native-runner branch December 15, 2025 20:28

cubic-dev-ai Bot mentioned this pull request Aug 25, 2026

fix(antigravity-cli): read the per-generation timestamp from the agy 1.1.18 gen_metadata layout #1196

Merged

Navigation Menu

fix(cli): use subprocess IPC for non-blocking native calls - #10

fix(cli): use subprocess IPC for non-blocking native calls#10
junhoyeo wants to merge 2 commits into
mainfrom
fix/subprocess-native-runner

junhoyeo commented Dec 13, 2025

Uh oh!

junhoyeo commented Dec 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

junhoyeo commented Dec 13, 2025

Summary

Problem

Solution

Benefits over temp file approach

Testing

Uh oh!

junhoyeo commented Dec 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant