Repository navigation
crash-visibility: add launch sentinel for Force-Quit detection - #109
Merged
Merged
Conversation
Operator hit two app crashes on 0.43 today. Forensics across IPS, Sentry queued cache, MetricKit, and async.log all came back empty. The failure mode was likely hang+Force-Quit, which SIGKILLs the process and bypasses every in-process crash handler. Add LaunchSentinel: writes an active.json marker to ~/Library/Caches/<bundle-id>/sessions/ at app launch and deletes it in applicationWillTerminate. If the marker is found at next launch, it's archived as unclean-exit-<ts>.json: proof the previous launch ended without calling applicationWillTerminate (Force Quit, SIGKILL, jetsam, in-process fault that Sentry didn't flush). Telemetry-independent by design. The file never leaves the machine, so it works regardless of TelemetrySettings.enabledForCurrentLaunch. Becomes the fifth rail for the planned c11 health CLI (C11-24). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
5 of 7 tasks
* c11-24: scaffold HealthCommand + dispatch wiring (no rails yet)
Adds the c11 health CLI scaffold: empty rails, dispatch wired into
CLI/c11.swift after the help-dispatch (so --help routes through
dispatchSubcommandHelp), case "health" added to subcommandUsage and
usage(). HealthCommandCore.swift is dual-target so c11Tests can exercise
the parsers and renderers in later commits while CLI/HealthCommand.swift
stays a thin shim. Empty result line: "c11 health: nothing in the last
24h across ips, sentry, metrickit, sentinel."
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: IPS rail (scanner + first-line JSON parser + tests)
Reads ~/Library/Logs/DiagnosticReports/*c11*.ips at the top level and
~/Library/Logs/DiagnosticReports/com.stage11.c11*/*.ips per-bundle.
Parses only the first JSON line of each file (Apple CrashReporter
header) to extract incident_id, bundleID, bug_type. Severity = crash;
timestamp = file mtime. Filters by since-window. Missing directories
return [] instead of throwing.
Tests: parser unit cases (crash/hang/malformed), tmp-HOME scanner
covering top-level match, per-bundle subdir match, non-c11 rejection,
since-window filtering, and graceful behavior on missing directory.
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: Sentry queued rail (file count + bundle-id tag, no envelope parsing)
Walks ~/Library/Caches/com.stage11.c11*/io.sentry/ recursively (legacy
c11mux* bundles included per plan note: same product line, stale
envelopes are still legitimate signal). One row per regular file.
Severity = queued; summary = "<bundle>/<filename>"; timestamp = mtime.
Envelope contents are never parsed.
Tests cover: per-bundle envelope detection, legacy c11mux inclusion,
bundles without an io.sentry/ dir, non-c11 bundle rejection,
since-window filtering, and graceful missing-Caches behavior.
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: MetricKit rail (filename grammar parser + tests)
Reads ~/Library/Logs/c11/metrickit/*.json. Filename grammar is
<stamp>-<kind>.json where <stamp> is YYYY-MM-DDTHH-MM-SS.fffZ (UTC,
fixed 24 chars; matches CrashDiagnostics.persist in Sources/SentryHelper.swift).
Kind tokens: metric (skipped), diagnostic, or hyphen-joined
crash<n>/hang<n>/cpu<n>/disk<n> in a fixed order.
Severity mapping: crash-only crash, hang-only hang,
cpu/disk-only resource, multi-category mixed, diagnostic diagnostic.
Tests cover: each kind shape, the metric-skip rule, malformed names
(missing fractionals, unknown tokens, no count suffix), UTC stamp
correctness, end-to-end scan against a tmp HOME, and graceful
missing-directory behavior.
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: launch sentinel rail (unclean-exit JSON parser + tests)
Reads ~/Library/Caches/com.stage11.c11*/sessions/unclean-exit-*.json
written by LaunchSentinel.recordLaunchAndArchivePrevious() on the next
launch after a non-clean exit. Filename ISO stamp is the source of
truth for the timestamp; JSON body provides version, build, and short
commit (8 chars) for the summary. Severity = unclean_exit.
Tests cover: fixture parse against a c11-shaped JSON body, filename
timestamp correctness, since-window cutoff rejection, scan against a
tmp HOME, active.json exclusion, non-c11 bundle exclusion, and
graceful missing-directory behavior.
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: flags + JSON output + diagnostic warnings
Adds parseSinceFlag (30m, 2h, 24h, 3d), bootTime() via sysctlbyname
kern.boottime, parseHealthCLIArgs (parses --since, --since-boot,
--rail, --json, rejects unknown flags and the mutually-exclusive
--since + --since-boot pair), and the two diagnostic-warning helpers:
metricKitBaselineWarning fires when MetricKit count is 0 and the
most recent active.json/unclean-exit-*.json marker has a different
version with launched_at within 24h. Reads launched_at from the JSON
body, falls back to filename stamp, then mtime.
telemetryAmbiguityFooter fires when the production
com.stage11.c11/io.sentry/ cache exists but holds no regular files.
renderHealthTable now formats events into a fixed-width table and
appends a Warnings: footer when warnings are present. CLI shim parses
flags, builds the window from mode + duration, populates rail filter,
collects events, computes warnings, and routes to JSON or table.
Tests cover: parseSinceFlag accept/reject cases, all flag parsing
paths, mutual-exclusion error, bootTime sanity bounds, both warnings'
silent and firing branches with synthetic on-disk state.
Build verified green via xcodebuild -scheme c11. Smoke-tested CLI:
empty result line + telemetry-ambiguity warning fires on this host
(matches plan note's live-machine baseline).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: sandboxed end-to-end runtime test + golden snapshots
Adds c11Tests/CLIHealthRuntimeTests.swift exercising the full pipeline
under a tmp HOME: scaffolds one event per rail (IPS top-level file,
Sentry envelope under per-bundle io.sentry/, MetricKit crash1 file,
unclean-exit archive), then asserts collectHealthEvents returns four
reverse-chronological events with rail counts {ips:1, sentry:1,
metrickit:1, sentinel:1}. JSON shape test verifies the top-level keys
window, rails, events, warnings, and the per-event field grammar.
Adds two golden-file snapshots under Fixtures/health/golden/: empty.txt
(canonical "nothing in the last 24h" line) and four-events.txt (a
fixed-input rendered table). renderHealthTable gains an optional
timeZone parameter so the four-event golden test renders deterministic
UTC output; production callers leave it nil to keep operator-local
time. Locale is pinned to en_US_POSIX inside the formatter so the
golden does not drift across CI runners.
Build verified green via xcodebuild -scheme c11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: resolve running c11 app version for MetricKit baseline warning (B1)
The c11-cli target ships without an Info.plist, so `Bundle.main` is empty
when the binary runs standalone (`/usr/local/bin/c11` or symlinks to a
tagged build). Reading `Bundle.main.infoDictionary["CFBundleShortVersionString"]`
returned nil, and `metricKitBaselineWarning` exited at the
`let curr = bundleVersion` guard before doing any work — the warning
the MVP explicitly ships never fired in production.
Walk up from the executable's path to the first `.app` bundle and read
its CFBundleShortVersionString. Mirrors `CLISocketSentryTelemetry`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: exclude active.json from sentinel marker selection (B2)
`mostRecentSentinelMarker` was treating the current launch's `active.json`
as a candidate baseline alongside archived `unclean-exit-*.json` files.
After a version bump, `active.json` carries the *new* version with the
newest timestamp, so `metricKitBaselineWarning`'s `marker.version != curr`
guard returned nil and the warning never fired.
Restrict the candidate set to `unclean-exit-*.json`. active.json is
by definition the current session and not a prior baseline. While here,
prefer the filename's stamp over reading `launched_at` from the body
(filename is the producer's source of truth) so we can short-circuit
parse errors without a Data load.
Update the warning tests to write `unclean-exit-<stamp>.json` fixtures
and add a regression test asserting `active.json` alone produces no
warning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: telemetry ambiguity footer walks com.stage11.c11* family (B3)
`telemetryAmbiguityFooter` previously probed only the literal production
path `~/Library/Caches/com.stage11.c11/io.sentry`. On a dev machine where
only `c11_DEV` (`com.stage11.c11.debug.*`) ever ran, the probe returned
false and the footer stayed silent in the exact ambiguous state it was
designed to flag. `scanSentryQueued` already walks the `com.stage11.c11*`
family; this helper had diverged.
Walk the same family. Fire the footer when at least one `io.sentry/`
directory exists under a `com.stage11.c11*` bundle and every such
directory is empty. Update the message to reflect the family scope
("Sentry cache empty across c11 bundles") so the footer doesn't claim
"Production" when it fires on a debug-only machine.
Add two regression tests: the debug-only-bundle case (must fire) and
the mixed populated-and-empty case (must stay silent).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: redact home prefix in JSON path output (B4)
`renderHealthJSON` was emitting absolute filesystem paths in each event's
`path` field, leaking the operator's macOS username and cache layout into
output that gets pasted into tickets, Zulip threads, GitHub issues, and
agent transcripts. The table form intentionally omits the path; the JSON
form silently regressed that privacy posture.
Add `redactHomePath(_:home:)` and a `home` parameter to `renderHealthJSON`
so the renderer can rewrite a leading home prefix as `~`. Default the
parameter to `NSHomeDirectory()` so existing call sites keep working;
the CLI shim threads the same `home` it gave to `collectHealthEvents`.
Component-aware: a non-trailing-slash home does not match a sibling
prefix (e.g., `/Users/alice` does not redact `/Users/alice2/...`).
Test: the runtime four-rail JSON test now asserts every event path
starts with `~/` and contains no absolute home, plus five direct unit
tests for the redactor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: --rail duplicate, bootTime fallback diagnostic, --rail help (I1+I2+M1)
I1: `c11 health --rail ips --rail sentinel` previously parsed silently
to `railFilter = .sentinel`, dropping the first rail. Add
`HealthCLIError.duplicateFlag` and reject the second occurrence with a
clear message. The natural operator instinct given four rails is to
think `--rail` accumulates; rejecting is the smallest correct change
that keeps the v1 contract honest. Future v1.1 may widen `railFilter`
to a `Set` and accumulate.
I2: `bootTime()` previously returned a 24h-ago Date on `sysctlbyname`
failure with no log, no throw, no marker. `--since-boot` silently became
the default 24h window without telling the operator. Add an `onFallback`
closure (defaults to stderr) so the operator gets one diagnostic line
naming `errno` before the function returns the fallback Date. Keep the
fallback so the command still produces output.
M1: `--rail` help text now mentions "Specify at most once. Default: all
rails." so a reader can tell from `--help` alone whether omitting the
flag queries one rail or all.
Tests: rejection of duplicate `--rail`; assertion that the bootTime
success path does not call onFallback (because sysctl works on macOS,
this case is the one we'd regress without notice).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: empty-result line reflects --rail and window mode (I3)
`healthEmptyResultLine` was a hardcoded constant claiming "nothing in
the last 24h across ips, sentry, metrickit, sentinel." even when the
operator scoped down with `--rail sentinel` or widened with
`--since-boot`. The output lied about which rails were checked and
about the window.
Promote it to a function `healthEmptyResultLine(rails:window:)` that
joins the actual rails queried and turns the actual `HealthCollectionWindow`
into a phrase ("in the last 30m", "since boot", "in the last 24h").
`renderHealthTable` now accepts `rails` and `window` parameters; both
default so existing callers (and the existing `empty.txt` golden test)
keep their previous output.
CLI shim threads the actual `rails` and `window` it computed into the
table renderer.
Add three regression tests: rail-filter scoping, --since window phrase,
--since-boot phrase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: stable sort tiebreak + lossy UTF-8 first-line read (I4+I5)
I4: `events.sorted { $0.timestamp > $1.timestamp }` only compared
timestamps, but Swift's `sorted` is not stable. Two events with the
same timestamp (sub-millisecond batches across rails are common) could
appear in non-deterministic order, causing `--json` output to flap for
downstream consumers (jq filters, CI gates, fleet aggregators) even
when the underlying disk state is unchanged. Sort by timestamp desc,
then `rail.rawValue` asc, then `path` asc.
Regression test: scaffold MetricKit + two sentinel files at the same
parsed timestamp and assert two back-to-back collections produce the
same event order.
I5: `readFirstLine` was strictly UTF-8 decoding the 8192-byte read
buffer. If the 8192nd byte sat in the middle of a multi-byte sequence
(any IPS payload with non-ASCII metadata), the decode returned nil and
the IPS row fell back to filename-only summary, dropping the bundle /
bug-type / incident-id signal. Switch to `String(decoding:as:UTF8.self)`,
which inserts U+FFFD for malformed bytes; the caller cuts at the first
`\n`, so a trailing replacement char is harmless.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: hoist ISO formatter helper, drop dead severity case (M2+M3)
M2: Three call sites in HealthCommandCore each instantiated their own
`ISO8601DateFormatter` with the same `[.withInternetDateTime,
.withFractionalSeconds]` setup; `mostRecentSentinelMarker` did so per
file inside its loop. Extract `makeFractionalISOFormatter()` and route
all three call sites through it.
M3: `HealthEvent.Severity.metrickit` was never produced by any rail
(`metricKitSeverity(forKind:)` returns one of crash/hang/resource/mixed/
diagnostic). Remove the dead case so future maintainers see only valid
wire-format values.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: add schema_version to health JSON output (EW1)
The first downstream consumer (a jq filter, a CI gate, a fleet
aggregator, an agent that auto-runs `c11 health --json` on crash)
locks in the JSON shape whether we mean it to or not. Add
`"schema_version": 1` now so any future shape change can be branched
on by consumers without guessing at the version.
Exported as `healthJSONSchemaVersion` so the runtime test asserts the
emitted value matches the constant rather than a hardcoded literal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: collapse duplicate error emission in runHealth (S4)
Top-level main() in CLI/c11.swift writes 'Error: <error>\n' to stderr for
any thrown CLIError, and CLIError.description == .message. The previous
runHealth caught HealthCLIError, wrote 'c11 health: ...\n' to stderr,
then threw CLIError with the same description, so operators saw the same
line twice. Drop the explicit stderr write and fold the 'c11 health:'
prefix into the CLIError message. Single line output, prefix preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: render 'unknown' for missing unclean-exit JSON fields (S7)
parseUncleanExitFile previously rendered '? (?) ????????' when the JSON
body was unparseable or missing keys, leaving operators staring at a row
that looks broken. The intent is correct (filename ISO timestamp is the
source of truth, and a corrupt body is intentionally surfaced rather
than dropped, since the file's presence is the signal of an unclean
exit), but the placeholder was illegible.
Replace the placeholders with the literal string 'unknown' so the row
reads 'unknown (unknown) unknown' when the body is unparseable. Add a
code comment at the parse site explaining the design intent so a future
maintainer does not 'fix' the row by silently dropping it. Add two
sentinel-parser tests covering corrupt-body and missing-keys paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: review-fix: IPS rail uses internal timestamp from first-line JSON (S2)
CrashReporter can take seconds-to-minutes to finish writing an IPS file
after the actual crash, so file mtime is a strictly worse proxy for the
event time than the OS-reported timestamp the report itself carries. An
operator running 'c11 health --since 30m' after a recent crash should
not have to guess whether mtime lag pushed the file outside the window.
Extend IPSHeader with an optional 'timestamp' field, parsed from the
first-line JSON via a 'yyyy-MM-dd HH:mm:ss.SSS Z' formatter (Apple
CrashReporter convention, e.g. '2026-05-03 15:19:00.000 -0400'). When
present, scanIPS uses it for both the row's TIME column and the --since
filter; when missing or malformed, it falls back to mtime silently.
Add a one-line note beneath the table header so the operator knows the
TIME column reflects OS-reported event time when available, file mtime
otherwise. Update the four-events golden snapshot accordingly.
Tests: parseIPSTimestamp roundtrip + reject malformed; scanIPS uses
parsed timestamp for since-filtering; scanIPS falls back to mtime when
the JSON has no timestamp; scanIPS falls back to mtime on malformed
timestamp. Update CLIHealthRuntime scaffolding to generate the IPS body
with a fresh timestamp so the test stays in window across calendar days.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* c11-24: validation report + trident review pack
Live-machine smoke of c11 health (verdict PASS) plus the full nine-agent
trident review output that drove the 12 review-fix commits. Both committed so
PR reviewers can read the artifacts without re-running tooling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LaunchSentinel(inSources/SentryHelper.swift). Writes a per-launch JSON marker under~/Library/Caches/<bundle-id>/sessions/and archives the previous launch's marker asunclean-exit-<ts>.jsonifapplicationWillTerminatedid not run.applicationDidFinishLaunching(write) andapplicationWillTerminate(delete) inSources/AppDelegate.swift.Motivated by today's incident: two c11 crashes on 0.43 in a 30-minute window, all four existing rails (IPS, Sentry queue, MetricKit, debug log) came back empty. Most likely cause was hang+Force-Quit; current rails can't see that.
This is the fifth rail for the upcoming
c11 healthCLI (C11-24, plantask_01KQQM2D3V8VJ421RV2Y9XYPQC).Test plan
xcodebuild -project GhosttyTabs.xcodeproj -scheme c11 -configuration Debugexit 0)unclean-exit-*.jsonappears under~/Library/Caches/<bundle-id>/sessions/active.jsonexists🤖 Generated with Claude Code