Token report: quiet-window-safe refresh timer templates; Loki query_source and effort-split panels

adoption/templates/systemd/token-report-refresh.service

@@ -0,0 +1,140 @@
+# Drafted, not run: no host has loaded, started or enabled this unit (the
+# coordinator deploys it after review; this repository's worktree that
+# authored it made no host changes). @REPOSITORY@ is this repository's
+# checkout path and @REPORT_CONFIG@ is the private token-report
+# configuration path created by `token_manifest.py init-config` (see
+# tools/token-report/README.md, "Create one explicit configuration and
+# refresh"); both must be substituted before installing this unit, matching
+# this repository's existing placeholder convention
+# (adoption/templates/systemd/codex-broker-reaper.service's @REPOSITORY@;
+# observability/native-data/native-data.service.example's @PRIVATE_CONFIG@
+# for the same kind of per-user `--config` value). This unit references no
+# systemd specifier (such as the %h just named as an example); the only %/$
+# text in any directive is the %%/$$ escaping inside ExecCondition= below,
+# which exists solely to survive systemd's own specifier/variable expansion
+# of the unit file and is explained in that directive's own comment. (Other
+# %/$ characters below, such as the "$REPORT_TOOLS" shell variable quoted
+# just below, appear only in comment prose, never in a directive systemd
+# itself parses.)
+#
+# The configuration this points at must give `--rtk`, `--headroom` and, if
+# set, "toon" their absolute installed paths (for example
+# "$REPORT_TOOLS/rtk-0.50.0/rtk" and
+# "$REPORT_TOOLS/headroom-0.37.0/bin/headroom" at `init-config` time, or
+# edited directly into an existing config's "rtk", "headroom" and "toon"
+# keys). `initialize_config` (tools/token-report/token_manifest.py) stores
+# those fields verbatim -- unlike jcodemunch/mcporter, which it resolves
+# once with `shutil.which()` at init time -- so a bare name, such as the
+# README's interactive-shell example (`--rtk rtk --headroom headroom`),
+# only resolves through that shell's own exported PATH and would fail under
+# a systemd --user manager's own minimal PATH. There is no `--toon` flag at
+# all: set an absolute "toon" path directly in an existing config's JSON.
+#
+# The same restricted PATH applies beyond --rtk/--headroom/toon. ExecStart=
+# below pins /usr/bin/python3 directly, so that interpreter itself must be
+# 3.11 or later (tools/token-report/README.md's own "Python 3.11+ is
+# required"). When this configuration also selects jcodemunch/mcporter
+# capture or sets "toon": mcporter is a `#!/usr/bin/env node` script
+# requiring Node >=24 per its own package.json; the optional toon
+# executable (tools/token-report/README.md, "Exact artifact comparisons")
+# is also a `#!/usr/bin/env node` script -- its own package.json sets no
+# engines.node floor, but its bin/toon.mjs still needs a working node to
+# run that same kind of shebang. `initialize_config`'s `shutil.which()`
+# resolves the configured --mcporter value to an absolute path once, at
+# `init-config` time, but running that absolute path -- or toon's own
+# absolute path -- still re-resolves `node` through `env` against *this
+# service's own* PATH, not the shell `init-config` ran in. Confirm a
+# qualifying `node` already resolves under `systemctl --user
+# show-environment` before installing; with "toon" configured, a `node`
+# missing from this unit's PATH fails *every* scheduled run (an issue
+# makes `refresh` exit nonzero), not just one capture among several. If
+# needed, prepend a qualifying `node`'s directory to the manager's own
+# full PATH (read it first with `systemctl --user show-environment`) in an
+# explicit Environment=PATH=... line -- replacing the whole PATH instead
+# of prepending to it would also change what any other bare-name command
+# in this unit resolves to (ExecCondition= below calls /bin/date by
+# absolute path for exactly this reason, so it is unaffected either way).
+# A separate, absolute "node" key set directly in config.json (parallel to
+# "rtk"/"headroom"/"toon") only covers this reporter's own internal
+# tokenizer-count invocation used while comparing toon output; it does not
+# change what toon's own `env node` shebang above resolves at run time.
+#
+# No HEADROOM_WORKSPACE_DIR redirect: this refresh only ever calls
+# `headroom savings --json` (no `--reset`), and the installed headroom-ai
+# 0.37.0 source shows that whole call path is read-only. `cli/savings.py`'s
+# `savings()` command calls only `savings_ledger.aggregate_savings()`, which
+# calls `_read_events()`; the ledger-compaction helper `_maybe_compact()` is
+# reachable only from the separate write path, `append_event()`
+# (savings_ledger.py:184), which this refresh never calls. The Click group
+# callback that runs before every subcommand (`cli/main.py`) also only
+# reads: `settings_store.load()` is a fail-open `path.read_text()` with no
+# write branch (settings_store.py:852-861), and `apply_to_environ()` only
+# edits this process's own in-memory `os.environ`. With no
+# HEADROOM_WORKSPACE_DIR override, both of those reads
+# (`~/.headroom/settings.json` and `~/.headroom/savings_events.jsonl`, the
+# latter via `paths.savings_events_path()`) resolve to the real, shared
+# `~/.headroom` state -- which is what this refresh's dashboard row is
+# supposed to reflect in the first place. No `RuntimeDirectory=` is declared
+# either, because nothing left in this unit references a `%t`-rooted path
+# for systemd to create or clean up.
+#
+# The three Headroom variables below guard the one call in that same group
+# callback that is not a pure read: `maybe_check_async()`, an opt-out
+# background update-check network probe (`update_check.py`). Verified
+# against the installed source:
+#   - `update_check.py:57-70` -- `is_update_check_enabled()` is "Disabled by
+#     HEADROOM_UPDATE_CHECK=off, offline mode (HEADROOM_OFFLINE), stateless
+#     mode ... or any CI environment"; both HEADROOM_UPDATE_CHECK=off and
+#     HEADROOM_OFFLINE=1 are set here for defense in depth, so
+#     `maybe_check_async()` returns before any network call or cache write.
+#   - `telemetry/beacon.py:11,84` -- the telemetry beacon separately honors
+#     DO_NOT_TRACK=1 (`dnt = os.environ.get("DO_NOT_TRACK", ...)`).
+# This mirrors the guard already recorded for the interactive `headroom`
+# launcher (a separate change) without touching that launcher or ~/.headroom
+# itself.
+#
+# ExecCondition enforces the Sat/Sun 05:00-09:00 quiet window at the moment
+# this service actually runs, not only at schedule-authoring time: `man
+# systemd.timer` on this host documents that a calendar timer which elapses
+# while the system is sleeping "will not be acted on immediately, but once
+# the system is later resumed it will catch up and process all timers that
+# triggered while the system was sleeping" -- this is unconditional on
+# calendar timers and separate from Persistent= (see the paired .timer's own
+# comments), so a resume can still start a run inside the window. `/bin/date
+# +%%u%%H` prints the ISO weekday (6=Saturday, 7=Sunday) concatenated with
+# the 24h hour, so e.g. Saturday 06:30 prints "606". It is called by its
+# absolute path -- like /bin/sh itself above -- so this guard resolves
+# regardless of this unit's own PATH (see the Node/mcporter/toon PATH
+# discussion above): a bare `date` this guard could not find would silently
+# fail the window open instead of closed, since `$(date ...)` then expands
+# to an empty string and a `case` with no matching pattern exits 0, not 1.
+# `$$(...)` and `%%` escape systemd's own `$`/`%` expansion inside this unit
+# file (systemd.service(5), "To pass a literal dollar sign, use $$";
+# systemd.unit(5) specifier table, "%%" -> "Use %% in place of %"). Exit
+# code 1 makes systemd skip ExecStart for this run without marking the unit
+# failed (systemd.service(5) ExecCondition=: "when an ExecCondition=
+# command exits with exit code 1 through 254 ... the remaining commands are
+# skipped and the unit is not marked as failed"). Verified for this repo on
+# 2026-09-25: for all 168 (weekday, hour) combinations the pattern skips
+# exactly Sat/Sun 05:00-08:59 and runs everywhere else, and the guard's
+# result is unchanged when this unit's own PATH cannot resolve any bare
+# command name (see tests/test_token_report_refresh_units.py).
+#
+# This unit has no [Install] section: it is triggered only by
+# token-report-refresh.timer, matching this repository's existing
+# timer-triggered oneshot convention (adoption/templates/systemd/
+# codex-broker-reaper.service and observability/native-data/
+# native-data.service.example + its paired .timer.example).
+[Unit]
+Description=Refresh the portable native token-report lifetime manifest
+
+[Service]
+Type=oneshot
+UMask=0077
+NoNewPrivileges=true
+TimeoutStartSec=900
+ExecCondition=/bin/sh -c 'case "$$(/bin/date +%%u%%H)" in 60[5-8]|70[5-8]) exit 1;; esac'
+Environment=HEADROOM_UPDATE_CHECK=off
+Environment=HEADROOM_OFFLINE=1
+Environment=DO_NOT_TRACK=1
+ExecStart=/usr/bin/python3 @REPOSITORY@/tools/token-report/token_manifest.py refresh --config @REPORT_CONFIG@

adoption/templates/systemd/token-report-refresh.timer

@@ -0,0 +1,45 @@
+# Drafted, not run: see token-report-refresh.service in this directory for
+# the environment guard, the specifier notes and the deployment notes that
+# also apply here.
+[Unit]
+Description=Quiet-window-safe trigger for the token-report lifetime refresh
+
+[Timer]
+# Four fixed local times a day -- 03:15, 10:15, 16:15, 22:15 -- chosen so no
+# *on-schedule* run ever falls in a Sat/Sun 05:00-09:00 local quiet window
+# (or any other day's 05:00-09:00, since the schedule does not vary by
+# weekday): 03:15 is the last run before 05:00 and 10:15 is the first run
+# after 09:00, so the whole band sits strictly between two scheduled ticks.
+# `systemd-analyze calendar --iterations=12 --base-time="2026-09-25 20:30:00"
+# '*-*-* 03,10,16,22:15:00'` on this host (America/New_York, systemd
+# 255.4-1ubuntu8.17) confirms all 12 iterations from that base time through
+# Mon 2026-09-28 16:15 EDT fall only on :03,10,16,22:15, never inside
+# 05:00-09:00 (with 4 ticks/day, 8 iterations would only reach 2026-09-27
+# and cannot demonstrate coverage through the 28th; this file previously
+# understated the iteration count needed).
+OnCalendar=*-*-* 03,10,16,22:15:00
+# Left explicit rather than omitted (systemd's own default is already
+# false): Persistent=true would additionally fire a catch-up run right after
+# a reboot or resume from a fully stopped/powered-off state if a tick was
+# missed while the timer itself was inactive (`man systemd.timer` on this
+# host: "triggered ... immediately if it would have been triggered at least
+# once during the time when the timer was inactive"); that catch-up run is
+# not calendar-aligned and could land inside 05:00-09:00, so it stays off.
+#
+# Persistent= does NOT cover every way a run can slip into the window,
+# though: the same man page separately documents that a calendar timer which
+# elapses while the system is merely *sleeping* (suspend/hibernate, not
+# stopped) "will not be acted on immediately, but once the system is later
+# resumed it will catch up and process all timers that triggered while the
+# system was sleeping" -- unconditionally, regardless of this setting. A
+# host asleep at a scheduled tick that resumes between 05:00 and 09:00 would
+# still fire that catch-up run inside the quiet window. The paired
+# token-report-refresh.service's ExecCondition= is what actually enforces
+# the window at run time for that case; this comment only explains why
+# Persistent= alone is not that guard.
+Persistent=false
+AccuracySec=1min
+Unit=token-report-refresh.service
+
+[Install]
+WantedBy=timers.target

manifests/evidence.json

@@ -32174,8 +32174,8 @@
     },
     {
       "path": "observability/native-data/render.py",
-      "sha256": "3e29ec76e51620a6d130f8dbe90f108b1429fd2b6d8ec5dfd1a9c83d4d2501e1",
-      "bytes": 11151
+      "sha256": "973afa3b840c7151245bd6219a0c1eb3f99e99cb7ea421671a66b3411aaf3820",
+      "bytes": 15024
     },
     {
       "path": "observability/native-data/snapshot.py",
@@ -32859,8 +32859,8 @@
     },
     {
       "path": "tests/test_native_dashboard_data.py",
-      "sha256": "656b44c32b1cf9ce59182ec2a37220b7942206274830ff5824b96c150c022e18",
-      "bytes": 20844
+      "sha256": "2626ca6f28a4ffd84dadabefb68f32ef48ab36d61a8683c51436b68501beee3f",
+      "bytes": 24707
     },
     {
       "path": "tests/test_native_faults_min.py",
@@ -33199,8 +33199,8 @@
     },
     {
       "path": "tools/token-report/README.md",
-      "sha256": "a3daadc82d2c100444c7fc65842bea033182e26b8819ba66a556e54aa3f41e71",
-      "bytes": 12436
+      "sha256": "78d24ac4c9ee810ba748e15ff90a65d5d88a70fa19ee356395d4542d60063aa8",
+      "bytes": 18650
     },
     {
       "path": "tools/token-report/config.example.json",

observability/native-data/render.py

@@ -14,6 +14,17 @@ def latest(kind, entity=None):
             ' == on() group_left() max(last_over_time(' + marker + '))')
 
 
+def by_query_source(field):
+    """Sum a claude-code api_request token field by query_source, a free-form string the
+    client sets per request (recent examples observed live: repl_main_thread:outputStyle:Concise,
+    agent:custom, agent:builtin:workflow-subagent, agent:builtin:general-purpose, agent_summary,
+    compact, away_summary -- not an exhaustive or fixed set). Closer to actual usage than the
+    Prometheus ecosystem_claude_code_token_usage_tokens_total counters, which have been observed
+    to undercount; still overlaps them (both read the same provider usage)."""
+    return ('sum by (query_source) (sum_over_time({service_name="claude-code"} | event_name="api_request"'
+            ' | unwrap ' + field + ' [$__interval]))')
+
+
 def dashboard():
     panels = []
 
@@ -129,15 +140,49 @@ def table(pid, kind, title, y, height, fields, entity=None, widths=None):
     add(9, 'Codex provider telemetry · typed counters', 'timeseries', 0, 52, 12, 8,
         'sum by (token_type) (ecosystem_codex_turn_token_usage_sum)', source='ecosystem-prometheus',
         description='Native exported usage; overlapping cache/input/total fields must not be added. Not a counterfactual saving.')
-    add(10, 'Claude provider telemetry · typed counters', 'timeseries', 12, 52, 12, 8,
+    add(10, 'Claude provider telemetry · typed counters (Prometheus; known to undercount)', 'timeseries', 12, 52, 12, 8,
         'sum by (type) (ecosystem_claude_code_token_usage_tokens_total)', source='ecosystem-prometheus',
-        description='Only native exported samples in the selected range. An inactive client may have no current series; use its archive/report for recorded usage.')
+        description='Only native exported samples in the selected range. This Prometheus counter has been '
+                    'observed to undercount actual usage against the Loki api_request panels below (about half '
+                    'the total on a measured day); the cause is open. Cross-check against the cache-read '
+                    'Loki panel below (id 15): for the completed 2026-09-24 EDT day it reconciled with '
+                    'transcript-based counters (ccusage) within about 1% once query_source=agent_summary was '
+                    'excluded. The other three query_source-split panels (input, output, cache-creation tokens) '
+                    'have not yet been reconciled this way. An inactive client may have no current series; use '
+                    'its archive/report for recorded usage.')
     table(11, 'coverage', 'All selected repositories · recorded acceptance, not live process status', 60, 18,
           ['title', 'coverage_status', 'canonical_receipt_count', 'state', 'source_updated_at', 'timestamp_basis', 'boundary'])
     add(12, 'Sanitized native client activity', 'logs', 0, 78, 24, 10,
         '{service_name=~"Codex Desktop|codex-app-server|claude-code|codex-sdk-receipt"}',
         options={'showTime': True, 'sortOrder': 'Descending', 'wrapLogMessage': True},
         description='Existing native telemetry; prompt/tool bodies are removed upstream in the collector.')
+
+    for pid, field, title, x, y in (
+        (15, 'cache_read_tokens', 'Cache-read tokens by query_source', 0, 88),
+        (16, 'input_tokens', 'Input tokens by query_source', 12, 88),
+        (17, 'output_tokens', 'Output tokens by query_source', 0, 96),
+        (18, 'cache_creation_tokens', 'Cache-creation tokens by query_source', 12, 96),
+    ):
+        add(pid, title + ' (Loki api_request)', 'timeseries', x, y, 12, 8, by_query_source(field),
+            description='Live claude-code api_request events unwrapped and split by query_source, a free-form '
+                        'string the client sets per request (recent examples: repl_main_thread:outputStyle:Concise, '
+                        'agent:custom, agent:builtin:workflow-subagent, agent:builtin:general-purpose, '
+                        'agent_summary, compact, away_summary). Closer to actual usage than the Prometheus '
+                        'typed-counters panel above; still overlaps it (both read the same underlying provider '
+                        'usage) and must not be summed with it or with the savings table. The legend sum equals '
+                        'the range total only when the query step equals $__interval; verify that in Grafana '
+                        'before reading it as a total.',
+            options={'legend': {'displayMode': 'table', 'placement': 'bottom', 'calcs': ['sum']}})
+    add(19, 'Claude provider telemetry · effort split (Prometheus; known to undercount)', 'timeseries', 0, 104, 12, 8,
+        'sum by (effort) (increase(ecosystem_claude_code_token_usage_tokens_total[$__rate_interval]))',
+        source='ecosystem-prometheus',
+        description='Per-interval increase, split by effort level (the metrics pipeline keeps the "effort" '
+                    'datapoint attribute; see observability/collector/collector.yaml). increase() is used rather '
+                    'than a raw sum of the cumulative counter because the collector expires stale series '
+                    '(deltatocumulative max_stale 1h) and sets no metric_expiration, so exposed series appear and '
+                    'disappear with session activity; a plain sum by (effort) of the raw counter follows series '
+                    'lifetimes, not token use. Subject to the same undercount as the typed-counters panel; read it '
+                    'for effort-level shape over the selected range, not an absolute total.')
     return dict(uid='native-foundation-data', title='Native foundation · memory, retrieval and savings',
                 schemaVersion=39, version=4, editable=False, preload=True, timezone='browser', refresh='30s',
                 time={'from': 'now-6h', 'to': 'now'}, tags=['ecosystem', 'native', 'memory', 'tokens'],