Skip to content

feat(telemetry): add Prometheus metrics with feature-gated instrumentation - #35

Merged
jamiepine merged 6 commits into
spacedriveapp:mainfrom
l33t0:feat/prometheus-metrics
Feb 19, 2026
Merged

feat(telemetry): add Prometheus metrics with feature-gated instrumentation#35
jamiepine merged 6 commits into
spacedriveapp:mainfrom
l33t0:feat/prometheus-metrics

Conversation

@l33t0

@l33t0 l33t0 commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add metrics cargo feature flag with optional prometheus dependency
  • Implement global metric registry with 8 handles (counters, histograms, gauges) behind a LazyLock singleton
  • Expose /metrics (Prometheus text format) and /health endpoints via a dedicated axum server on configurable port
  • Instrument 5 call sites: LLM completions, tool execution, memory reads/writes, worker lifecycle
  • All telemetry code compiles out to nothing without the feature flag — zero runtime cost

Metrics

Metric Type Labels
spacebot_llm_requests_total Counter agent_id, model, tier
spacebot_llm_request_duration_seconds Histogram agent_id, model, tier
spacebot_tool_calls_total Counter agent_id, tool_name
spacebot_tool_call_duration_seconds Histogram —
spacebot_memory_reads_total Counter —
spacebot_memory_writes_total Counter —
spacebot_active_workers Gauge agent_id
spacebot_memory_entry_count Gauge agent_id (not yet wired)

Known limitations

  • agent_id and tier labels on LLM metrics are hardcoded to "unknown" (SpacebotModel lacks process context)
  • LLM histogram buckets max at 10s; needs upper buckets for retry/fallback chains
  • memory_entry_count gauge is registered but not instrumented

All tracked as inline TODO comments and documented in METRICS.md.

Test plan

  • cargo check passes with and without --features metrics
  • cargo clippy --features metrics -- -D warnings clean
  • cargo test — 43 tests pass in both configurations
  • Manual: start with metrics.enabled = true, curl localhost:9090/metrics returns valid Prometheus text
  • Manual: verify /health returns 200

l33t0 and others added 6 commits February 19, 2026 02:32
@l33t0 @claude
feat(telemetry): add metrics registry and /metrics HTTP server
d24968a 
Add opt-in Prometheus metrics infrastructure behind a `metrics` cargo
feature flag. Defines metric handles (counters, histograms, gauges) for
LLM requests, tool calls, memory operations, and active workers via a
global registry. Exposes a /metrics endpoint on a configurable port
(default 9090) using axum, spawned as a separate tokio task.

Metric handles are defined but call-site instrumentation is deferred to
a follow-up change.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@l33t0 @claude
feat(telemetry): instrument call sites with metrics counters and hist…
aacd1c4 
…ograms

Wire metric handles into LLM completions (counter + duration histogram),
tool execution (counter + duration via hook timer map), memory operations
(read/write counters), and worker lifecycle (active_workers gauge inc/dec).
All instrumentation is cfg-gated behind the `metrics` feature flag.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@l33t0 @claude
docs(metrics): add METRICS.md reference and example prometheus.yml
3660537 
METRICS.md documents every exposed metric with type, labels,
instrumentation location, cardinality estimates, and known limitations.
examples/prometheus.yml provides a ready-to-use scrape config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@l33t0 @claude
docs(metrics): flag review findings as inline TODO comments
588b4ad 
- LLM histogram buckets max at 10s, need upper buckets for 15-60s range
- memory_entry_count gauge defined but not wired to any call site
- TOOL_CALL_TIMERS can retain orphaned entries on abnormal termination
- LLM metrics use hardcoded "unknown" for agent_id and tier labels

No implementation changes. Comments only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@l33t0
Merge remote-tracking branch 'upstream/main' into feat/prometheus-met…
e55083c 
…rics

# Conflicts:
#	src/config.rs
@jamiepine
Merge branch 'main' into feat/prometheus-metrics
ed3a56e
@jamiepine
jamiepine merged commit e0cb50d into spacedriveapp:main Feb 19, 2026
l33t0 added a commit to l33t0/spacebot that referenced this pull request Feb 21, 2026
@l33t0 @claude
feat(telemetry): complete metrics instrumentation with cost tracking …
ce355c7 
…and per-agent context

Wire the metrics server startup, fix LLM histogram buckets, and resolve all
known limitations from spacedriveapp#35: agent_id/tier labels are no longer hardcoded to
"unknown", memory_entry_count gauge is instrumented, and six new metrics
cover token usage, estimated USD cost, branch/worker lifecycle, process
errors, and memory audit trail.

- Wire start_metrics_server() call in main.rs
- Extend LLM duration buckets to [0.1 … 120s]
- Add agent_id + process_type context to SpacebotModel, wired at all 7 call sites
- Add spacebot_llm_tokens_total (input/output/cached_input)
- Add spacebot_llm_estimated_cost_dollars with static pricing table (src/llm/pricing.rs)
- Add spacebot_active_branches gauge
- Add spacebot_worker_duration_seconds histogram
- Add spacebot_process_errors_total counter with error classification
- Add spacebot_memory_updates_total counter (save/delete/forget)
- Wire memory_entry_count gauge in MemoryStore save/delete
- Add metrics docs page for docs.spacebot.sh
- Update METRICS.md and docs/metrics.md with full inventory

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@l33t0 l33t0 mentioned this pull request Feb 21, 2026
Merged
6 tasks
rktmeister pushed a commit to rktmeister/spacebot that referenced this pull request Mar 11, 2026
@jamiepine
Merge pull request spacedriveapp#35 from l33t0/feat/prometheus-metrics
de74462 
feat(telemetry): add Prometheus metrics with feature-gated instrumentation
rktmeister pushed a commit to rktmeister/spacebot that referenced this pull request Mar 11, 2026
@l33t0 @claude
feat(telemetry): complete metrics instrumentation with cost tracking …
d509d64 
…and per-agent context

Wire the metrics server startup, fix LLM histogram buckets, and resolve all
known limitations from spacedriveapp#35: agent_id/tier labels are no longer hardcoded to
"unknown", memory_entry_count gauge is instrumented, and six new metrics
cover token usage, estimated USD cost, branch/worker lifecycle, process
errors, and memory audit trail.

- Wire start_metrics_server() call in main.rs
- Extend LLM duration buckets to [0.1 … 120s]
- Add agent_id + process_type context to SpacebotModel, wired at all 7 call sites
- Add spacebot_llm_tokens_total (input/output/cached_input)
- Add spacebot_llm_estimated_cost_dollars with static pricing table (src/llm/pricing.rs)
- Add spacebot_active_branches gauge
- Add spacebot_worker_duration_seconds histogram
- Add spacebot_process_errors_total counter with error classification
- Add spacebot_memory_updates_total counter (save/delete/forget)
- Wire memory_entry_count gauge in MemoryStore save/delete
- Add metrics docs page for docs.spacebot.sh
- Update METRICS.md and docs/metrics.md with full inventory

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pasogott pushed a commit to pasogott/spacebot that referenced this pull request Aug 15, 2026
@jamiepine
Record every worker run against its task
1ca4452 
Task spacedriveapp#35, the linkage autonomy needs before it can run the board.

tasks.worker_id names the run executing now and is overwritten by the
next spawn, so a task retried three times remembered only the last one,
and it is cleared on reassignment rather than archived. worker_runs
carries no task reference at all, so nothing could answer "what has
already been tried on this task and how did it end" — the question a
loop has to answer before spawning.

task_worker_runs is that history: append-only, one row per attempt, with
the outcome, who asked for it, and which channel it came from. The
attempt ordinal is allocated inside the transaction so racing spawns
cannot claim the same one, re-recording a worker returns its existing
row so a retried bind is idempotent, and terminal state is written once
so a duplicated completion cannot rewrite how a run ended.

The worker reference carries no foreign key deliberately. Tasks live in
the instance database and worker_runs in the per-agent one, so the link
crosses a database boundary that SQLite cannot enforce. A run whose
worker row was pruned still records that the attempt happened.

Spawning refuses a task that already has a live run. The existing
delegation check is per-channel, so two channels could previously spawn
on the same task without either noticing.

The board renders what has been tried inline, in one query for the whole
board rather than one per task, naming at most three attempts so a
heavily retried task cannot crowd out the rest. GET
/tasks/{number}/attempts exposes the same history, and a worker resolves
back to its task.

Outcomes mirror the worker's own rather than collapsing to
success/failure, so partial and blocked stay distinguishable from failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@l33t0 @jamiepine