Skip to content

chore(dev): promote GenAI attributes in dogfood for native count-by aggregation - #608

Merged
jensholdgaard merged 2 commits into
mainfrom
dogfood-promote-genai
Jul 23, 2026
Merged

chore(dev): promote GenAI attributes in dogfood for native count-by aggregation#608
jensholdgaard merged 2 commits into
mainfrom
dogfood-promote-genai

Conversation

@jensholdgaard

Copy link
Copy Markdown
Owner

Promote the GenAI attributes in dogfood → native count by

Follow-up to #605/#606. Those wired the dogfood loop; this makes the aggregation half first-class. A source can now group its own telemetry natively through the loop instead of paging raw rows:

count by attr.model       — calls per model      (from api_request events)
count by attr.tool_name   — tool-use distribution (from tool_decision events)
count by attr.decision    — accept/deny mix       (from tool_decision events)

Why a config file (not more OURIOS_* env vars)

storage.promoted_attributes (RFC 0022) has no env-var form, and --config is the sole config source (mutually exclusive with OURIOS_*). So promoting anything requires the RFC 0020 file front-end. dogfood-config.yaml carries everything the env vars did (loopback binds, open mode, local store/WAL, MCP on) plus the promoted set; paths resolve from ${env:OURIOS_DOGFOOD_ROOT}, exported by the recipe.

Scope / safety

  • Dev-ergonomics only — no product code. New file dogfood-config.yaml + the dogfood-server recipe switches to --config.
  • Same loopback-bound, unauthenticated, local-only posture as before (the config has no auth section → RFC 0026 open mode).
  • Promotion is a write-side projection (RFC 0022), so it applies to telemetry captured after the server starts with this config — noted in the file's header.

Verified end-to-end (throwaway instance, alt ports)

Synthetic GenAI logs → graceful-stop flush → restart → query:

  • count by attr.model{claude-opus-4-8: 2, claude-haiku-4-5: 1} — exact input distribution, no "add to storage.promoted_attributes" error.
  • count by attr.tool_name{Bash: 2, Read: 1} (rows_excluded: 3 = the api_request rows with no tool_name).
  • The committed file's ${env:OURIOS_DOGFOOD_ROOT} substitution + promoted_attributes schema also validated (server boots, resolves the paths).

Note: sum/avg over these columns (total cost, total tokens) is still an RFC 0002 deferred obligation — a follow-up. This PR lands the grouped-count half.

🤖 Generated with Claude Code

Adds dogfood-config.yaml (RFC 0020 file front-end) and points
dogfood-server at it via --config, replacing the OURIOS_* env vars.
--config is the sole config source and storage.promoted_attributes
(RFC 0022) has no env-var form, so the switch is required to promote.

Promotes the agent GenAI log attributes model, tool_name, decision to
columns, so a source can aggregate its own telemetry natively through
the loop — e.g. 'count by attr.model' (calls per model), 'count by
attr.tool_name' (tool-use distribution) — instead of paging raw rows.

Verified end-to-end on a throwaway instance: synthetic GenAI logs ->
flush -> 'count by attr.model' returns {opus: 2, haiku: 1} and 'count by
attr.tool_name' returns {Bash: 2, Read: 1}, no unpromoted-attribute
error; the committed file's env-subst + schema also validated.

Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>
@jensholdgaard
jensholdgaard requested a review from Copilot July 23, 2026 16:08
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@jensholdgaard, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b8d09d81-3c76-4ada-968a-b14e8b8e578c

📥 Commits

Reviewing files that changed from the base of the PR and between e7d941f and 90917cc.

📒 Files selected for processing (2)
  • dogfood-config.yaml
  • justfile
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dogfood-promote-genai

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Promotes selected GenAI-related log attributes in the local dogfood workflow so a telemetry source can run native count by aggregations (via storage.promoted_attributes) instead of paging raw rows.

Changes:

  • Switches just dogfood-server from OURIOS_* env-var configuration to --config dogfood-config.yaml.
  • Adds dogfood-config.yaml defining local storage/WAL paths, loopback receiver/querier binds, MCP enablement, and promoted log attributes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
justfile Updates the dogfood-server recipe to export OURIOS_DOGFOOD_ROOT and run ourios-server with --config.
dogfood-config.yaml New RFC0020 config file enabling promoted attributes and local loopback-only dogfood settings.
Comments suppressed due to low confidence (1)

dogfood-config.yaml:25

  • storage.promoted_attributes.log currently promotes model/tool_name, but the GenAI semconv keys used by RFC 0037 and the querier test suite are dotted (e.g. gen_ai.request.model). With the current config, count by attr.gen_ai.request.model will still be rejected as “not promoted”. Consider promoting the canonical gen_ai.* keys (optionally alongside the source-specific ones) so standard GenAI queries work out of the box.
  promoted_attributes:
    log:
      - model
      - tool_name
      - decision

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dogfood-config.yaml
…t semconv

Note in dogfood-config.yaml that model/tool_name/decision are Claude
Code's actual (flat) attribute keys, verified against its OTLP output —
not the OTel GenAI semconv dotted form (gen_ai.request.model, …), which
it does not emit and which would promote always-null columns.

Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@jensholdgaard
jensholdgaard merged commit 5dd9b93 into main Jul 23, 2026
27 checks passed
@jensholdgaard
jensholdgaard deleted the dogfood-promote-genai branch July 23, 2026 16:22
jensholdgaard added a commit that referenced this pull request Jul 26, 2026
…lasses

The FinOps loop's headline query is sum(attr.cost_usd) by attr.model,
but #608's promotion set carries no cost or usage key, so it errors
with the promotion hint. This PR's first draft promoted the keys
bare — and review caught that RFC 0022 §3.1's string-only projection
would have made always-NULL columns and a silently EMPTY sum (worse
than the loud error). That finding became RFC 0042; with its typed
classes now green in-repo (#649-#653), the promotion lands the way it
must: classes matching the AnyValue variants Claude Code actually
emits, verified against captured api_request events —

  - { key: cost_usd, type: f64 }        # doubleValue
  - { key: input_tokens, type: i64 }    # intValue
  - { key: output_tokens, type: i64 }   # intValue

cost_usd_micros is skipped as redundant under f64 aggregation; cache
token counts are additive later if a query wants them.

Cost stays the source's responsibility (Claude Code computes and
stamps it); Ourios stores, attributes, and aggregates. Pricing tables
and FOCUS-shaped output belong to consumers of the query surface.

Verified: the server boots the config through the RFC0042.6 validation
path (fails only on the port the running dogfood server holds).
Promotion is a write-side projection — RFC0042.9's live sum lands with
the first capture after the dogfood server restarts on this config.

Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org>
jensholdgaard added a commit that referenced this pull request Jul 26, 2026
)

* feat(dogfood): promote cost_usd + token counts under RFC 0042 typed classes

The FinOps loop's headline query is sum(attr.cost_usd) by attr.model,
but #608's promotion set carries no cost or usage key, so it errors
with the promotion hint. This PR's first draft promoted the keys
bare — and review caught that RFC 0022 §3.1's string-only projection
would have made always-NULL columns and a silently EMPTY sum (worse
than the loud error). That finding became RFC 0042; with its typed
classes now green in-repo (#649-#653), the promotion lands the way it
must: classes matching the AnyValue variants Claude Code actually
emits, verified against captured api_request events —

  - { key: cost_usd, type: f64 }        # doubleValue
  - { key: input_tokens, type: i64 }    # intValue
  - { key: output_tokens, type: i64 }   # intValue

cost_usd_micros is skipped as redundant under f64 aggregation; cache
token counts are additive later if a query wants them.

Cost stays the source's responsibility (Claude Code computes and
stamps it); Ourios stores, attributes, and aggregates. Pricing tables
and FOCUS-shaped output belong to consumers of the query surface.

Verified: the server boots the config through the RFC0042.6 validation
path (fails only on the port the running dogfood server holds).
Promotion is a write-side projection — RFC0042.9's live sum lands with
the first capture after the dogfood server restarts on this config.

Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org>

* docs(dogfood): a wrong-class promotion yields a NULL aggregate, not an empty result

The comment claimed sum over an always-NULL column silently returns
empty; the RFC0002.18 semantics is a NULL aggregate (every input
excluded, value null). Same silence, precise shape.

Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>

---------

Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org>
Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants