Skip to content

refactor(v1): reuse Pydantic adapters across validation - #2233

Merged
xeophon merged 3 commits into
mainfrom
agent/reuse-pydantic-type-adapters
Aug 4, 2026
Merged

refactor(v1): reuse Pydantic adapters across validation#2233
xeophon merged 3 commits into
mainfrom
agent/reuse-pydantic-type-adapters

Conversation

@xeophon

@xeophon xeophon commented Aug 3, 2026

Copy link
Copy Markdown
Member

Overview

Uses reusable Pydantic adapters where V1 handles schema-shaped data, while keeping shallow transport JSON parsing lightweight.

Details

  • shares rubric criteria and verdict validation between rubric and agentic judges, including finite composition weights
  • reuses cached adapters for typed trace output and rollout state channels
  • validates legacy tool definitions through their full Pydantic schema
  • replaces manual numeric config checks with native finite and positive Pydantic types

Note

Low Risk
Mostly validation consolidation and adapter reuse; scoring semantics stay the same with stricter rejection of NaN/inf weights and malformed verdicts at config/parse time.

Overview
Pulls rubric criteria loading and verdict scoring into load_criteria and score_verdicts in rubric.py, so the plugged RubricJudge and agentic-judge env share one path instead of duplicated Criterion models and inline parsing in finalize().

Config and rubric fields now reject bad numbers at parse time: FiniteFloat on judge/score weights, CriterionWeight (non‑negative, finite) on rubric weights, PositiveInt for RLM summarize_at_tokens, and max_criteria with ge=1. Agentic verdict JSON is validated through RubricVerdicts before scoring.

Performance/refactor: functools.cache wraps TypeAdapter for CLI episode read/write; each RolloutSession exposes a cached state_adapter for GET/PUT /state; legacy v0→v1 bridging validates tool defs via a shared TOOLS_ADAPTER with OnErrorOmit instead of a manual loop.

Reviewed by Cursor Bugbot for commit 04a2390. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Centralize rubric validation and reuse Pydantic TypeAdapter instances across v1

  • Introduces load_criteria and score_verdicts helpers in rubric.py to centralize criteria loading and verdict validation, replacing duplicated inline logic in the judge env and RubricJudge.
  • Adds CriterionWeight (non-negative, finite) and tightens field constraints on JudgeConfig, ScoreConfig, and RubricJudgeConfig to reject NaN/inf weights and non-positive max_criteria at parse time.
  • Caches TypeAdapter construction via cache(TypeAdapter) in output.py and adds a state_adapter cached property to RolloutSession so adapters are built once and reused.
  • Behavioral Change: AgenticJudgeEnv.finalize and RubricJudge.grade_batch now strictly validate verdict names against the rubric and reject off-menu answers, where previously this may have passed silently.

Changes since #2233 opened

  • Renamed cached TypeAdapter factory and adapter instances from private naming convention to public naming convention [04a2390]

Macroscope summarized b987707.

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Aug 3, 2026
@macroscopeapp

macroscopeapp Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved 04a2390

This refactoring consolidates duplicated Pydantic validation logic into shared functions and replaces manual checks with built-in Pydantic types (FiniteFloat, PositiveInt). The changes are mechanical DRY improvements with no new features or behavioral changes. Open review comments are minor naming style suggestions.

You can customize Macroscope's approvability policy. Learn more.

Comment thread verifiers/v1/judges/rubric.py Outdated
from verifiers.v1.trace import Trace
from verifiers.v1.types import ID

_CriterionWeight = Annotated[float, Field(ge=0, allow_inf_nan=False)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove leading _ from al lthese

Comment thread verifiers/v1/judges/rubric.py Outdated
return v


_CRITERIA_ADAPTER = TypeAdapter(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Comment thread verifiers/v1/judges/rubric.py Outdated
Comment on lines +82 to +84
lambda value: (
value.get("criteria", []) if isinstance(value, dict) else value
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need this?

Comment thread verifiers/v1/judges/rubric.py Outdated
)


def _load_criteria(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no _

@xeophon
xeophon force-pushed the agent/reuse-pydantic-type-adapters branch from d31aa17 to 83efd59 Compare August 4, 2026 12:30
Comment thread verifiers/v1/judges/rubric.py Outdated
Comment thread verifiers/v1/judges/rubric.py

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 83efd59. Configure here.

Comment thread verifiers/v1/judges/rubric.py Outdated
@xeophon
xeophon force-pushed the agent/reuse-pydantic-type-adapters branch from 83efd59 to 010bee9 Compare August 4, 2026 12:37
@xeophon
xeophon force-pushed the agent/reuse-pydantic-type-adapters branch 2 times, most recently from 010bee9 to b987707 Compare August 4, 2026 13:10
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Aug 4, 2026
@xeophon
xeophon requested a review from mikasenghaas August 4, 2026 14:07
@xeophon
xeophon merged commit 67576db into main Aug 4, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants