feat: Fleet client - #256

Open

dzorlu wants to merge 92 commits into

huggingface:mainfrom

fleet-ai:deniz/fleet_client

Commits on Dec 12, 2025

fleet integartion step 0

Deniz
authored and
Deniz
committed
38edfc2
updated README

Deniz
authored and
Deniz
committed
f67bc43
readme update

Deniz
authored and
Deniz
committed
164853c

Commits on Dec 13, 2025

another iteraton

Deniz
authored and
Deniz
committed
935826f

Commits on Dec 18, 2025

readme

Deniz
authored and
Deniz
committed
7c09d5b

Commits on Jan 17, 2026

Add FleetTaskEnv for Gymnasium-compatible task environments

authored and
Deniz
committed
eac8d0e

Commits on Jan 21, 2026

conb

Deniz
authored and
Deniz
committed
7efae22

Commits on Jan 22, 2026

Add __init__.py to envs package for pip install compatibility

Deniz
authored and
Deniz
committed
791a071
fix: Remove default image_type="mcp" to use standard ECR images

authored and
Deniz
committed
a24eaf6
fix: Add data_key and data_version params to from_fleet()

authored and
Deniz
committed
9df9351
fix: Combine data_key and data_version into Fleet SDK format

authored and
Deniz
committed
46d1779
fix: Remove seed parameter from reset() call

authored and
Deniz
committed
7852847

Commits on Jan 23, 2026

feat: Add request_timeout_s parameter to FleetTaskEnv

authored and
Deniz
committed
7697fcd

Commits on Jan 25, 2026

Fix: make reset() a sync wrapper around reset_async()

authored and
Deniz
committed
98ec667
Fix: extract text content from MCP CallToolResult

authored and
Deniz
committed
b62c7e6
Add tests for tool extraction and reset behavior

authored and
Deniz
committed
9ef1dec
feat: fetch tools in __init__, simplify reset_async

authored and
Deniz
committed
1a3e27b

Commits on Jan 26, 2026

fix: add detailed logging for verifier execution failures

authored and
Deniz
committed
336ff02
fix: Use Fleet SDK Task.verify_detailed() for verifier execution

authored and
Deniz
committed
abb6936
Fix: fetch tools lazily in reset_async to avoid asyncio.run in async context

Deniz
authored and
Deniz
committed
ced5eca

Commits on Jan 28, 2026

fix: add retry with backoff for MCP list_tools and log errors

Deniz
authored and
Deniz
committed
d23f08d
fix: add retry logic to call_tool for connection failures

authored and
Deniz
committed
f938ab9

Commits on Jan 29, 2026

debug: add logging for call_tool to trace success/failure paths

Deniz
authored and
Deniz
committed
a2f3531
fix: unwrap ExceptionGroup to show actual error cause

Deniz
authored and
Deniz
committed
a08cb6d
debug: add logging for Fleet instance creation timing

Deniz
authored and
Deniz
committed
9806eb8

Commits on Feb 5, 2026

fix: Add retry logic to Fleet.make() for transient failures

authored and
Deniz
committed
584d613

Commits on Feb 8, 2026

feat(fleet_env): add ContextManager for context management tools

authored and
Deniz
committed
93638df

Commits on Feb 10, 2026

Use image_type='mcp' for computer_use tasks

Deniz
authored and
Deniz
committed
a1ac1a7

Commits on Feb 11, 2026

fix: Fetch tools for all modalities (tool_use and computer_use)

authored and
Deniz
committed
1b66bab
Filter to only computer tool for computer_use modality

authored and
Deniz
committed
f40cd91

Commits on Feb 12, 2026

fix: Handle ImageContent in MCP and filter tools for computer_use

authored and
Deniz
committed
3ac6f82
Add initial screenshot on reset for computer_use tasks

authored and
Deniz
committed
461413e

Commits on Feb 13, 2026

debug: Log actual screenshot result format from MCP

Deniz
authored and
Deniz
committed
6e4a522
fix: Handle Fleet MCP base64_image format for VL models

Deniz
authored and
Deniz
committed
80be63b
Remove debug logging from task_env.py

Deniz
authored and
Deniz
committed
7a1a755
test: Add tests for base64_image format handling

Deniz
authored and
Deniz
committed
0c0b535

Commits on Feb 16, 2026

Add reset_timeout_s to avoid blocking on broken manager APIs

authored and
Deniz
committed
675e652

Commits on Feb 19, 2026

feat: Add env_key to error logs for better debugging

authored and
Deniz
committed
9e7390d

Commits on Feb 22, 2026

fix: Add HTTP-level timeouts to streamablehttp_client calls

authored and
Deniz
committed
e2486cb

Commits on Feb 25, 2026

fix: Raise on exhausted list_tools retries, allow retry on next reset

authored and
Deniz
committed
8370cd1

Commits on Feb 26, 2026

feat: Add Logfire error tracking to fleet env

authored and
Deniz
committed
272b0b3

Commits on Feb 27, 2026

fix(telemetry): consistent schema with env_key, env_version, task_key, modality

authored and
Deniz
committed
216d16a
feat(telemetry): add fleet_mcp_tool_error for MCP server errors

authored and
Deniz
committed
34dd0d9
fix(telemetry): set context BEFORE Fleet.make() so init failures have full context

authored and
Deniz
committed
03fbb92
fix: address bugbot issues in PR #6

authored and
Deniz
committed
9d12c3e
fix: add hard timeout to MCP operations to prevent hanging

authored and
Deniz
committed
0bc9cfd
fix: async Fleet.make() to prevent event loop blocking

authored and
Deniz
committed
b061b4e
fix: enrich fleet_mcp_tool_error with env:version and step info

authored and
Deniz
committed
d8c5ddc

Commits on Feb 28, 2026

fix: suppress noisy logfire console output and tracebacks

authored and
Deniz
committed
e627cd0
Fix telemetry dashboard: count init failures as rollouts, add total_steps

authored and
Deniz
committed
73b81b5
Add fleet_provisioning_completed telemetry event with provisioning_time_s

authored and
Deniz
committed
1e9bfce

Commits on Mar 3, 2026

Fix MCP endpoint routing, telemetry gap, and retry config

authored and
Deniz
committed
f4ed59b
fix: call_tool retry used non-existent retry_base_delay attr

authored and
Deniz
committed
327c782
feat: auto-select instance TTL based on modality

authored and
Deniz
committed
887fd1f

Commits on Mar 4, 2026

Emit fleet_rollout_completed on close() for orphaned rollouts

authored and
Deniz
committed
84de403
Simplify orphaned rollout stop reasons to max_steps / abandoned

authored and
Deniz
committed
77b9d6a
Add Fleet telemetry section to README

authored and
Deniz
committed
540530a
Increase tool_use TTL from 600s to 900s to reduce 502s from instance expiry

authored and
Deniz
committed
199f67f
fix: use asyncio.to_thread(Fleet.make()) instead of AsyncFleet.make()

authored and
Deniz
committed
33d53c9
fix: wrap sync blocking calls in asyncio.to_thread() to unblock event loop

authored and
Deniz
committed
0d37811

Commits on Mar 7, 2026

feat: add trace upload utilities for eval rollouts

authored and
Deniz
committed
f86fa49

Commits on Mar 8, 2026

fix: Convert OpenAI image_url blocks to Fleet ingest format for proper UI rendering

authored and
Deniz
committed
290600e
fix: Pass reward as score to ingest API so sessions complete

authored and
Deniz
committed
c99c1e5
Add hint-based reward for solver RL training (Options B, C, D)

authored and
Deniz
committed
fc0508f
Revert "Add hint-based reward for solver RL training (Options B, C, D)"

Deniz
authored and
Deniz
committed
31fa602

Commits on Mar 11, 2026

Merge pull request #6 from fleet-ai/deniz/fleet-logfire
dzorlu
authored
b8ee588
feat: add partial reward support behind flag

authored and
Deniz
committed
cc5bf37
Merge pull request #9 from fleet-ai/feat/partial-reward
dzorlu
authored
7c5d64f

Commits on Mar 12, 2026

feat: Run verifier at close() for orphaned rollouts

authored and
Deniz
committed
06dcdd4
Merge pull request #10 from fleet-ai/feat/close-verifier
dzorlu
authored
baa192a

Commits on Mar 13, 2026

feat: Add submit_final_answer synthetic tool for carlisle tasks

authored and
Deniz
committed
d651a01
merge: resolve conflicts with deniz/fleet_client (_reward_computed)

authored and
Deniz
committed
5111c78
Merge pull request #11 from fleet-ai/deniz/submit-final-answer
dzorlu
authored
cf91b04

Commits on Mar 15, 2026

feat: Add TaskEvaluator for task generation inner loop

authored and
Deniz
committed
2290cb8
refactor: rewrite TaskEvaluator to use Fleet harness (POST /v1/jobs)

authored and
Deniz
committed
3c5f902
Fix model ID format: use provider/model prefix for Fleet harness

authored and
Deniz
committed
fafc3ea
Make _poll_job async to avoid blocking the event loop

authored and
Deniz
committed
6bc5939
Fix model ID mismatch between Fleet API and configured models

authored and
Deniz
committed
0695f71
fix: Remove unused json import, defensive copy DEFAULT_MODELS

authored and
Deniz
committed
0051e92
Merge pull request #12 from fleet-ai/deniz/task-evaluator
dzorlu
authored
8605142

Commits on Mar 16, 2026

Expose verifier feedback properties for hint generation

authored and
Deniz
committed
3566e7c

Commits on Mar 17, 2026

Merge pull request #14 from fleet-ai/deniz/hint-feedback
dzorlu
authored
438c7b3

Commits on Mar 18, 2026

feat: Add DB query methods to FleetEnvClient

authored and
Deniz
committed
a5c82e0

Commits on Mar 20, 2026

fix: properly await async describe/query on AsyncFleet env handles

authored and
Deniz
committed
270e010

Commits on Mar 29, 2026

fix: handle null base64_image from Fleet MCP screenshot responses

authored and
Deniz
committed
534fd30

Commits on Apr 14, 2026

feat: browser_use modality provisions browser sidecar

authored and
Deniz
committed
4362dd7

Commits on Apr 15, 2026

feat: browser_use provisions browser lease sidecar

authored and
Deniz
committed
76dcd59

Commits on Apr 16, 2026

fix: use browser API fallback token

authored and
Deniz
committed
725ddba

Commits on Jun 16, 2026

trace: point trace job + log ingest at the real Fleet API

authored and
Deniz
committed
f9c01a5

Commits on Jun 17, 2026

trace: forward verifier_execution_id to make Fleet UI score show up

authored and
Deniz
committed
88f2667
fleet-env: use async verify_detailed + retry on 502/503 + warn on missing exec_id

authored and
Deniz
committed
0eb891b

Commits on Aug 26, 2026

fix: restore cluster_name at browser lease call sites

authored and
Deniz
committed
df97268