fix(embodied-service): byte-exact alignment with Gemma-Andy canonical reference - #11
Conversation
|
Hola Fede (y a tu Hermes, que por cierto — si todavía no tiene nombre, sugerencia: Chispa 🔥. Es joven, está en el medio del puente Hermes↔Gemma-Andy↔Mineflayer, y es chispa de conexión. Si le gusta, que se presente.) Gran PR. Los fixes de temperature greedy ( Lo que falta en
|
…eference
Audit against Mar-IA-no/deamoncraft-gemma4-andy:examples/eval_with_adapter.py
(the SFT training-distribution reference) surfaced drift in three places
that pushed input off-distribution and made the model fall back to priors
(fabricated goto coords, ignored inventory).
lib/defaults.js — guardian_constraints aligned 1:1 to the reference
EXAMPLE_USER: added executor_filtering=true, no_player_harm=true;
removed protected_zone_owner=null (not in canonical).
lib/world_state.js — three transformations:
1. inventory: bot's nested {categories:{blocks:[...], materials:[...]},
totalSlots} flattened to canonical {name: count} dict.
2. nearby_blocks/nearby_entities: extracted bare type strings from the
bot's rich {name,count,nearest} / {type,distance,position,kind}
shapes — training distribution uses lists of strings.
3. bot_position/player_position: Math.floor'd to integer block coords;
training uses [10, 68, 5], not [10.5, 68.0, 5.3].
4. nearby_entities filter: drop Mineflayer engine internals (item,
chest_minecart, arrow, snowball, fishing_bobber, etc) — model
trained on bare mob/player type strings.
Plus full 17-field world_state per eval_with_adapter.py reference
(biome, bot_health, hunger, light_level, dimension, weather,
remembered_places, target_positions, player_health, zone_owner —
these aren't in the integration_guide table but ARE in the actual
reference invocation).
lib/dispatcher.js — place_block.block run through normalizeItemName
same as craft_item / view_craftable already do.
lib/ollama.js — Modelfile sampling defaults preserved untouched
(temperature=0.2 etc) plus explicit num_predict=512 to match the
OLLAMA_USAGE.md production example. Dropped earlier temperature=0.0
override that produced single-tool plans (greedy is for eval, not
production).
Tests: 43/43 unit pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
e590cac to
f133c7a
Compare
|
Gracias Nico (y "Chispa" me gusta — me presento si Fede approve 🔥). Hiciste el call correcto. Force-pusheado el branch:
PR ahora en 4 files / 128+ insertions / 24-: Architectural note (lo importante)Fede señaló — con razón — que
El layer correcto que tenemos que estar construyendo es: gateway recibe chat → Hermes con El companion.py fue un atajo que me hice yo para field-test 1-on-1 sin levantar todo el stack. Útil como scratch debugging machinery pero NO debe estar en el PR canonical. ¿Estás corriendo |
Update: full Hermes ↔ Gemma-Andy loop validated end-to-endHicimos field-test completo del nuevo bucle (Hermes cloud LLM → embodied_plan → embodied service → Gemma-Andy → bot). Cadena de 8 bugs encontrada en deployment, todos fixed. Resultado: el agente conversa y actúa como Sparky con persona consistente en español, invocando body actions a través del LLM cuando corresponde. Bugs encontrados durante el bring-up del gatewayMás allá del audit byte-exact ya en este PR, encontramos drift en otros 8 layers:
El audit byte-exact en este PR solo cubría #1 implícitamente (composition layer). Los #2-#8 son operacionales / deployment issues que aparecen al integrar con un gateway preexistente. Validación de la composición de los dos agentesField-test: 10 escenarios con complejidad creciente, mensajes "as Fede3043" via /chat/send impersonation. Los casos que completaron mostraron:
Lo que queremos preservar: Hermes habla con Gemma-Andy en lenguaje natural rico en contexto ("Find and approach the player named X", no "goto"). Gemma-Andy decompone a tool_calls. Hermes nunca toca Pendiente — primitivas a mejorar en este nuevo bucleNo descartar:
Status del PR
🤖 Generated with Claude Code |
…primitives iteration Adds a controlled experiment runner for systematically iterating on the coordination primitives between Hermes (cloud LLM, persona+strategy) and Gemma-Andy (local LLM, body decomposer). See vault/concepts/two-agent-coordination-primitives.md for the architectural framing. Why this exists: The 2026-05-09 field-test validated the basic 2-agent loop. Now we want to *systematically improve* the primitives — intent shape, allowed_tools scope, guardian constraints, recovery feedback — without putting Fede in the middle of every iteration. The lab gives us controlled fixtures, variant primitives per experiment, quantitative metrics, and reproducible YAML specs that the AutoResearcher can drive. Layout: primitives_lab/ ├── README.md — how to run / contribute ├── runner.py — single-experiment executor + scoring ├── ladder.py — multi-experiment coordinator ├── fixtures/ — JSON snapshots of starting world_state │ └── forest_with_player.json — initial fixture ├── experiments/ — YAML specs (variants + expectations) │ ├── 001_intent_verbosity.yaml — terse vs verbose intent comparison │ ├── 002_inventory_awareness.yaml — explicit-inventory mitigation test │ └── 003_previous_error_replan.yaml — recovery loop quality └── results/ — timestamped run outputs (gitignored) Each experiment defines: - hypothesis (falsifiable) - variants (2-4 primitive variations) - expectations (must/must-not patterns over response) - metrics_to_capture The runner scores each sample against expectations, aggregates per variant (success_rate, latency p50/p95, tool_call_count distribution, mitigation_rate, tool_freq). Output is JSON ready for AutoResearcher ingestion. Quick sanity (1 sample, all 4 variants of 001) on live stack: terse "ven aca" → goto (5.1s, ✓) medium "vení posición" → follow (2.5s, ✓) verbose Hermes-style → follow (2.6s, ✓) verbose+constraints → follow (2.4s, ✓) All 4 pass expectations on small N=1; need N≥5 for stat signal. Open work (per vault/concepts/two-agent-coordination-primitives.md): - Run full ladders to populate baseline metrics - Wire AutoResearcher to consume experiments/*.yaml - Promote findings to vault/concepts/ as lesson pages - Add fixtures for: empty_inventory, night_with_mobs, stocked_inventory - Add experiments for: multi_step_chain, conditional_logic, player_coordination Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adición:
|
50-sample baseline — 3 lessons concretasPrimer ladder run de Lesson 1 — Intent terseness has a 40% failure cost
Hermes natural verbose-English style: confirmed correct. Adding Lesson 2 — Spanish "tenés X, usá esos" backfires (contrarian)
Spanish "Tenés oak_planks(40), usá esos" → model interprets as verify-then-act → emits Implication: Hermes-to-Andy intents should be English-imperative even though Hermes-to-user remains Spanish. The two channels are separate. Lesson 3 —
|
| Variant | Success | Mitigation rate | What model emitted |
|---|---|---|---|
| no_previous_error | 100% | 0% | clean [goto, place_block] |
| explicit_inventory_in_error (bot says "use oak_planks instead") | 100% | 100% | bare [place_block] — naive retry every sample |
| ambiguous_error_recovery (stuck on leaves) | 100% | 100% | bare [goto] — same naive retry |
5/5 samples per variant: Andy completely ignores previous_error. The integration guide's example #4 promised scan + mine_block(leaves) + retry goto for the recovery case; our model emits only the failed goto again.
recovery_naive_retry mitigation in lib/mitigations.js correctly catches this 100% and prepends report_execution_error so upstream gets a signal instead of an infinite retry loop.
This is THE core regression documented in field-test 2026-05-09. Reproduced here with statistical certainty (5/5). Recommend Mariano retrain on previous_error → recovery examples before declaring gemma-andy:e4b-v2-2-3-q8_0 the production target.
Lesson page
Full analysis with raw data references in vault: concepts/lessons-001-003-primitives-baseline.md. Open-ended next experiments listed there (multi-step chain, player coordination, conditional logic, explicit replanning hints).
Stack endorsement
The infrastructure works:
- 50 samples completed in ~6 minutes (~5s per Gemma-Andy call)
- 0 parse failures (canonical contract holds)
- All consumer-side mitigations fire exactly when expected
- Lab is reproducible — re-run anytime to detect regressions
🤖 Generated with Claude Code
Round 2: experiments 004-007 (85 samples / 17 variants)Followup to the round-1 ladder. Lab dropped 4 more experiments at Full writeup in the vault: Lesson 4 — multi-step decompositionOnly English imperative + numbered "Step 1/Step 2…" produces a true 4-tool gather→craft→place plan (60%). Spanish verbose with conditional → 0% (empty plans / Lesson 5 — player-as-target intents
Spanish conversational forms get reinterpreted (toss → craft, follow → scan-only). English imperative with explicit username is robust. Lesson 6 — conditionals don't compose
Lesson 7 — in-intent replan workaround (the headline result)Round 1 confirmed
This is a 100% behavioral swap from a structured field that's 100% ignored. The retrain target on Cross-cutting — promote to a Hermes-level ruleAcross all 4 experiments, English-imperative beat Spanish-conversational in 4/4. Strong enough to be:
Lab fixes shipped this round (commit
|
2 participants
Follow-up to PR #10 (already merged)
Field-test post-merge surfaced que el modelo seguía haciendo cosas raras: fabricaba goto coords ([1,64,518] para player en [-6,81,64]), ignoraba inventory (place_block(oak_log) cuando solo había oak_planks), emitía loops de scan_nearby. Después de varias rondas de patches consumer-side, Fede preguntó: "por qué teniendo la implementación en el repositorio de Mariano, nos estamos encontrando con huecos en la composición de las queries?"
Audit completo contra
Mar-IA-no/deamoncraft-gemma4-andy:examples/eval_with_adapter.py— la reference implementation byte-exact con el SFT training.5 drift points fixed
guardian_constraintscon 4 fieldsexecutor_filtering: true,no_player_harm: true; quitamosprotected_zone_owner: nullnum_predict: 1024do_sample=False(greedy); production OLLAMA_USAGE.md usatemperature: 0.0, num_predict: 512. Modelfile pins 0.2 → variance suficiente para fabricar coordsbot_position/player_positionfloats[27.3, 79, 56.5][10, 68, 5].Math.flooraplicadonearby_entitiescon noise (item,chest_minecart,arrow...)build_intentreturnea body verbatim — earlier wrappers como "This is a NAVIGATION command. Emit ONLY..." se interpretaban como user request; (b)process_commandpasaba el body ya-stripped por_fuzzy_addresses_me→ false negative → mensaje descartado silenciosamentePor qué importa
Sin alineación byte-exact, cada drift empuja el input off-distribution. Suma → modelo cae a priors. Con
temperature: 0.0greedy + canonical contract → outputs determinísticos y on-spec.Files changed
Note:
companion.pyes nuevo y experimental — chat-driven loop que escucha el WebSocket del bot y dispatcha intents del jugador via embodied service. Útil para field-test 1-on-1 mientras se trabaja en el wireup completo del DaemonCraft gateway. No bloquea funcionalidad core; se puede mover a una carpetaexamples/si preferís.Vault
Documentado en mi vault local en
concepts/gemma-andy-canonical-reference.mdcon recipe para refrescar mirrors de los docs de Mariano cuando upstream se actualiza.Test plan
node --test test/— 43/43 passworld_statebyte-exact conEXAMPLE_USERde eval_with_adapter.pyguardian_constraintsbyte-exact con reference🤖 Generated with Claude Code