Adding ThinkingBox to OpenEnv

.claude/docs/REPO_WALKTHROUGH.md

@@ -89,6 +89,19 @@ envs/
 │       ├── echo_environment.py       # Environment implementation
 │       └── Dockerfile                # Container definition
 │
+├── thinkingbox_env/           # Stateful MCP business-workflow benchmark adapter
+│   ├── client.py                  # Typed trusted-harness client
+│   ├── models.py                  # Public action, observation, and state wire models
+│   ├── openenv.yaml               # Port and FastAPI application entry point
+│   ├── pyproject.toml             # Python 3.12 package and pinned ThinkingBox runtime
+│   ├── README.md                  # Usage, trust boundary, configuration, and citation
+│   └── server/
+│       ├── app.py                    # WebSocket server plus liveness/readiness routes
+│       ├── config.py                 # Runtime settings and installed-package provenance
+│       ├── data_loader.py            # Commit-pinned executable release validation/cache
+│       ├── thinkingbox_environment.py # Native episode orchestration and grading
+│       └── Dockerfile                # Standalone Python 3.12 image
+│
 ├── coding_env/               # Python code execution environment
 ├── chat_env/                 # Conversational environment
 ├── textarena_env/            # Text-based games (TextArena)
@@ -115,6 +128,7 @@ tests/
 ├── envs/                     # Per-environment integration tests
 │   ├── test_echo_environment.py
 │   ├── test_coding_environment.py
+│   ├── test_thinkingbox_env.py  # Focused wire, fidelity, provenance, and evaluator suite
 │   └── ...
 │
 ├── test_cli/                 # CLI command tests
@@ -218,6 +232,7 @@ docs/
     ├── environments/         # Per-environment documentation
     │   ├── echo.md
     │   ├── coding.md
+    │   ├── thinkingbox.md    # Generated from envs/thinkingbox_env/README.md
     │   └── ...
     │
     ├── getting_started/      # Sphinx Gallery executable tutorials
@@ -235,6 +250,10 @@ docs/
     └── _static/              # Static assets (versions.json, etc.)
 ```
 
+ThinkingBox runnable examples live in `examples/thinkingbox/`:
+`example_usage.py` drives one configured episode, and `eval_testlist.py`
+produces canonical repeated-evaluation JSONL with strict coverage validation.
+
 ## Key Files to Know
 
 | File | Purpose |

.github/workflows/test.yml

@@ -96,6 +96,33 @@ jobs:
         env:
           PYTHONPATH: src:envs
 
+  test-thinkingbox-env:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v7
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: ${{ env.UV_VERSION }}
+
+      - name: Set up Python 3.12
+        run: uv python install 3.12
+
+      - name: Install ThinkingBox environment dependencies
+        working-directory: envs/thinkingbox_env
+        run: uv sync --python 3.12 --frozen --all-groups --all-extras
+
+      - name: Run ThinkingBox environment tests
+        run: |
+          uv run --python 3.12 --frozen --project envs/thinkingbox_env \
+            --all-groups --all-extras \
+            pytest tests/envs/test_thinkingbox_env.py -v --tb=short
+        env:
+          PYTHONPATH: src:envs
+
   check-env-docs:
     runs-on: ubuntu-latest
     steps:

docs/source/_toctree.yml

@@ -125,6 +125,8 @@
     title: Reasoning Gym
   - local: environments/tbench2
     title: TBench2
+  - local: environments/thinkingbox
+    title: ThinkingBox
   - local: environments/unity
     title: Unity
   - local: environments/wildfire

docs/source/environments.md

@@ -229,6 +229,13 @@ The OpenEnv community has built a catalog of ready-to-run environments that cove
         <a href="environments/tbench2" class="!no-underline border dark:border-gray-700 px-3 py-1 rounded text-sm hover:shadow">📄 Docs</a>
       </div>
     </div>
+    <div class="border dark:border-gray-700 p-5 rounded-lg shadow">
+      <div class="font-bold mb-2">ThinkingBox</div>
+      <p class="text-sm">Stateful ThinkingBox workflows with isolated MCP tools, agent and simulated-user interaction, and executable evaluation of backend state, side effects, and response requirements.</p>
+      <div class="flex gap-2 mt-3">
+        <a href="environments/thinkingbox" class="!no-underline border dark:border-gray-700 px-3 py-1 rounded text-sm hover:shadow">📄 Docs</a>
+      </div>
+    </div>
     <div class="border dark:border-gray-700 p-5 rounded-lg shadow">
       <div class="font-bold mb-2">Unity</div>
       <p class="text-sm">OpenEnv wrapper for Unity ML-Agents environments, providing access to Unity's RL environments through HTTP/WebSocket interfaces.</p>

docs/source/environments/thinkingbox.md

@@ -0,0 +1,122 @@
+<!-- openenv-source: thinkingbox_env -->
+# ThinkingBox
+
+## Introduction
+
+[ThinkingBox](https://github.com/microsoft/thinkingbox) is an agentic harness for
+defining isolated MCP tool environments, creating stateful scenarios and test
+cases, running an LLM agent with simulated-user interaction, and evaluating
+the resulting backend state, side effects, and response requirements. It can
+support offline evaluation or conversation generation and agentic training
+workflows.
+
+This OpenEnv adapter exposes the public
+[ThinkingBox-Bench](https://arxiv.org/abs/2608.19741) through the standard
+WebSocket reset/step interface. The adapter is currently designed for
+evaluation only.
+
+## Quick Start
+
+Start the externally managed ThinkingBox services described under
+[Deployment Boundary](#deployment-boundary), then run the OpenEnv server:
+
+```bash
+OPENENV_TB_CONFIG=/path/to/thinkingbox.yaml \
+uv run --project envs/thinkingbox_env --frozen server
+```
+
+Connect from a trusted harness:
+
+```python
+from thinkingbox_env import ThinkingBoxEnv
+
+async with ThinkingBoxEnv(base_url="http://127.0.0.1:8901") as env:
+    result = await env.reset("file.py:test_name")
+    tools = await env.list_tools()
+    result = await env.call_tool("tool_name", {"argument": "value"})
+    result = await env.submit_message("Final response")
+```
+
+For native model orchestration, repetition IDs, privacy-reviewed canonical
+JSONL, and strict aggregation, use
+[`examples/thinkingbox/eval_testlist.py`](../../examples/thinkingbox/eval_testlist.py).
+
+## Environment Details
+
+The adapter preserves native ThinkingBox hydration, history replay, tool
+overrides, parallel tool batches, direct responses, simulated users, effects,
+fixtures, `TestContext`, `TestScript`, `Judge`, error latching, and teardown.
+Final rewards are binary pass/fail; infrastructure failures are reported
+separately and never converted into benchmark failures.
+
+The model-facing action surface contains tool discovery, tool calls, and
+assistant messages. A trusted harness may call `finish()`, but finish is not
+advertised in the public action schema. Private user context, initialization
+state, assertion source, credentials, proxy session identifiers, effects, and
+grading internals remain inside the environment.
+
+`seed` is accepted for OpenEnv API compatibility but intentionally unused
+because tasks are selected deterministically by UID. Concurrent sessions are
+stacked independent environment instances, one trajectory per instance; they
+are not multiplexed trajectories.
+
+Executable assets come from the pinned `thinkingbox-bench-v1.0` release in
+[`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data).
+The [Hugging Face `microsoft/ThinkingBox-Bench`
+dataset](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench) is a
+viewer-friendly representation, while GitHub remains the executable source.
+
+## Configuration
+
+The server accepts an ordinary native ThinkingBox YAML configuration through
+`OPENENV_TB_CONFIG` or the reset `config=` argument. The file supplies
+`mcp_proxy`, `orchestrator.agent_model`, `user_model`, `judge_model`,
+`judge_type`, and user-completion behavior; see the
+[ThinkingBox LLM configuration
+guide](https://github.com/microsoft/thinkingbox/blob/main/docs/llm_endpoint_config.md).
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `OPENENV_TB_PROXY_URL` | `http://127.0.0.1:7111` | Session Proxy fallback |
+| `OPENENV_TB_DATASET` | pinned release | Local executable-data override |
+| `OPENENV_TB_DATA_CACHE` | `~/.cache/openenv/thinkingbox_bench` | Verified release cache |
+| `OPENENV_TB_AGENT` | `think` | Native agent definition |
+| `OPENENV_TB_CONFIG` | unset | Native server-side model/proxy config |
+| `OPENENV_TB_PROXY_TIMEOUT` | `120` | Proxy timeout without config override |
+| `OPENENV_TB_DATA_TIMEOUT` | `120` | Release download timeout |
+| `OPENENV_TB_MAX_CONCURRENT_ENVS` | `8` | Independent environment instances |
+
+The client message timeout remains longer than server-side model and episode
+operations so a valid long-running turn is not abandoned prematurely.
+
+## Deployment Boundary
+
+The image starts only the OpenEnv API. It currently needs an externally
+managed ThinkingBox Session Proxy, benchmark MCP servers, Typesense, and agent,
+user-simulator, and judge model endpoints. `GET /health` reports process
+liveness; `GET /ready` reports the observable external dependencies and marks
+scenario-specific Typesense readiness as unobservable from this process.
+
+## Citation
+
+Please cite the paper if ThinkingBox is useful in your experiments.
+
+- Paper: [One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark
+  for Agents in Stateful Business Workflows](https://arxiv.org/abs/2608.19741)
+- Framework: [`microsoft/thinkingbox`](https://github.com/microsoft/thinkingbox)
+- Executable benchmark:
+  [`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data)
+- Viewer dataset:
+  [`microsoft/ThinkingBox-Bench`](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench)
+
+```bibtex
+@misc{li2026successisntreliabilitythinkingbox,
+  title={One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
+  author={Zhuochun Li and Youngmin Ko and Ali Keramati and Nicola Ferri and Susana Palmaz Lopez Pelaez and Liang-Chun Tsai and Calvin Wang and Mirco Milletari and Tuhin Kundu and Vadim Smolyakov and Kjartan Olafsson and Tommy Guy},
+  year={2026},
+  eprint={2608.19741},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2608.19741},
+}
+```

envs/thinkingbox_env/.dockerignore

@@ -0,0 +1,17 @@
+.venv
+__pycache__
+.pytest_cache
+*.pyc
+*.pyo
+logs
+*.log
+output*
+result*
+error*
+*.jsonl
+*.out
+*.output
+*.result
+*.results
+*.error
+*.errors

envs/thinkingbox_env/README.md

@@ -0,0 +1,133 @@
+---
+title: ThinkingBox Environment
+emoji: 🧠
+colorFrom: indigo
+colorTo: blue
+sdk: docker
+pinned: false
+app_port: 8901
+tags:
+  - openenv
+---
+
+# ThinkingBox
+
+## Introduction
+
+[ThinkingBox](https://github.com/microsoft/thinkingbox) is an agentic harness for
+defining isolated MCP tool environments, creating stateful scenarios and test
+cases, running an LLM agent with simulated-user interaction, and evaluating
+the resulting backend state, side effects, and response requirements. It can
+support offline evaluation or conversation generation and agentic training
+workflows.
+
+This OpenEnv adapter exposes the public
+[ThinkingBox-Bench](https://arxiv.org/abs/2608.19741) through the standard
+WebSocket reset/step interface. The adapter is currently designed for
+evaluation only.
+
+## Quick Start
+
+Start the externally managed ThinkingBox services described under
+[Deployment Boundary](#deployment-boundary), then run the OpenEnv server:
+
+```bash
+OPENENV_TB_CONFIG=/path/to/thinkingbox.yaml \
+uv run --project envs/thinkingbox_env --frozen server
+```
+
+Connect from a trusted harness:
+
+```python
+from thinkingbox_env import ThinkingBoxEnv
+
+async with ThinkingBoxEnv(base_url="http://127.0.0.1:8901") as env:
+    result = await env.reset("file.py:test_name")
+    tools = await env.list_tools()
+    result = await env.call_tool("tool_name", {"argument": "value"})
+    result = await env.submit_message("Final response")
+```
+
+For native model orchestration, repetition IDs, privacy-reviewed canonical
+JSONL, and strict aggregation, use
+[`examples/thinkingbox/eval_testlist.py`](../../examples/thinkingbox/eval_testlist.py).
+
+## Environment Details
+
+The adapter preserves native ThinkingBox hydration, history replay, tool
+overrides, parallel tool batches, direct responses, simulated users, effects,
+fixtures, `TestContext`, `TestScript`, `Judge`, error latching, and teardown.
+Final rewards are binary pass/fail; infrastructure failures are reported
+separately and never converted into benchmark failures.
+
+The model-facing action surface contains tool discovery, tool calls, and
+assistant messages. A trusted harness may call `finish()`, but finish is not
+advertised in the public action schema. Private user context, initialization
+state, assertion source, credentials, proxy session identifiers, effects, and
+grading internals remain inside the environment.
+
+`seed` is accepted for OpenEnv API compatibility but intentionally unused
+because tasks are selected deterministically by UID. Concurrent sessions are
+stacked independent environment instances, one trajectory per instance; they
+are not multiplexed trajectories.
+
+Executable assets come from the pinned `thinkingbox-bench-v1.0` release in
+[`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data).
+The [Hugging Face `microsoft/ThinkingBox-Bench`
+dataset](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench) is a
+viewer-friendly representation, while GitHub remains the executable source.
+
+## Configuration
+
+The server accepts an ordinary native ThinkingBox YAML configuration through
+`OPENENV_TB_CONFIG` or the reset `config=` argument. The file supplies
+`mcp_proxy`, `orchestrator.agent_model`, `user_model`, `judge_model`,
+`judge_type`, and user-completion behavior; see the
+[ThinkingBox LLM configuration
+guide](https://github.com/microsoft/thinkingbox/blob/main/docs/llm_endpoint_config.md).
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `OPENENV_TB_PROXY_URL` | `http://127.0.0.1:7111` | Session Proxy fallback |
+| `OPENENV_TB_DATASET` | pinned release | Local executable-data override |
+| `OPENENV_TB_DATA_CACHE` | `~/.cache/openenv/thinkingbox_bench` | Verified release cache |
+| `OPENENV_TB_AGENT` | `think` | Native agent definition |
+| `OPENENV_TB_CONFIG` | unset | Native server-side model/proxy config |
+| `OPENENV_TB_PROXY_TIMEOUT` | `120` | Proxy timeout without config override |
+| `OPENENV_TB_DATA_TIMEOUT` | `120` | Release download timeout |
+| `OPENENV_TB_MAX_CONCURRENT_ENVS` | `8` | Independent environment instances |
+
+The client message timeout remains longer than server-side model and episode
+operations so a valid long-running turn is not abandoned prematurely.
+
+## Deployment Boundary
+
+The image starts only the OpenEnv API. It currently needs an externally
+managed ThinkingBox Session Proxy, benchmark MCP servers, Typesense, and agent,
+user-simulator, and judge model endpoints. `GET /health` reports process
+liveness; `GET /ready` reports the observable external dependencies and marks
+scenario-specific Typesense readiness as unobservable from this process.
+
+## Citation
+
+Please cite the paper if ThinkingBox is useful in your experiments.
+
+- Paper: [One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark
+  for Agents in Stateful Business Workflows](https://arxiv.org/abs/2608.19741)
+- Framework: [`microsoft/thinkingbox`](https://github.com/microsoft/thinkingbox)
+- Executable benchmark:
+  [`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data)
+- Viewer dataset:
+  [`microsoft/ThinkingBox-Bench`](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench)
+
+```bibtex
+@misc{li2026successisntreliabilitythinkingbox,
+  title={One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
+  author={Zhuochun Li and Youngmin Ko and Ali Keramati and Nicola Ferri and Susana Palmaz Lopez Pelaez and Liang-Chun Tsai and Calvin Wang and Mirco Milletari and Tuhin Kundu and Vadim Smolyakov and Kjartan Olafsson and Tommy Guy},
+  year={2026},
+  eprint={2608.19741},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2608.19741},
+}
+```

envs/thinkingbox_env/SKIP_HF_DEPLOYMENT

@@ -0,0 +1 @@
+disabled now, coming soon