Web Search
diff --git a/envs/openapp_env/.gitignore b/envs/openapp_env/.gitignore
new file mode 100644
index 000000000..389f6e995
--- /dev/null
+++ b/envs/openapp_env/.gitignore
@@ -0,0 +1 @@
+OpenApps
diff --git a/envs/openapp_env/README.md b/envs/openapp_env/README.md
new file mode 100644
index 000000000..7c258c38f
--- /dev/null
+++ b/envs/openapp_env/README.md
@@ -0,0 +1,584 @@
+---
+title: OpenApp Environment Server
+emoji: š
+colorFrom: blue
+colorTo: green
+sdk: docker
+pinned: false
+app_port: 8000
+base_path: /web
+tags:
+ - openenv
+ - OpenApps
+ - BrowserGym
+ - UI-Agents
+ - Reinforcement-Learning
+---
+
+
+
+
+
+# OpenApp Environment
+
+
+
+*A web application simulation environment for OpenEnv that wraps the [OpenApps](https://github.com/facebookresearch/OpenApps) framework and BrowserGym.*
+
+
+
+## Overview
+
+The OpenApp environment provides a simulated web application ecosystem where agents can interact with various apps (calendar, todo, messenger, maps) using browser-based actions.
+
+
+
+
+
+This environment is ideal for:
+
+- Training and evaluating UI agents
+- Testing web automation strategies
+- Researching human-computer interaction
+- Developing multimodal agents
+
+## Features
+
+- **Multiple Apps**: Interact with calendar, todo list, messenger, and map applications
+- **Browser-Based Actions**: Click, fill forms, navigate, scroll, and more
+- **Task-Based Evaluation**: Optional task goals with automatic reward calculation
+- **Configurable**: Customize app configurations and behavior
+- **BrowserGym Integration**: Built on top of BrowserGym for robust browser interaction
+
+## Directory Structure
+
+```
+openapp_env/
+āāā __init__.py # Package exports
+āāā client.py # HTTP client for connecting to OpenApp
+āāā models.py # Data models for actions and observations
+āāā pyproject.toml # Package dependencies and configuration
+āāā openenv.yaml # OpenEnv environment configuration
+āāā test_openapp_env.py # Unit tests for environment structure
+āāā README.md # This file
+āāā IMPLEMENTATION.md # Implementation details and design decisions
+āāā example_usage.py # Basic usage example (legacy)
+āāā assets/ # Images and media
+ā āāā OpenApps_OpenEnv_RL.png # Environment overview diagram
+ā āāā openapps-demo.gif # Demo animation
+āāā server/ # Server-side environment implementation
+ āāā __init__.py
+ āāā app.py # FastAPI server application
+ āāā openapp_environment.py # Core environment logic (BrowserGym + OpenApps)
+ āāā Dockerfile # Docker image definition
+ āāā start.sh # Container startup script (runs both servers)
+```
+
+**Key Components:**
+
+- **client.py**: `OpenAppEnv` class that extends `HTTPEnvClient` for remote environment interaction
+- **models.py**: `OpenAppAction` and `OpenAppObservation` dataclasses with validation
+- **server/openapp_environment.py**: `OpenAppEnvironment` class that wraps BrowserGym and OpenApps
+- **server/app.py**: FastAPI server that exposes the environment via HTTP endpoints
+- **server/Dockerfile**: Self-contained Docker image with OpenApps server and FastAPI server
+- **server/start.sh**: Startup script that launches both OpenApps (port 5001) and FastAPI (port 8000)
+
+## Installation
+
+There are two ways to use the OpenApp environment: **Docker mode** (recommended, fully self-contained) or **Local mode** (requires manual server setup).
+
+### Option 1: Docker Mode (Recommended)
+
+Docker mode is fully self-contained and handles all dependencies automatically. No local installation required!
+
+**Step 1: Build the Docker image**
+
+The Docker image can be built in standalone mode using only public base images:
+
+```bash
+# Build from the environment directory
+cd envs/openapp_env
+docker build -t openapp-env:latest -f server/Dockerfile .
+```
+
+**Note for Meta/Corporate Networks:** If you're behind a proxy (HTTP_PROXY/HTTPS_PROXY set), you may need to bypass it for localhost connections:
+```bash
+export NO_PROXY=localhost,127.0.0.1
+cd envs/openapp_env
+docker build -t openapp-env:latest -f server/Dockerfile .
+```
+
+**What gets installed in Docker:**
+- **OpenEnv core**: Installed as a dependency
+- **OpenApps**: Cloned from GitHub and installed (runs server inside container)
+- **Core packages**: FastAPI, Uvicorn, Pydantic, Requests (from pyproject.toml)
+- **BrowserGym**: For browser automation
+- **Playwright**: Chromium browser for UI interaction
+- **Web interface support**: Enabled by default via `ENABLE_WEB_INTERFACE=true`
+
+**How Docker mode works:**
+The Docker container runs TWO services automatically:
+1. **OpenApps server** (port 5001) - Provides the web applications (calendar, todo, messenger, maps)
+2. **FastAPI server** (port 8000) - Exposes the OpenEnv HTTP API
+
+Both servers start automatically when the container launches. You only interact with port 8000.
+
+**Build details:**
+- Base image: `python:3.11-slim` (public)
+- Installation: Uses `pip install -e .` with pyproject.toml
+- System deps: Playwright/Chromium dependencies for browser automation
+- Size: ~5.7GB (includes Chromium browser and all dependencies)
+
+**Step 2: Run the example**
+```bash
+# For Meta/Corporate Networks with proxy, also set NO_PROXY:
+export NO_PROXY=localhost,127.0.0.1
+
+python examples/openapp_example.py --mode docker
+```
+
+**Note:** For Docker mode, you only need Python installed locally to run the example script. All environment dependencies are inside the Docker container.
+
+### Option 2: Local Mode
+
+Local mode requires manual setup of the OpenApps server. This mode is useful for development or when you need to customize the OpenApps configuration.
+
+**Prerequisites:**
+- Python 3.11+ installed
+- UV package manager (recommended) or pip
+
+**Step 1: Install openapp_env**
+```bash
+cd envs/openapp_env
+pip install -e .
+```
+
+This installs the environment package along with dependencies (BrowserGym, Playwright, etc.).
+
+**Step 2: Install Playwright browsers**
+```bash
+playwright install chromium
+```
+
+**Step 3: Clone and set up OpenApps** (for running the server)
+```bash
+# Clone OpenApps repository
+git clone https://github.com/facebookresearch/OpenApps.git
+cd OpenApps
+
+# Install dependencies
+uv sync # or: pip install -e .
+```
+
+**Why do I need the OpenApps repository?**
+
+The OpenApps Python package (installed via pip in Step 1) provides the library code, but the repository contains:
+- `launch.py` - The server startup script
+- `config/` - Hydra configuration files
+- Application templates and assets
+
+In Docker mode, all of this is included in the container, so you don't need to clone anything.
+
+## Quick Start
+
+### Running with Docker (Recommended)
+
+Docker mode is the easiest way - everything is automated:
+
+```bash
+# For Meta/Corporate networks with proxy, set NO_PROXY first:
+export NO_PROXY=localhost,127.0.0.1
+
+# Run the example
+python examples/openapp_example.py --mode docker
+```
+
+The Docker container automatically:
+- Starts the OpenApps server (port 5001)
+- Starts the FastAPI server (port 8000)
+- Manages both services for you
+
+No manual server setup required!
+
+**What happens inside the container:**
+
+When you run `from_docker_image()`, the following happens automatically:
+
+1. **Container Startup** (`/app/start.sh` runs):
+ ```bash
+ # Launches OpenApps server in background
+ cd /app/openapps
+ python launch.py &
+
+ # Waits for port 5001 to be ready
+ # Then starts FastAPI server
+ uvicorn openapp_env.server.app:app --host 0.0.0.0 --port 8000
+ ```
+
+2. **Your client code** interacts only with port 8000:
+ ```python
+ client = OpenAppEnv.from_docker_image("openapp-env:latest")
+ # Client -> FastAPI (port 8000) -> OpenApps (port 5001)
+ ```
+
+3. **On cleanup**, both servers are automatically stopped when the container is removed.
+
+### Running Locally
+
+For local usage, you need the OpenApps repository to run the server:
+
+**Step 1: Clone OpenApps (if you haven't already)**
+```bash
+git clone https://github.com/facebookresearch/OpenApps.git
+cd OpenApps
+uv sync
+```
+
+**Step 2: Start OpenApps Server** (in terminal 1)
+
+To run the server in **headless mode** (no browser window):
+```bash
+cd OpenApps # or wherever you cloned it
+uv run launch.py
+
+# or instead of the uv run you can use the Python command:
+python OpenApps/launch.py
+```
+
+To run the server with **visible browser** for visualization:
+```bash
+cd OpenApps
+python OpenApps/launch.py browsergym_env_args.headless=False
+```
+
+Wait for the server to start (you'll see "Port 5001 is available" or similar).
+
+**Step 3: Run your code** (in terminal 2)
+```bash
+export OPENAPPS_URL=http://localhost:5001
+python examples/openapp_example.py --mode local
+```
+
+**Note:** The OpenApps Python package (installed via pip) provides the modules, but you need the full repository to run launch.py with its config files.
+
+### Example Script
+
+```bash
+# Run with Docker (recommended)
+python examples/openapp_example.py --mode docker
+
+# Run locally (requires OpenApps server running)
+export OPENAPPS_URL=http://localhost:5001
+python examples/openapp_example.py --mode local
+
+# Show browser window to visualize agent actions
+python examples/openapp_example.py --mode local --show-browser
+
+# Run with custom number of steps
+python examples/openapp_example.py --mode docker --num-steps 20
+
+# See all options
+python examples/openapp_example.py --help
+```
+
+### Visualizing Agent Interactions
+
+There are multiple ways to see what the agent is doing:
+
+**Option 1: Show Browser Window (Local Mode)**
+
+The key is to start the OpenApps server with visualization enabled:
+
+```bash
+# Terminal 1: Start OpenApps server with visible browser
+cd OpenApps
+python OpenApps/launch.py browsergym_env_args.headless=False
+
+# Terminal 2: Run your agent code
+export OPENAPPS_URL=http://localhost:5001
+python examples/openapp_example.py --mode local
+```
+
+**Important:** The browser visualization is controlled by the OpenApps server, not the client. You must launch the server with `browsergym_env_args.headless=False` to see the browser window.
+
+**Option 2: Access Web Interface Directly**
+
+While the OpenApps server is running, open your browser to:
+- Main page: `http://localhost:5001`
+- Calendar: `http://localhost:5001/calendar`
+- Todo: `http://localhost:5001/todo`
+- Messenger: `http://localhost:5001/messages`
+- Maps: `http://localhost:5001/maps`
+
+**Option 3: Docker Web Interface**
+
+When running in Docker mode, you can also access a web interface for manual testing:
+
+```bash
+# Start a container and keep it running
+docker run -d -p 8000:8000 openapp-env:latest
+
+# Access the web interface
+# - Interactive UI: http://localhost:8000/web
+# - API docs: http://localhost:8000/docs
+# - OpenApps (internal): http://localhost:5001 (inside container)
+```
+
+**Note:** In Docker mode, the OpenApps server runs inside the container and is not directly accessible from your host machine. The FastAPI server at port 8000 acts as a proxy to interact with OpenApps.
+
+### Basic Usage
+
+```python
+from envs.openapp_env import OpenAppAction, OpenAppEnv
+
+# Create environment from Docker image
+client = OpenAppEnv.from_docker_image("openapp-env:latest")
+
+# Reset to initial state
+result = client.reset()
+print(f"Starting URL: {result.observation.url}")
+
+# Navigate to calendar app
+result = client.step(OpenAppAction(
+ action_type="goto",
+ url="http://localhost:5001/calendar"
+))
+
+# Click on a button (example bid)
+result = client.step(OpenAppAction(
+ action_type="click",
+ bid="add-event-btn"
+))
+
+# Fill in a form field
+result = client.step(OpenAppAction(
+ action_type="fill",
+ bid="event-title-input",
+ text="Team Meeting"
+))
+
+print(f"Reward: {result.reward}")
+print(f"Done: {result.done}")
+
+# Cleanup
+client.close()
+```
+
+### Action Types
+
+The environment supports the following action types:
+
+- **click**: Click on an element
+ - Required: `bid` (BrowserGym element ID)
+
+- **fill**: Fill a text input field
+ - Required: `bid`, `text`
+
+- **select_option**: Select from dropdown
+ - Required: `bid`, `value`
+
+- **goto**: Navigate to a URL
+ - Required: `url`
+
+- **scroll**: Scroll the page
+ - Required: `direction` ("up" or "down")
+
+- **send_keys**: Send keyboard input
+ - Required: `text`
+
+- **noop**: No operation
+
+### Observations
+
+Each observation includes:
+
+- **html**: Current page HTML content
+- **url**: Current page URL
+- **open_pages_urls**: List of all open page URLs
+- **active_page_index**: Index of currently active page
+- **screenshot**: Base64-encoded screenshot (optional)
+- **axtree_txt**: Accessibility tree for element interaction
+- **app_state**: Current state of all apps (calendar events, todos, messages, etc.)
+- **task_info**: Information about current task (if using tasks)
+- **last_action_error**: Error message if last action failed
+
+## Configuration
+
+### Environment Parameters
+
+```python
+from envs.openapp_env.server.openapp_environment import OpenAppEnvironment
+
+env = OpenAppEnvironment(
+ web_app_port=5001, # Port for OpenApps server
+ headless=True, # Run browser in headless mode
+ task_name="add_meeting", # Optional task name
+ apps_config={}, # App-specific configuration
+ max_steps=50, # Maximum steps per episode
+)
+```
+
+**Note:** OpenApps is automatically detected from the installed Python package. You can optionally override with `openapps_path` parameter or `OPENAPPS_PATH` environment variable if needed.
+
+## Tasks and Rewards
+
+The environment can be configured with specific tasks from OpenApps. Tasks define:
+- Goal state (e.g., "Add a meeting with Dennis to the calendar")
+- Reward function based on app state changes
+- Success criteria
+
+See [OpenApps documentation](https://facebookresearch.github.io/OpenApps/) for available tasks.
+
+## Example: Task-Based Training
+
+```python
+from envs.openapp_env import OpenAppAction, OpenAppEnv
+
+# Create environment with a specific task
+client = OpenAppEnv.from_docker_image("openapp-env:latest")
+
+# The task will guide the agent toward a specific goal
+# Rewards will be based on progress toward completing the task
+result = client.reset()
+
+# Agent interacts to complete the task
+# ... agent logic here ...
+
+client.close()
+```
+
+## Development
+
+### Running Server Locally (without Docker)
+
+```bash
+cd envs/openapp_env
+uv run server
+```
+
+The server will start at `http://localhost:8000`
+
+### Testing
+
+```python
+from openapp_env.server.openapp_environment import OpenAppEnvironment
+from openapp_env.models import OpenAppAction
+
+def test_environment():
+ env = OpenAppEnvironment()
+
+ # Test reset
+ obs = env.reset()
+ assert obs.url != ""
+
+ # Test step
+ action = OpenAppAction(action_type="noop")
+ obs = env.step(action)
+ assert env.state.step_count == 1
+
+ # Cleanup
+ env.close()
+
+test_environment()
+```
+
+## Attribution
+
+This environment integrates:
+- [OpenApps](https://github.com/facebookresearch/OpenApps) - Web application simulation framework
+- [BrowserGym](https://github.com/ServiceNow/BrowserGym) - Browser automation environment
+
+## Troubleshooting
+
+### Docker Build Issues
+
+**Error: `Container did not become ready`**
+
+If you're behind a corporate proxy (Meta/Facebook networks), set `NO_PROXY`:
+```bash
+export NO_PROXY=localhost,127.0.0.1
+docker build -t openapp-env:latest -f envs/openapp_env/server/Dockerfile .
+```
+
+**Error: `Environment variable 'USER' not found`**
+
+This is automatically handled in the Dockerfile with `ENV USER=root`. If you see this, rebuild the image.
+
+**Container exits immediately**
+
+Check the logs to see which server failed:
+```bash
+docker logs
+```
+
+Common causes:
+- OpenApps server failed to start (check for port conflicts)
+- Missing dependencies (rebuild with `--no-cache`)
+
+### Local Mode Issues
+
+**Error: `OPENAPPS_URL not set`**
+
+Set the environment variable before running:
+```bash
+export OPENAPPS_URL=http://localhost:5001
+python examples/openapp_example.py --mode local
+```
+
+**Error: `Connection refused to localhost:5001`**
+
+Make sure the OpenApps server is running:
+```bash
+cd OpenApps
+uv run launch.py
+```
+
+**Browser visualization not working**
+
+The visualization is controlled by the **server**, not the client:
+```bash
+# Start server with visible browser
+cd OpenApps
+python launch.py browsergym_env_args.headless=False
+```
+
+### Performance Issues
+
+**Docker container is slow**
+
+The container runs both a full Chromium browser and web applications. For faster performance:
+- Increase Docker memory allocation (6GB+ recommended)
+- Use headless mode (default)
+- Reduce `max_steps` in environment configuration
+
+**Large Docker image size**
+
+The image is ~5.7GB due to:
+- Chromium browser (~1.5GB)
+- OpenApps dependencies (~2GB)
+- BrowserGym and ML libraries (~2GB)
+
+This is expected for a full browser automation environment.
+
+## License
+
+BSD 3-Clause License (see LICENSE file in OpenEnv root directory)
+
+## Citation
+
+If you use this environment in your research, please cite both OpenEnv and OpenApps:
+
+```bibtex
+@article{ullrich2025openapps0,
+ title = {OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability},
+ author = {Karen Ullrich and Jingtong Su and Claudia Shi and Arjun Subramonian and Amir Bar and Ivan Evtimov and Nikolaos Tsilivis and Randall Balestriero and Julia Kempe and Mark Ibrahim},
+ year = {2025},
+ journal = {arXiv preprint arXiv: 2511.20766}
+}
+```
diff --git a/envs/openapp_env/__init__.py b/envs/openapp_env/__init__.py
new file mode 100644
index 000000000..6bddd9367
--- /dev/null
+++ b/envs/openapp_env/__init__.py
@@ -0,0 +1,12 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""OpenApp Environment - Web application simulation environment for UI agents."""
+
+from .client import OpenAppEnv
+from .models import OpenAppAction, OpenAppObservation
+
+__all__ = ["OpenAppAction", "OpenAppObservation", "OpenAppEnv"]
diff --git a/envs/openapp_env/assets/01-messages.mov b/envs/openapp_env/assets/01-messages.mov
new file mode 100644
index 000000000..4efa54bf2
Binary files /dev/null and b/envs/openapp_env/assets/01-messages.mov differ
diff --git a/envs/openapp_env/assets/02-editor.mov b/envs/openapp_env/assets/02-editor.mov
new file mode 100644
index 000000000..ef2c21dad
Binary files /dev/null and b/envs/openapp_env/assets/02-editor.mov differ
diff --git a/envs/openapp_env/assets/03-calendar.mov b/envs/openapp_env/assets/03-calendar.mov
new file mode 100644
index 000000000..03adce59f
Binary files /dev/null and b/envs/openapp_env/assets/03-calendar.mov differ
diff --git a/envs/openapp_env/assets/04-todo.mov b/envs/openapp_env/assets/04-todo.mov
new file mode 100644
index 000000000..0b2cf43b1
Binary files /dev/null and b/envs/openapp_env/assets/04-todo.mov differ
diff --git a/envs/openapp_env/assets/OpenApps_OpenEnv_RL.png b/envs/openapp_env/assets/OpenApps_OpenEnv_RL.png
new file mode 100644
index 000000000..70514d7ea
Binary files /dev/null and b/envs/openapp_env/assets/OpenApps_OpenEnv_RL.png differ
diff --git a/envs/openapp_env/assets/demo-showcase.html b/envs/openapp_env/assets/demo-showcase.html
new file mode 100644
index 000000000..d4583c466
--- /dev/null
+++ b/envs/openapp_env/assets/demo-showcase.html
@@ -0,0 +1,624 @@
+
+
+
+
+
+ OpenApp + OpenEnv Demo Showcase
+
+
+
+
+
+
+
+
+
+
+
OpenApp + OpenEnv
+
+ A powerful integration bringing realistic web application environments to reinforcement learning agents.
+ Watch AI agents interact with calendar, messaging, code editor, and task management apps.
+
Watch our AI agents perform real tasks in realistic web environments
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
š¬
+
Messenger App
+
+
+ AI agent navigates conversations, types messages interactively, and sends them to contacts.
+ Demonstrates natural language input and real-time chat interactions.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
š»
+
Code Editor
+
+
+ Agent creates files and writes a complete PyTorch training loop with syntax highlighting.
+ Shows code generation, file management, and save operations.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
š
+
Calendar App
+
+
+ Navigate between calendar and agenda views, browse events across months,
+ and view detailed event information. Perfect for scheduling tasks.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
ā
+
Todo Manager
+
+
+ Browse and manage task lists, edit task details, mark items complete,
+ and organize priorities. Demonstrates CRUD operations on structured data.
+
+
+
+
+
+
+
+
+
āØ Key Features
+
Why OpenApp + OpenEnv is perfect for AI agent research
+
+
+
+
š®
+
Gymnasium Compatible
+
+ Standard RL interface with observations, actions, and rewards.
+ Drop-in replacement for existing training pipelines.
+
+
+
+
š
+
Real Web Apps
+
+ Authentic web applications with HTML, CSS, and JavaScript.
+ No simplified simulations ā real browser interactions.
+
+
+
+
š
+
Configurable Tasks
+
+ YAML-based configuration for custom scenarios, data, and rewards.
+ Easily create new training environments.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/envs/openapp_env/assets/openapps-demo.gif b/envs/openapp_env/assets/openapps-demo.gif
new file mode 100644
index 000000000..97c3298a2
Binary files /dev/null and b/envs/openapp_env/assets/openapps-demo.gif differ
diff --git a/envs/openapp_env/client.py b/envs/openapp_env/client.py
new file mode 100644
index 000000000..6efc93710
--- /dev/null
+++ b/envs/openapp_env/client.py
@@ -0,0 +1,139 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+OpenApp Environment HTTP Client.
+
+This module provides the client for connecting to an OpenApp Environment server
+over HTTP.
+"""
+
+from typing import Any, Dict
+
+# Support both in-repo and standalone imports
+try:
+ # In-repo imports (when running from OpenEnv repository)
+ from openenv.core.client_types import StepResult
+ from openenv.core.env_server.types import State
+ from openenv.core.env_client import EnvClient
+ from .models import OpenAppAction, OpenAppObservation
+except ImportError:
+ # Standalone imports (when environment is standalone with openenv-core from pip)
+ from openenv.core.client_types import StepResult
+ from openenv.core.env_server.types import State
+ from openenv.core.env_client import EnvClient
+ from openapp_env.models import OpenAppAction, OpenAppObservation
+
+
+class OpenAppEnv(EnvClient[OpenAppAction, OpenAppObservation, State]):
+ """
+ HTTP client for the OpenApp Environment.
+
+ This client connects to an OpenAppEnvironment HTTP server and provides
+ methods to interact with it: reset(), step(), and state access.
+
+ The OpenApp environment simulates web applications (calendar, todo, messenger, maps)
+ and allows agents to interact with them using browser-based actions.
+
+ Example:
+ >>> # Connect to a running server
+ >>> client = OpenAppEnv(base_url="http://localhost:8000")
+ >>> result = client.reset()
+ >>> print(result.observation.url)
+ >>>
+ >>> # Click on an element
+ >>> result = client.step(OpenAppAction(action_type="click", bid="123"))
+ >>> print(result.observation.html)
+ >>> print(result.reward)
+
+ Example with Docker:
+ >>> # Automatically start container and connect
+ >>> client = OpenAppEnv.from_docker_image("openapp-env:latest")
+ >>> result = client.reset()
+ >>> # Fill a text field
+ >>> result = client.step(OpenAppAction(
+ ... action_type="fill",
+ ... bid="456",
+ ... text="Meeting with team"
+ ... ))
+ """
+
+ def _step_payload(self, action: OpenAppAction) -> Dict:
+ """
+ Convert OpenAppAction to JSON payload for step request.
+
+ Args:
+ action: OpenAppAction instance
+
+ Returns:
+ Dictionary representation suitable for JSON encoding
+ """
+ payload = {
+ "action_type": action.action_type,
+ }
+
+ # Add optional fields if present
+ if action.bid is not None:
+ payload["bid"] = action.bid
+ if action.text is not None:
+ payload["text"] = action.text
+ if action.value is not None:
+ payload["value"] = action.value
+ if action.url is not None:
+ payload["url"] = action.url
+ if action.direction is not None:
+ payload["direction"] = action.direction
+ if action.metadata:
+ payload["metadata"] = action.metadata
+
+ return payload
+
+ def _parse_result(self, payload: Dict) -> StepResult[OpenAppObservation]:
+ """
+ Parse server response into StepResult[OpenAppObservation].
+
+ Args:
+ payload: JSON response from server
+
+ Returns:
+ StepResult with OpenAppObservation
+ """
+ obs_data = payload.get("observation", {})
+ observation = OpenAppObservation(
+ html=obs_data.get("html", ""),
+ url=obs_data.get("url", ""),
+ open_pages_urls=obs_data.get("open_pages_urls", []),
+ active_page_index=obs_data.get("active_page_index", 0),
+ screenshot=obs_data.get("screenshot"),
+ axtree_txt=obs_data.get("axtree_txt", ""),
+ app_state=obs_data.get("app_state", {}),
+ task_info=obs_data.get("task_info"),
+ last_action_error=obs_data.get("last_action_error"),
+ done=payload.get("done", False),
+ reward=payload.get("reward"),
+ metadata=obs_data.get("metadata", {}),
+ )
+
+ return StepResult(
+ observation=observation,
+ reward=payload.get("reward"),
+ done=payload.get("done", False),
+ )
+
+ def _parse_state(self, payload: Dict) -> State:
+ """
+ Parse server response into State object.
+
+ Args:
+ payload: JSON response from /state endpoint
+
+ Returns:
+ State object with episode_id and step_count
+ """
+ return State(
+ episode_id=payload.get("episode_id"),
+ step_count=payload.get("step_count", 0),
+ )
diff --git a/envs/openapp_env/example_usage.py b/envs/openapp_env/example_usage.py
new file mode 100644
index 000000000..7010c684f
--- /dev/null
+++ b/envs/openapp_env/example_usage.py
@@ -0,0 +1,279 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+Example usage of OpenApp Environment.
+
+This script demonstrates how to use the OpenApp environment with OpenEnv.
+
+For a complete runnable example, see: examples/openapp_example.py
+
+Visualization Options:
+ To see the browser window and watch agent interactions:
+
+ Terminal 1: Start OpenApps server with visible browser
+ cd OpenApps
+ python OpenApps/launch.py browsergym_env_args.headless=False
+
+ Terminal 2: Run your agent code
+ export OPENAPPS_URL=http://localhost:5001
+ python examples/openapp_example.py --mode local
+
+ Or access OpenApps web interface at http://localhost:5001
+ Docker mode web interface at http://localhost:8000/web
+
+Important:
+ Browser visualization is controlled by the OpenApps SERVER, not the client.
+ Launch the server with 'browsergym_env_args.headless=False' to see the browser.
+"""
+
+import sys
+from pathlib import Path
+
+# Add src to path for local testing
+sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
+
+from envs.openapp_env import OpenAppAction, OpenAppEnv
+
+
+def example_basic_usage():
+ """Basic usage example."""
+ print("=" * 60)
+ print("OpenApp Environment - Basic Usage Example")
+ print("=" * 60)
+
+ # Option 1: Connect to a running server
+ print("\nOption 1: Connect to running server")
+ print("client = OpenAppEnv(base_url='http://localhost:8000')")
+
+ # Option 2: Start from Docker image (recommended)
+ print("\nOption 2: Start from Docker image")
+ print("client = OpenAppEnv.from_docker_image('openapp-env:latest')")
+
+ print("\n" + "-" * 60)
+
+
+def example_actions():
+ """Example of different action types."""
+ print("\nExample Actions")
+ print("-" * 60)
+
+ # Navigate to a page
+ print("\n1. Navigate to calendar app:")
+ print("action = OpenAppAction(")
+ print(" action_type='goto',")
+ print(" url='http://localhost:5001/calendar'")
+ print(")")
+ print("result = client.step(action)")
+
+ # Click on an element
+ print("\n2. Click on a button:")
+ print("action = OpenAppAction(")
+ print(" action_type='click',")
+ print(" bid='add-event-btn' # BrowserGym element ID")
+ print(")")
+ print("result = client.step(action)")
+
+ # Fill a form field
+ print("\n3. Fill in text input:")
+ print("action = OpenAppAction(")
+ print(" action_type='fill',")
+ print(" bid='event-title-input',")
+ print(" text='Team Meeting'")
+ print(")")
+ print("result = client.step(action)")
+
+ # Select from dropdown
+ print("\n4. Select from dropdown:")
+ print("action = OpenAppAction(")
+ print(" action_type='select_option',")
+ print(" bid='time-select',")
+ print(" value='14:00'")
+ print(")")
+ print("result = client.step(action)")
+
+ # Scroll the page
+ print("\n5. Scroll down:")
+ print("action = OpenAppAction(")
+ print(" action_type='scroll',")
+ print(" direction='down'")
+ print(")")
+ print("result = client.step(action)")
+
+ # No operation
+ print("\n6. No operation (useful for observation):")
+ print("action = OpenAppAction(action_type='noop')")
+ print("result = client.step(action)")
+
+
+def example_observations():
+ """Example of observation structure."""
+ print("\n\nObservation Structure")
+ print("-" * 60)
+
+ print("\nAfter reset() or step(), you receive:")
+ print("result.observation.html # Current page HTML")
+ print("result.observation.url # Current URL")
+ print("result.observation.open_pages_urls # All open pages")
+ print("result.observation.axtree_txt # Accessibility tree")
+ print("result.observation.app_state # App states (calendar, todo, etc.)")
+ print("result.observation.task_info # Task information (if using tasks)")
+ print("result.observation.screenshot # Page screenshot (base64)")
+ print("result.observation.last_action_error # Error from last action")
+ print("result.reward # Step reward")
+ print("result.done # Episode done flag")
+
+
+def example_complete_workflow():
+ """Complete workflow example."""
+ print("\n\nComplete Workflow Example")
+ print("=" * 60)
+
+ example_code = """
+from envs.openapp_env import OpenAppAction, OpenAppEnv
+
+# Create client (starts Docker container)
+client = OpenAppEnv.from_docker_image("openapp-env:latest")
+
+try:
+ # Reset environment
+ result = client.reset()
+ print(f"Starting at: {result.observation.url}")
+
+ # Navigate to calendar
+ result = client.step(OpenAppAction(
+ action_type="goto",
+ url="http://localhost:5001/calendar"
+ ))
+
+ # Click to add new event
+ result = client.step(OpenAppAction(
+ action_type="click",
+ bid="new-event-button"
+ ))
+
+ # Fill event title
+ result = client.step(OpenAppAction(
+ action_type="fill",
+ bid="title-input",
+ text="Project Review Meeting"
+ ))
+
+ # Fill event date
+ result = client.step(OpenAppAction(
+ action_type="fill",
+ bid="date-input",
+ text="2025-12-15"
+ ))
+
+ # Submit form
+ result = client.step(OpenAppAction(
+ action_type="click",
+ bid="submit-button"
+ ))
+
+ print(f"Reward: {result.reward}")
+ print(f"Done: {result.done}")
+ print(f"App State: {result.observation.app_state}")
+
+finally:
+ # Always cleanup
+ client.close()
+"""
+
+ print(example_code)
+
+
+def example_with_tasks():
+ """Example using OpenApps tasks."""
+ print("\n\nUsing Tasks (Task-Based RL)")
+ print("=" * 60)
+
+ example_code = """
+# Environment can be configured with specific tasks
+# Tasks define goals and automatic reward calculation
+
+from envs.openapp_env.server.openapp_environment import OpenAppEnvironment
+
+env = OpenAppEnvironment(
+ openapps_url="http://localhost:5001", # OpenApps server URL
+ task_name="add_meeting_with_dennis", # Optional task name
+ headless=False, # Set to False to watch the browser
+ max_steps=50,
+)
+
+obs = env.reset()
+# Now the environment has a goal: add a meeting with Dennis
+# Rewards will be based on progress toward this goal
+
+# Agent loop
+done = False
+while not done:
+ action = agent.get_action(obs) # Your agent
+ obs = env.step(action)
+ done = obs.done
+
+print(f"Task completed! Reward: {obs.reward}")
+env.close()
+"""
+
+ print(example_code)
+
+
+def example_visualization():
+ """Example of visualization options."""
+ print("\n\nVisualization Options")
+ print("=" * 60)
+
+ example_code = """
+# Option 1: Show browser window (watch agent in real-time)
+from envs.openapp_env.server.openapp_environment import OpenAppEnvironment
+
+env = OpenAppEnvironment(
+ openapps_url="http://localhost:5001",
+ headless=False, # Show browser window
+)
+
+obs = env.reset()
+# You'll see a browser window open!
+
+# Option 2: Access web interface manually
+# While OpenApps server is running, open in your browser:
+# - Main: http://localhost:5001
+# - Calendar: http://localhost:5001/calendar
+# - Todo: http://localhost:5001/todo
+# - Messenger: http://localhost:5001/messenger
+# - Maps: http://localhost:5001/maps
+
+# Option 3: Use the example script with --show-browser
+# python examples/openapp_example.py --mode local --show-browser
+"""
+
+ print(example_code)
+
+
+def main():
+ """Run all examples."""
+ example_basic_usage()
+ example_actions()
+ example_observations()
+ example_complete_workflow()
+ example_with_tasks()
+ example_visualization()
+
+ print("\n" + "=" * 60)
+ print("For a complete runnable example:")
+ print(" python examples/openapp_example.py --mode local --show-browser")
+ print("\nFor more information, see:")
+ print("- README.md in this directory")
+ print("- OpenApps docs: https://facebookresearch.github.io/OpenApps/")
+ print("- OpenEnv docs: https://meta-pytorch.org/OpenEnv/")
+ print("=" * 60)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/envs/openapp_env/models.py b/envs/openapp_env/models.py
new file mode 100644
index 000000000..d4d706286
--- /dev/null
+++ b/envs/openapp_env/models.py
@@ -0,0 +1,86 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+Data models for the OpenApp Environment.
+
+The OpenApp environment provides a simulated web application environment
+for training and evaluating UI agents that interact with various apps
+(calendar, todo, messenger, maps, etc.) using browser actions.
+"""
+
+from typing import Any, Dict, List, Optional
+
+from pydantic import Field
+
+# Support both in-repo and standalone imports
+try:
+ # In-repo imports (when running from OpenEnv repository)
+ from openenv.core.env_server.types import Action, Observation
+except ImportError:
+ # Standalone imports (when environment is standalone with openenv-core from pip)
+ from openenv.core.env_server.types import Action, Observation
+
+
+class OpenAppAction(Action):
+ """
+ Action for the OpenApp environment.
+
+ Supports BrowserGym-style actions for web interaction:
+ - click: Click on an element (requires bid - BrowserGym ID)
+ - fill: Fill a text field (requires bid and text)
+ - select_option: Select from dropdown (requires bid and value)
+ - goto: Navigate to URL (requires url)
+ - scroll: Scroll the page (requires direction)
+ - send_keys: Send keyboard input (requires text)
+ - noop: No operation
+
+ Attributes:
+ action_type: Type of action to perform
+ bid: BrowserGym element ID (for click, fill, select_option)
+ text: Text content (for fill, send_keys)
+ value: Value to select (for select_option)
+ url: URL to navigate to (for goto)
+ direction: Scroll direction - 'up' or 'down' (for scroll)
+ """
+
+ action_type: str = Field(
+ ..., description="Type of action: click, fill, select_option, goto, scroll, send_keys, noop"
+ )
+ bid: Optional[str] = Field(default=None, description="BrowserGym element ID")
+ text: Optional[str] = Field(default=None, description="Text content for fill or send_keys")
+ value: Optional[str] = Field(default=None, description="Value for select_option")
+ url: Optional[str] = Field(default=None, description="URL for goto action")
+ direction: Optional[str] = Field(default=None, description="Scroll direction: 'up' or 'down'")
+
+
+class OpenAppObservation(Observation):
+ """
+ Observation from the OpenApp environment.
+
+ Provides comprehensive state information about the web apps and browser state.
+
+ Attributes:
+ html: Current page HTML content
+ url: Current page URL
+ open_pages_urls: List of all open page URLs
+ active_page_index: Index of currently active page
+ screenshot: Base64-encoded screenshot (optional)
+ axtree_txt: Accessibility tree as text (for element interaction)
+ app_state: Current state of all apps (calendar, todo, messenger, map)
+ task_info: Information about the current task (if any)
+ last_action_error: Error message from last action (if failed)
+ """
+
+ html: str = Field(default="", description="Current page HTML content")
+ url: str = Field(default="", description="Current page URL")
+ open_pages_urls: List[str] = Field(default_factory=list, description="List of all open page URLs")
+ active_page_index: int = Field(default=0, ge=0, description="Index of currently active page")
+ screenshot: Optional[str] = Field(default=None, description="Base64-encoded screenshot")
+ axtree_txt: str = Field(default="", description="Accessibility tree as text")
+ app_state: Dict[str, Any] = Field(default_factory=dict, description="State of all apps")
+ task_info: Optional[Dict[str, Any]] = Field(default=None, description="Current task information")
+ last_action_error: Optional[str] = Field(default=None, description="Error from last action")
diff --git a/envs/openapp_env/openenv.yaml b/envs/openapp_env/openenv.yaml
new file mode 100644
index 000000000..91341f735
--- /dev/null
+++ b/envs/openapp_env/openenv.yaml
@@ -0,0 +1,6 @@
+spec_version: 1
+name: openapp_env
+type: space
+runtime: fastapi
+app: server.app:app
+port: 8000
diff --git a/envs/openapp_env/pyproject.toml b/envs/openapp_env/pyproject.toml
new file mode 100644
index 000000000..b9504973e
--- /dev/null
+++ b/envs/openapp_env/pyproject.toml
@@ -0,0 +1,58 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openenv-openapp_env"
+version = "0.1.0"
+description = "OpenApp Environment for OpenEnv - web application simulation environment for UI agents"
+requires-python = ">=3.11,<3.14"
+dependencies = [
+ # NOTE: openenv-core is NOT listed here to avoid openai version conflict
+ # It is installed separately in the Dockerfile with --no-deps to avoid
+ # openai>=2.7.2 conflicting with OpenApps' openai<2 requirement.
+ # For local development, install manually:
+ # pip install --no-deps "openenv-core @ git+https://github.com/meta-pytorch/OpenEnv.git"
+ # pip install fastapi pydantic uvicorn requests websockets
+ #
+ # NOTE: open_apps is also NOT listed here for the same reason.
+ # Install manually for local development:
+ # pip install git+https://github.com/facebookresearch/OpenApps.git
+ #
+ # Server dependencies (these are installed by Dockerfile separately for openenv-core)
+ "fastapi>=0.115.0",
+ "pydantic>=2.0.0",
+ "uvicorn[standard]>=0.24.0",
+ "requests>=2.31.0",
+ "websockets>=15.0.1",
+ # BrowserGym dependencies
+ "browsergym>=0.13.3",
+ "playwright>=1.40.0",
+ # Additional dependencies for web app interaction
+ "python-multipart>=0.0.20",
+]
+
+[project.optional-dependencies]
+dev = [
+ "pytest>=8.0.0",
+ "pytest-cov>=4.0.0",
+]
+
+[project.scripts]
+server = "openapp_env.server.app:main"
+
+[tool.setuptools]
+packages = ["openapp_env", "openapp_env.server"]
+package-dir = { "openapp_env" = ".", "openapp_env.server" = "server" }
+
+[tool.setuptools.package-data]
+openapp_env = ["**/*.yaml", "**/*.yml", "**/*.md"]
+
+[tool.hatch.metadata]
+allow-direct-references = true
diff --git a/envs/openapp_env/server/Dockerfile b/envs/openapp_env/server/Dockerfile
new file mode 100644
index 000000000..ebfc0fad3
--- /dev/null
+++ b/envs/openapp_env/server/Dockerfile
@@ -0,0 +1,128 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Dockerfile for OpenApp Environment
+# This image provides OpenApps web application simulation for UI agent training
+#
+# This Dockerfile works for both local builds and HuggingFace Spaces deployment:
+# - Local build: cd envs/openapp_env && docker build -t openapp-env:latest -f server/Dockerfile .
+# - HuggingFace: Automatically deployed via `openenv push`
+#
+# Run with web interface:
+# docker run -p 8000:8000 -e ENABLE_WEB_INTERFACE=true openapp-env:latest
+
+FROM python:3.11-slim
+
+# Set metadata
+LABEL maintainer="OpenEnv Team"
+LABEL description="OpenApp Environment with BrowserGym for UI agent training"
+LABEL org.opencontainers.image.source="https://github.com/meta-pytorch/OpenEnv"
+
+# Set working directory
+WORKDIR /app/env
+
+# Install system dependencies
+# - git: required to clone OpenApps from GitHub
+# - curl: for healthcheck
+# - Playwright/BrowserGym dependencies: fonts, libraries for browser automation
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends \
+ git \
+ curl \
+ ca-certificates \
+ wget \
+ gnupg \
+ # Playwright/Chromium dependencies
+ libnss3 \
+ libnspr4 \
+ libatk1.0-0 \
+ libatk-bridge2.0-0 \
+ libcups2 \
+ libdrm2 \
+ libdbus-1-3 \
+ libxkbcommon0 \
+ libxcomposite1 \
+ libxdamage1 \
+ libxfixes3 \
+ libxrandr2 \
+ libgbm1 \
+ libasound2 \
+ libpango-1.0-0 \
+ libcairo2 \
+ libatspi2.0-0 \
+ libxshmfence1 \
+ fonts-liberation \
+ libappindicator3-1 \
+ xdg-utils && \
+ rm -rf /var/lib/apt/lists/*
+
+# Set environment variables
+ENV PYTHONUNBUFFERED=1
+
+# Set working directory
+WORKDIR /app/env
+
+# Copy environment files
+# Context is always the env directory (envs/openapp_env/)
+# - GitHub Actions: uses context: envs/openapp_env
+# - HuggingFace: openenv push uploads env dir as context
+COPY . /app/env
+
+# Install OpenApps FIRST to establish openai<2 (required by agentlab)
+# This must happen before openenv-core to avoid version conflict
+WORKDIR /app
+RUN git clone https://github.com/facebookresearch/OpenApps.git openapps && \
+ cd openapps && \
+ pip install --no-cache-dir -e .
+
+# Verify OpenApps installation
+RUN python -c "import open_apps; print('ā OpenApps installed')"
+
+# Install openenv-core from GitHub with --no-deps to avoid openai>=2.7.2 conflict
+# Then install only the server dependencies (no openai needed for server)
+RUN pip install --no-cache-dir --no-deps "openenv-core @ git+https://github.com/meta-pytorch/OpenEnv.git" && \
+ pip install --no-cache-dir fastapi pydantic uvicorn requests websockets
+
+# Install openapp_env and remaining dependencies
+WORKDIR /app/env
+RUN pip install --no-cache-dir -e .
+
+# Verify installation
+RUN python -c "import openapp_env; print('ā openapp_env installed')" && \
+ python -c "import openapp_env.server.app; print('ā openapp_env.server.app importable')"
+
+# Install Playwright browsers (Chromium for BrowserGym)
+# We already installed system dependencies above, so just install the browser
+RUN playwright install chromium
+
+# Copy startup script
+WORKDIR /app/env
+COPY server/start.sh /app/start.sh
+RUN chmod +x /app/start.sh
+
+# OpenApp-specific environment variables (can be overridden at runtime)
+ENV OPENAPPS_URL=http://localhost:5001
+ENV OPENAPPS_PORT=5001
+ENV OPENAPP_HEADLESS=true
+ENV OPENAPP_MAX_STEPS=50
+
+# Hydra requires USER environment variable
+ENV USER=root
+
+# Enable web interface by default (set to false to disable)
+ENV ENABLE_WEB_INTERFACE=true
+
+# Expose ports (8000 for FastAPI, 5001 for OpenApps)
+EXPOSE 8000 5001
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
+ CMD curl -f http://localhost:8000/health || exit 1
+
+# Run the startup script that launches both OpenApps server and FastAPI server
+# Web interface will be available at /web if ENABLE_WEB_INTERFACE=true
+# API documentation available at /docs
+CMD ["/app/start.sh"]
diff --git a/envs/openapp_env/server/__init__.py b/envs/openapp_env/server/__init__.py
new file mode 100644
index 000000000..44ce51c6e
--- /dev/null
+++ b/envs/openapp_env/server/__init__.py
@@ -0,0 +1,7 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""OpenApp Environment Server."""
diff --git a/envs/openapp_env/server/app.py b/envs/openapp_env/server/app.py
new file mode 100644
index 000000000..18f3b984d
--- /dev/null
+++ b/envs/openapp_env/server/app.py
@@ -0,0 +1,59 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+FastAPI application for the OpenApp Environment.
+
+This module creates an HTTP server that exposes the OpenAppEnvironment
+over HTTP endpoints, making it compatible with HTTPEnvClient.
+
+Usage:
+ # Development (with auto-reload):
+ uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
+
+ # Production:
+ uvicorn server.app:app --host 0.0.0.0 --port 8000 --workers 4
+
+ # Or run directly:
+ uv run --project . server
+"""
+
+# Support both in-repo and standalone imports
+try:
+ # In-repo imports (when running from OpenEnv repository)
+ from openenv.core.env_server.http_server import create_app
+ from ..models import OpenAppAction, OpenAppObservation
+ from .openapp_environment import OpenAppEnvironment
+except ImportError:
+ # Standalone imports (when environment is standalone with openenv-core from pip)
+ from openenv.core.env_server.http_server import create_app
+ from openapp_env.models import OpenAppAction, OpenAppObservation
+ from openapp_env.server.openapp_environment import OpenAppEnvironment
+
+# Create the app with web interface and README integration
+# Pass the class (factory) instead of an instance for WebSocket session support
+# Each client gets its own environment instance. The environment reads
+# OPENAPPS_URL from environment variables in __init__.
+app = create_app(OpenAppEnvironment, OpenAppAction, OpenAppObservation, env_name="openapp_env")
+
+
+def main():
+ """
+ Entry point for direct execution via uv run or python -m.
+
+ This function enables running the server without Docker:
+ uv run --project . server
+ python -m envs.openapp_env.server.app
+ openenv serve openapp_env
+
+ """
+ import uvicorn
+
+ uvicorn.run(app, host="0.0.0.0", port=8000)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/envs/openapp_env/server/openapp_environment.py b/envs/openapp_env/server/openapp_environment.py
new file mode 100644
index 000000000..e58375bb6
--- /dev/null
+++ b/envs/openapp_env/server/openapp_environment.py
@@ -0,0 +1,659 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+OpenApp Environment Implementation.
+
+A web application simulation environment that wraps OpenApps and BrowserGym.
+This environment provides agent interaction with simulated web apps including
+calendar, todo, messenger, and maps applications.
+"""
+
+import logging
+import os
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+from typing import Any, Dict, Optional, Tuple
+from uuid import uuid4
+
+logger = logging.getLogger(__name__)
+
+# Support both in-repo and standalone imports
+try:
+ # In-repo imports (when running from OpenEnv repository)
+ from openenv.core.env_server.interfaces import Environment
+ from openenv.core.env_server.types import State
+ from ..models import OpenAppAction, OpenAppObservation
+except ImportError:
+ # Standalone imports (when environment is standalone with openenv-core from pip)
+ from openenv.core.env_server.interfaces import Environment
+ from openenv.core.env_server.types import State
+ from openapp_env.models import OpenAppAction, OpenAppObservation
+
+
+class GenericOpenAppsTask:
+ """
+ A generic task for OpenApps interaction without specific goals.
+
+ This is a simple wrapper that allows BrowserGym to interact with OpenApps
+ without requiring a specific task. For task-based interaction, use the
+ OpenAppsTask from open_apps.tasks.add_tasks_to_browsergym.
+ """
+
+ def __init__(
+ self,
+ base_url: str,
+ seed: int = 1,
+ **kwargs,
+ ) -> None:
+ """
+ Initialize generic OpenApps task.
+
+ Args:
+ base_url: Base URL of the OpenApps server
+ seed: Random seed (required by BrowserGym)
+ **kwargs: Additional arguments (ignored)
+ """
+ try:
+ from browsergym.core.task import AbstractBrowserTask
+ import playwright.sync_api
+ except ImportError:
+ raise ImportError(
+ "BrowserGym is required. Install with: pip install browsergym"
+ )
+
+ # Store as instance attributes
+ self.base_url = base_url
+ self.seed = seed
+
+ # BrowserGym task properties
+ self.viewport = {"width": 1024, "height": 768}
+ self.slow_mo = 100
+ self.timeout = 5000
+
+ # Additional properties that BrowserGym might expect
+ self.locale = None
+ self.timezone_id = None
+ self.geolocation = None
+
+ def setup(
+ self, page: "playwright.sync_api.Page"
+ ) -> Tuple[str, Dict[str, Any]]:
+ """
+ Set up the task by navigating to the base URL.
+
+ Args:
+ page: Playwright page object
+
+ Returns:
+ Tuple of (goal_string, info_dict)
+ """
+ page.goto(self.base_url)
+ return "Explore OpenApps", {}
+
+ def teardown(self) -> None:
+ """Clean up after task completion."""
+ pass
+
+ def validate(
+ self, page: "playwright.sync_api.Page", chat_messages: list[str]
+ ) -> Tuple[float, bool, str, Dict[str, Any]]:
+ """
+ Validate task state and return reward.
+
+ Args:
+ page: Playwright page object
+ chat_messages: List of chat messages
+
+ Returns:
+ Tuple of (reward, done, message, info)
+ """
+ # Generic task never completes automatically
+ return 0.0, False, "", {}
+
+ def cheat(
+ self, page: "playwright.sync_api.Page", chat_messages: list[str]
+ ) -> None:
+ """Cheat method (no-op for generic task)."""
+ pass
+
+
+class OpenAppEnvironment(Environment):
+ """
+ A web application environment that wraps OpenApps and BrowserGym.
+
+ This environment launches OpenApps web server and provides a BrowserGym-like
+ interface for agents to interact with simulated web applications.
+
+ Args:
+ openapps_path: Path to OpenApps directory (default: auto-detect)
+ web_app_port: Port for OpenApps web server (default: 5001)
+ headless: Run browser in headless mode (default: True)
+ task_name: Optional task name to evaluate (e.g., "add_meeting_with_dennis")
+ apps_config: Configuration for apps (default: all enabled)
+ max_steps: Maximum steps per episode (default: 50)
+
+ Example:
+ >>> env = OpenAppEnvironment()
+ >>> obs = env.reset()
+ >>> print(obs.url) # Starting page URL
+ >>>
+ >>> # Click on an element
+ >>> action = OpenAppAction(action_type="click", bid="calendar-btn")
+ >>> obs = env.step(action)
+ >>> print(obs.html)
+ """
+
+ def __init__(
+ self,
+ openapps_url: Optional[str] = None,
+ openapps_path: Optional[str] = None,
+ web_app_port: int = 5001,
+ headless: bool = True,
+ task_name: Optional[str] = None,
+ apps_config: Optional[Dict[str, Any]] = None,
+ max_steps: int = 50,
+ ):
+ """Initialize the OpenApp environment."""
+ self._state = State(episode_id=str(uuid4()), step_count=0)
+ self._max_steps = max_steps
+
+ # OpenApps configuration
+ # Priority: 1. openapps_url, 2. OPENAPPS_URL env var, 3. Try to find/launch
+ self.openapps_url = openapps_url or os.environ.get("OPENAPPS_URL")
+ if not self.openapps_url:
+ self.web_app_port = web_app_port
+ self.openapps_url = f"http://localhost:{web_app_port}"
+
+ self.openapps_path = openapps_path
+ self.headless = headless
+ self.task_name = task_name
+ self.apps_config = apps_config or {}
+
+ # Runtime state
+ self._apps_process: Optional[subprocess.Popen] = None
+ self._browser_env = None
+ self._current_html = ""
+ self._current_url = ""
+ self._current_axtree = ""
+ self._app_state = {}
+ self._last_action_error = None
+ self._episode_reward = 0.0
+
+ def _detect_openapps_path(self) -> str:
+ """
+ Auto-detect OpenApps path.
+
+ Since OpenApps is installed as a Python package, we use the installed
+ package location instead of requiring a separate directory.
+ """
+ # Check if user provided a custom path via environment variable
+ env_path = os.environ.get("OPENAPPS_PATH")
+ if env_path and Path(env_path).exists():
+ return env_path
+
+ # Try to find OpenApps as an installed package
+ try:
+ import open_apps
+
+ openapps_pkg_path = Path(open_apps.__file__).parent.parent
+ if openapps_pkg_path.exists():
+ return str(openapps_pkg_path)
+ except ImportError:
+ pass
+
+ raise ValueError(
+ "OpenApps not found. Please install it with: "
+ "pip install git+https://github.com/facebookresearch/OpenApps.git "
+ "or set OPENAPPS_PATH environment variable."
+ )
+
+ def _launch_openapps_server(self) -> Optional[subprocess.Popen]:
+ """
+ Launch OpenApps web server in background.
+
+ Returns None if server is expected to be already running (OPENAPPS_URL set).
+ """
+ # If OPENAPPS_URL is set, assume server is already running
+ if os.environ.get("OPENAPPS_URL"):
+ logger.info(f"Using existing OpenApps server at {self.openapps_url}")
+ # Wait for server to be available
+ self._wait_for_server(max_wait=5)
+ return None
+
+ # Otherwise, provide helpful error message
+ raise NotImplementedError(
+ "Automatic OpenApps server launch is not yet implemented.\n"
+ "\n"
+ "Please start OpenApps manually in a separate terminal:\n"
+ " 1. Clone OpenApps: git clone https://github.com/facebookresearch/OpenApps.git\n"
+ " 2. Install: cd OpenApps && uv sync\n"
+ " 3. Run: uv run launch.py\n"
+ "\n"
+ "Then set the OPENAPPS_URL environment variable:\n"
+ " export OPENAPPS_URL=http://localhost:5001\n"
+ "\n"
+ "Or use Docker mode which handles this automatically:\n"
+ " python examples/openapp_example.py --mode docker\n"
+ )
+
+ def _wait_for_server(self, max_wait: int = 30):
+ """Wait for OpenApps server to become available."""
+ for i in range(max_wait):
+ try:
+ response = urllib.request.urlopen(self.openapps_url, timeout=2)
+ if response.status == 200:
+ return
+ except Exception:
+ pass
+ time.sleep(1)
+
+ raise TimeoutError(f"OpenApps server did not start within {max_wait} seconds")
+
+ def _initialize_browser(self):
+ """Initialize BrowserGym environment for interaction."""
+ try:
+ from browsergym.core.env import BrowserEnv
+ except ImportError:
+ raise ImportError(
+ "BrowserGym is required for OpenApp environment. "
+ "Install it with: pip install browsergym"
+ )
+
+ # Create BrowserGym environment with generic OpenApps task
+ self._browser_env = BrowserEnv(
+ task_entrypoint=GenericOpenAppsTask,
+ task_kwargs={"base_url": self.openapps_url},
+ headless=self.headless,
+ slow_mo=200, # Slow down actions so they're visible (200ms delay)
+ )
+
+ def _get_current_observation(self) -> Dict[str, Any]:
+ """Extract current observation from browser state."""
+ if self._browser_env is None:
+ return {
+ "html": "",
+ "url": self.openapps_url,
+ "open_pages_urls": [self.openapps_url],
+ "active_page_index": 0,
+ "axtree_txt": "",
+ "app_state": {},
+ }
+
+ # Get browser state (implementation depends on BrowserGym API)
+ # This is a simplified version - actual implementation would use BrowserGym's observation
+ return {
+ "html": self._current_html,
+ "url": self._current_url,
+ "open_pages_urls": [self._current_url],
+ "active_page_index": 0,
+ "axtree_txt": self._current_axtree,
+ "app_state": self._app_state,
+ }
+
+ def reset(self) -> OpenAppObservation:
+ """
+ Reset the environment.
+
+ Returns:
+ OpenAppObservation with initial state
+ """
+ # Reset state
+ self._state = State(episode_id=str(uuid4()), step_count=0)
+ self._episode_reward = 0.0
+ self._last_action_error = None
+
+ # Check if OpenApps server is running, start if needed
+ if self._apps_process is None and not os.environ.get("OPENAPPS_URL"):
+ self._apps_process = self._launch_openapps_server()
+
+ # Initialize browser
+ if self._browser_env is None:
+ self._initialize_browser()
+
+ # Reset the BrowserGym environment
+ try:
+ obs, info = self._browser_env.reset()
+ # Extract observation data from BrowserGym
+ self._current_url = obs.get("url", self.openapps_url)
+ self._current_html = obs.get("dom_txt", "")
+ self._current_axtree = obs.get("axtree_txt", "")
+ self._app_state = {}
+ except Exception as e:
+ logger.warning(f"Failed to reset browser environment: {e}")
+ # Fallback to placeholder values
+ self._current_url = self.openapps_url
+ self._current_html = "OpenApps Ready"
+ self._current_axtree = ""
+ self._app_state = {}
+
+ obs_data = self._get_current_observation()
+
+ return OpenAppObservation(
+ html=obs_data["html"],
+ url=obs_data["url"],
+ open_pages_urls=obs_data["open_pages_urls"],
+ active_page_index=obs_data["active_page_index"],
+ axtree_txt=obs_data["axtree_txt"],
+ app_state=obs_data["app_state"],
+ task_info={"task_name": self.task_name} if self.task_name else None,
+ last_action_error=None,
+ done=False,
+ reward=0.0,
+ )
+
+ def step(self, action: OpenAppAction) -> OpenAppObservation: # type: ignore[override]
+ """
+ Execute a step in the environment.
+
+ Args:
+ action: OpenAppAction to execute
+
+ Returns:
+ OpenAppObservation with resulting state and reward
+ """
+ self._state.step_count += 1
+ self._last_action_error = None
+ reward = 0.0
+
+ try:
+ # Execute action based on type
+ if action.action_type == "click":
+ reward = self._execute_click(action.bid)
+ elif action.action_type == "fill":
+ reward = self._execute_fill(action.bid, action.text)
+ elif action.action_type == "select_option":
+ reward = self._execute_select(action.bid, action.value)
+ elif action.action_type == "goto":
+ reward = self._execute_goto(action.url)
+ elif action.action_type == "scroll":
+ reward = self._execute_scroll(action.direction)
+ elif action.action_type == "send_keys":
+ reward = self._execute_send_keys(action.text)
+ elif action.action_type == "noop":
+ reward = 0.0
+ else:
+ self._last_action_error = f"Unknown action type: {action.action_type}"
+ reward = -0.1
+
+ except Exception as e:
+ self._last_action_error = str(e)
+ reward = -0.1
+
+ # Update cumulative reward
+ self._episode_reward += reward
+
+ # Check if episode is done
+ done = self._state.step_count >= self._max_steps
+
+ # Get current observation
+ obs_data = self._get_current_observation()
+
+ return OpenAppObservation(
+ html=obs_data["html"],
+ url=obs_data["url"],
+ open_pages_urls=obs_data["open_pages_urls"],
+ active_page_index=obs_data["active_page_index"],
+ axtree_txt=obs_data["axtree_txt"],
+ app_state=obs_data["app_state"],
+ task_info={"task_name": self.task_name} if self.task_name else None,
+ last_action_error=self._last_action_error,
+ done=done,
+ reward=reward,
+ metadata={"cumulative_reward": self._episode_reward},
+ )
+
+ def _execute_click(self, bid: str) -> float:
+ """Execute click action. Returns reward.
+
+ Supports two modes:
+ 1. CSS selector mode: If bid starts with '#', '.', or '[', it's treated as a CSS selector
+ and uses Playwright directly (e.g., bid="#msg-input")
+ 2. BrowserGym mode: Otherwise, uses BrowserGym's accessibility tree bid
+ """
+ if self._browser_env is None:
+ return 0.0
+
+ try:
+ # Check if bid is a CSS selector (starts with # or other CSS selector chars)
+ if bid.startswith('#') or bid.startswith('.') or bid.startswith('['):
+ # Use Playwright directly for CSS selectors
+ return self._execute_click_playwright(bid)
+
+ # BrowserGym action format: click("bid")
+ action = f'click("{bid}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", self._current_url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Click failed: {str(e)}"
+ return -0.1
+
+ def _execute_fill(self, bid: str, text: str) -> float:
+ """Execute fill action. Returns reward.
+
+ Supports two modes:
+ 1. CSS selector mode: If bid starts with '#', it's treated as an HTML ID selector
+ and uses Playwright directly (e.g., bid="#msg-input")
+ 2. BrowserGym mode: Otherwise, uses BrowserGym's accessibility tree bid
+ """
+ if self._browser_env is None:
+ return 0.0
+
+ try:
+ # Check if bid is a CSS selector (starts with # or other CSS selector chars)
+ if bid.startswith('#') or bid.startswith('.') or bid.startswith('['):
+ # Use Playwright directly for CSS selectors
+ return self._execute_fill_playwright(bid, text)
+
+ # BrowserGym action format: fill("bid", "text")
+ action = f'fill("{bid}", "{text}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", self._current_url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Fill failed: {str(e)}"
+ return -0.1
+
+ def _execute_fill_playwright(self, selector: str, text: str) -> float:
+ """Execute fill action using Playwright directly with CSS selector."""
+ try:
+ # Access the underlying Playwright page from BrowserGym
+ page = self._browser_env.unwrapped.page
+
+ # Wait for element and fill it
+ page.wait_for_selector(selector, timeout=5000)
+ page.fill(selector, text)
+
+ # Small delay to let the page update
+ page.wait_for_timeout(200)
+
+ # Update observation after action
+ self._update_observation_from_page(page)
+
+ return 0.0
+ except Exception as e:
+ self._last_action_error = f"Fill (Playwright) failed: {str(e)}"
+ return -0.1
+
+ def _execute_click_playwright(self, selector: str) -> float:
+ """Execute click action using Playwright directly with CSS selector."""
+ try:
+ # Access the underlying Playwright page from BrowserGym
+ page = self._browser_env.unwrapped.page
+
+ # Wait for element and click it
+ page.wait_for_selector(selector, timeout=5000)
+ page.click(selector)
+
+ # Longer delay to let HTMX process the request
+ page.wait_for_timeout(500)
+
+ # Update observation after action
+ self._update_observation_from_page(page)
+
+ return 0.0
+ except Exception as e:
+ self._last_action_error = f"Click (Playwright) failed: {str(e)}"
+ return -0.1
+
+ def _execute_press_key_playwright(self, key: str) -> float:
+ """Execute key press using Playwright directly."""
+ try:
+ # Access the underlying Playwright page from BrowserGym
+ page = self._browser_env.unwrapped.page
+
+ # Press the key
+ page.keyboard.press(key)
+
+ # Delay to let the page update
+ page.wait_for_timeout(500)
+
+ # Update observation after action
+ self._update_observation_from_page(page)
+
+ return 0.0
+ except Exception as e:
+ self._last_action_error = f"Press key (Playwright) failed: {str(e)}"
+ return -0.1
+
+ def _update_observation_from_page(self, page) -> None:
+ """Update internal observation state from Playwright page."""
+ try:
+ self._current_url = page.url
+ # Note: We can't easily get axtree from Playwright directly,
+ # so we'll just update URL. The next BrowserGym action will sync the state.
+ except Exception:
+ pass
+
+ def _execute_select(self, bid: str, value: str) -> float:
+ """Execute select option action. Returns reward."""
+ if self._browser_env is None:
+ return 0.0
+
+ try:
+ # BrowserGym action format: select_option("bid", "value")
+ action = f'select_option("{bid}", "{value}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", self._current_url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Select failed: {str(e)}"
+ return -0.1
+
+ def _execute_goto(self, url: str) -> float:
+ """Execute navigation action. Returns reward."""
+ if self._browser_env is None:
+ self._current_url = url
+ return 0.0
+
+ try:
+ # BrowserGym action format: goto("url")
+ action = f'goto("{url}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Goto failed: {str(e)}"
+ self._current_url = url # Update URL even if failed
+ return -0.1
+
+ def _execute_scroll(self, direction: str) -> float:
+ """Execute scroll action. Returns reward."""
+ if self._browser_env is None:
+ return 0.0
+
+ try:
+ # BrowserGym action format: scroll("direction")
+ action = f'scroll("{direction}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", self._current_url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Scroll failed: {str(e)}"
+ return -0.1
+
+ def _execute_send_keys(self, text: str) -> float:
+ """Execute send keys action. Returns reward."""
+ if self._browser_env is None:
+ return 0.0
+
+ try:
+ # Special handling for Enter key - use Playwright directly for reliability
+ if text == "\n" or text.lower() == "enter":
+ return self._execute_press_key_playwright("Enter")
+
+ # BrowserGym action format: send_keys("text")
+ action = f'send_keys("{text}")'
+ obs, reward, done, truncated, info = self._browser_env.step(action)
+
+ # Update current state from observation
+ self._current_url = obs.get("url", self._current_url)
+ self._current_html = obs.get("dom_txt", self._current_html)
+ self._current_axtree = obs.get("axtree_txt", self._current_axtree)
+
+ return float(reward) if reward else 0.0
+ except Exception as e:
+ self._last_action_error = f"Send keys failed: {str(e)}"
+ return -0.1
+
+ @property
+ def state(self) -> State:
+ """
+ Get the current environment state.
+
+ Returns:
+ Current State with episode_id and step_count
+ """
+ return self._state
+
+ def close(self):
+ """Clean up resources."""
+ if hasattr(self, "_browser_env") and self._browser_env is not None:
+ try:
+ self._browser_env.close()
+ except Exception:
+ pass
+ self._browser_env = None
+
+ if hasattr(self, "_apps_process") and self._apps_process is not None:
+ try:
+ self._apps_process.terminate()
+ self._apps_process.wait(timeout=5)
+ except Exception:
+ self._apps_process.kill()
+ self._apps_process = None
+
+ def __del__(self):
+ """Cleanup on deletion."""
+ self.close()
diff --git a/envs/openapp_env/server/start.sh b/envs/openapp_env/server/start.sh
new file mode 100755
index 000000000..360bab3e9
--- /dev/null
+++ b/envs/openapp_env/server/start.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Startup script for OpenApp Environment Docker container
+# This script starts both the OpenApps server and the FastAPI environment server
+
+set -e
+
+echo "Starting OpenApp Environment..."
+
+# Start OpenApps server in the background
+echo "Starting OpenApps server on port ${OPENAPPS_PORT:-5001}..."
+cd /app/openapps
+# Run launch.py directly - it uses Hydra and needs the config directory
+# Redirect OpenApps output to a log file so we can debug if needed
+python launch.py > /tmp/openapps.log 2>&1 &
+OPENAPPS_PID=$!
+
+# Wait for OpenApps server to be ready
+echo "Waiting for OpenApps server to be ready..."
+for i in {1..60}; do
+ # Check if OpenApps server is responding using curl
+ if curl -sf http://localhost:${OPENAPPS_PORT:-5001} >/dev/null 2>&1; then
+ echo "OpenApps server is ready on port ${OPENAPPS_PORT:-5001}!"
+ break
+ fi
+ if [ $i -eq 60 ]; then
+ echo "ERROR: OpenApps server failed to start within 60 seconds"
+ echo "OpenApps log output:"
+ cat /tmp/openapps.log || echo "No log file found"
+ kill $OPENAPPS_PID 2>/dev/null || true
+ exit 1
+ fi
+ sleep 1
+done
+
+# Start the FastAPI environment server
+echo "Starting FastAPI environment server on port 8000..."
+cd /app/env
+exec uvicorn openapp_env.server.app:app --host 0.0.0.0 --port 8000
diff --git a/envs/openapp_env/test_openapp_env.py b/envs/openapp_env/test_openapp_env.py
new file mode 100644
index 000000000..f1d7ff94f
--- /dev/null
+++ b/envs/openapp_env/test_openapp_env.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+Simple test script for OpenApp Environment.
+
+This script tests the basic functionality of the OpenApp environment
+to ensure it follows OpenEnv standards.
+
+Usage:
+ # From OpenEnv root directory
+ python3 envs/openapp_env/test_openapp_env.py
+
+ # Or from openapp_env directory
+ cd envs/openapp_env
+ python3 test_openapp_env.py
+"""
+
+import sys
+from pathlib import Path
+
+# Add src to path for local testing
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+from openapp_env.models import OpenAppAction, OpenAppObservation
+from openapp_env.server.openapp_environment import OpenAppEnvironment
+
+
+def test_models():
+ """Test that models are properly defined."""
+ print("Testing models...")
+
+ # Test creating an action
+ action = OpenAppAction(action_type="noop")
+ assert action.action_type == "noop"
+
+ # Test click action
+ click_action = OpenAppAction(action_type="click", bid="test-btn")
+ assert click_action.bid == "test-btn"
+
+ # Test fill action
+ fill_action = OpenAppAction(action_type="fill", bid="input", text="Hello")
+ assert fill_action.text == "Hello"
+
+ print("ā Models test passed")
+
+
+def test_environment_basic():
+ """Test basic environment functionality."""
+ print("\nTesting environment...")
+
+ try:
+ # Create environment (note: this will check if OpenApps is installed as a package)
+ env = OpenAppEnvironment(
+ max_steps=10,
+ )
+
+ # Test that environment has required methods
+ assert hasattr(env, "reset")
+ assert hasattr(env, "step")
+ assert hasattr(env, "state")
+ assert hasattr(env, "close")
+
+ print("ā Environment structure test passed")
+
+ except (ValueError, ImportError) as e:
+ # Expected if OpenApps is not installed as a package
+ if "OpenApps not found" in str(e) or "open_apps" in str(e):
+ print(
+ "ā Environment structure test passed (OpenApps not installed, expected)"
+ )
+ else:
+ raise
+
+
+def test_client_server_contract():
+ """Test that client and server follow the contract."""
+ print("\nTesting client-server contract...")
+
+ # Test that action can be serialized to dict
+ action = OpenAppAction(
+ action_type="click", bid="test-123", metadata={"test": "value"}
+ )
+
+ # Simulate what client._step_payload would do
+ payload = {
+ "action_type": action.action_type,
+ "bid": action.bid,
+ "metadata": action.metadata,
+ }
+
+ assert payload["action_type"] == "click"
+ assert payload["bid"] == "test-123"
+
+ # Test observation construction
+ obs = OpenAppObservation(
+ html="",
+ url="http://localhost:5001",
+ open_pages_urls=["http://localhost:5001"],
+ done=False,
+ reward=0.0,
+ )
+
+ assert obs.url == "http://localhost:5001"
+ assert obs.done is False
+
+ print("ā Client-server contract test passed")
+
+
+def main():
+ """Run all tests."""
+ print("=" * 60)
+ print("OpenApp Environment - Structure Tests")
+ print("=" * 60)
+
+ try:
+ test_models()
+ test_environment_basic()
+ test_client_server_contract()
+
+ print("\n" + "=" * 60)
+ print("All tests passed! ā")
+ print("=" * 60)
+ print("\nNote: Full integration tests require:")
+ print(
+ "1. OpenApps installed: pip install git+https://github.com/facebookresearch/OpenApps.git"
+ )
+ print("2. Playwright browsers installed: playwright install chromium")
+ print("3. BrowserGym dependencies installed")
+
+ except Exception as e:
+ print(f"\nā Test failed: {e}")
+ import traceback
+
+ traceback.print_exc()
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/openapp_example.py b/examples/openapp_example.py
new file mode 100644
index 000000000..b7ca05fa1
--- /dev/null
+++ b/examples/openapp_example.py
@@ -0,0 +1,362 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+Example usage of the OpenApp Environment.
+
+This script demonstrates how to use the OpenApp environment with OpenEnv.
+It can be run in two modes:
+1. With Docker: Uses the Docker image to run the environment
+2. Local: Directly uses the OpenAppEnvironment class (requires OpenApps installed)
+
+Usage:
+ # Run with Docker (recommended)
+ python examples/openapp_example.py --mode docker
+
+ # Run locally without Docker
+ python examples/openapp_example.py --mode local
+
+ # Run with custom number of steps
+ python examples/openapp_example.py --mode docker --num-steps 20
+
+Visualization Options:
+ # To SEE the browser window and watch agent interactions in real-time:
+ #
+ # Terminal 1: Start OpenApps server with visible browser
+ cd OpenApps
+ python OpenApps/launch.py browsergym_env_args.headless=False
+
+ # Terminal 2: Run your agent code
+ export OPENAPPS_URL=http://localhost:5001
+ python examples/openapp_example.py --mode local
+
+ # Or access the web interface directly in your browser:
+ # - OpenApps: http://localhost:5001
+ # - Calendar: http://localhost:5001/calendar
+ # - Todo: http://localhost:5001/todo
+ # - Messenger: http://localhost:5001/messages
+ # - Maps: http://localhost:5001/maps
+
+ # Docker mode web interface
+ # - Web UI: http://localhost:8000/web
+ # - API docs: http://localhost:8000/docs
+
+Important:
+ The browser visualization is controlled by the OpenApps SERVER, not the client.
+ You must launch the server with 'browsergym_env_args.headless=False' to see
+ the browser window.
+"""
+
+import argparse
+import os
+import sys
+import time
+from pathlib import Path
+
+# Add src to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+
+def run_with_docker(num_steps: int = 15, headless: bool = True):
+ """Run OpenApp environment using Docker container."""
+ from openapp_env import OpenAppAction, OpenAppEnv
+
+ print("=" * 70)
+ print("Starting OpenApp environment with Docker...")
+ print(f"Headless mode: {headless}")
+ print("=" * 70)
+
+ try:
+ # Create environment from Docker image
+ env = OpenAppEnv.from_docker_image("openapp-env:latest")
+
+ # Reset to start a new session
+ print("\n[1/4] Resetting environment...")
+ result = env.reset()
+ print(f"ā Environment reset")
+ print(f" Starting URL: {result.observation.url}")
+ print(f" Open pages: {len(result.observation.open_pages_urls)}")
+ print(f" HTML length: {len(result.observation.html)} characters")
+
+ # Example actions to demonstrate different action types
+ actions = [
+ {
+ "description": "Navigate to calendar app",
+ "action": OpenAppAction(
+ action_type="goto", url="http://localhost:5001/calendar"
+ ),
+ },
+ {
+ "description": "Scroll down to see more content",
+ "action": OpenAppAction(action_type="scroll", direction="down"),
+ },
+ {
+ "description": "Navigate to todo app",
+ "action": OpenAppAction(
+ action_type="goto", url="http://localhost:5001/todo"
+ ),
+ },
+ {
+ "description": "Navigate to messenger app",
+ "action": OpenAppAction(
+ action_type="goto", url="http://localhost:5001/messenger"
+ ),
+ },
+ {
+ "description": "Navigate to maps app",
+ "action": OpenAppAction(
+ action_type="goto", url="http://localhost:5001/maps"
+ ),
+ },
+ {
+ "description": "Navigate back to home",
+ "action": OpenAppAction(
+ action_type="goto", url="http://localhost:5001"
+ ),
+ },
+ ]
+
+ # Run demonstration steps
+ print(f"\n[2/4] Running {min(num_steps, len(actions))} demonstration steps...")
+ for i, action_info in enumerate(actions[:num_steps]):
+ print(f"\nStep {i+1}: {action_info['description']}")
+ print(f" Action type: {action_info['action'].action_type}")
+
+ result = env.step(action_info["action"])
+
+ print(f" ā Action executed")
+ print(f" Current URL: {result.observation.url}")
+ print(f" Reward: {result.reward}")
+ print(f" Done: {result.done}")
+
+ if result.observation.last_action_error:
+ print(f" ā ļø Error: {result.observation.last_action_error}")
+
+ # Check app state if available
+ if result.observation.app_state:
+ print(
+ f" App state keys: {list(result.observation.app_state.keys())}"
+ )
+
+ # Small delay for readability
+ time.sleep(0.5)
+
+ if result.done:
+ print(f"\nā Episode finished at step {i+1}!")
+ break
+
+ # Get final state
+ print(f"\n[3/4] Getting final environment state...")
+ state = env.state()
+ print(f"ā Final state:")
+ print(f" Episode ID: {state.episode_id}")
+ print(f" Total steps: {state.step_count}")
+
+ # Summary
+ print(f"\n[4/4] Session Summary:")
+ print(f" Steps taken: {state.step_count}")
+ print(f" Final URL: {result.observation.url}")
+ print(f" Episode complete: {result.done}")
+
+ # Web interface info
+ print(f"\n" + "=" * 70)
+ print(f"š” TIP: Access the web interface at http://localhost:8000/web")
+ print(f" - Interactive UI for manual testing")
+ print(f" - API documentation at http://localhost:8000/docs")
+ print(f"=" * 70)
+
+ except Exception as e:
+ print(f"\nā Error: {e}")
+ import traceback
+
+ traceback.print_exc()
+ return 1
+ finally:
+ print("\n[Cleanup] Closing environment...")
+ env.close()
+ print("ā Environment closed")
+
+ return 0
+
+
+def run_local(num_steps: int = 15, headless: bool = True):
+ """Run OpenApp environment locally without Docker."""
+
+ # Check if OPENAPPS_URL is set
+ if not os.environ.get("OPENAPPS_URL"):
+ print("=" * 70)
+ print("ā ERROR: OPENAPPS_URL not set")
+ print("=" * 70)
+ print("\nLocal mode requires OpenApps server to be running.")
+ print("\nPlease follow these steps:")
+ print("\n1. Start OpenApps server in a separate terminal:")
+ print(" cd /path/to/OpenApps")
+ print(" uv run launch.py")
+ print("\n2. Set the OPENAPPS_URL environment variable:")
+ print(" export OPENAPPS_URL=http://localhost:5001")
+ print("\n3. Run this script again:")
+ print(" python examples/openapp_example.py --mode local")
+ print("\nAlternatively, use Docker mode (recommended):")
+ print(" python examples/openapp_example.py --mode docker")
+ print("=" * 70)
+ return 1
+
+ try:
+ # NOTE: This example imports from the server module directly for local development.
+ # This is intentional for local testing/debugging where no HTTP server is involved.
+ # In production, use the client API (OpenAppEnv) which communicates over HTTP/WebSocket.
+ # See run_with_docker() for the recommended production pattern.
+ from openapp_env.models import OpenAppAction
+ from openapp_env.server.openapp_environment import OpenAppEnvironment
+ except ImportError as e:
+ print(f"ā Error importing local modules: {e}")
+ print("\nMake sure you have installed the environment:")
+ print(" cd envs/openapp_env")
+ print(" pip install -e .")
+ return 1
+
+ print("=" * 70)
+ print("Starting OpenApp environment locally...")
+ print(f"Using OpenApps server at: {os.environ.get('OPENAPPS_URL')}")
+ print(f"Headless mode: {headless}")
+ print("=" * 70)
+
+ try:
+ # Create environment locally
+ print("\n[1/4] Initializing local environment...")
+ env = OpenAppEnvironment(
+ openapps_url=os.environ.get("OPENAPPS_URL"),
+ headless=headless,
+ max_steps=50,
+ )
+
+ # Reset environment
+ print("\n[2/4] Resetting environment...")
+ result = env.reset()
+ print(f"ā Environment reset")
+ print(f" Starting URL: {result.url}")
+ print(f" HTML length: {len(result.html)} characters")
+
+ # Take some example steps
+ print(f"\n[3/4] Running {num_steps} steps...")
+ for i in range(num_steps):
+ # Simple actions for demonstration
+ actions = [
+ OpenAppAction(
+ action_type="goto", url=f"{os.environ.get('OPENAPPS_URL')}/calendar"
+ ),
+ OpenAppAction(action_type="scroll", direction="down"),
+ OpenAppAction(
+ action_type="goto", url=f"{os.environ.get('OPENAPPS_URL')}/todo"
+ ),
+ OpenAppAction(action_type="noop"),
+ ]
+
+ action = actions[i % len(actions)]
+ result = env.step(action)
+
+ if i % 5 == 0 or result.done:
+ print(f"Step {i+1}:")
+ print(f" Action: {action.action_type}")
+ print(f" URL: {result.url}")
+ print(f" Reward: {result.reward}")
+ print(f" Done: {result.done}")
+
+ if result.done:
+ print(f"\nā Episode finished at step {i+1}!")
+ break
+
+ # Get final state
+ print(f"\n[4/4] Final state:")
+ print(f" Episode ID: {env.state.episode_id}")
+ print(f" Total steps: {env.state.step_count}")
+
+ except Exception as e:
+ print(f"\nā Error: {e}")
+ import traceback
+
+ traceback.print_exc()
+ return 1
+ finally:
+ print("\n[Cleanup] Closing environment...")
+ env.close()
+ print("ā Environment closed")
+
+ return 0
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="OpenApp Environment Example",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ # Run with Docker (recommended)
+ python examples/openapp_example.py --mode docker
+
+ # Run locally without Docker
+ python examples/openapp_example.py --mode local
+
+ # Show browser window to visualize agent actions
+ python examples/openapp_example.py --mode local --show-browser
+
+ # Run with custom number of steps
+ python examples/openapp_example.py --mode docker --num-steps 20
+
+Visualization:
+ - Use --show-browser to see the browser window and watch agent interactions
+ - Access OpenApps web interface at http://localhost:5001 (when server is running)
+ - Docker mode web interface: http://localhost:8000/web
+
+Note:
+ - Docker mode requires: docker build -t openapp-env:latest -f envs/openapp_env/server/Dockerfile envs/openapp_env
+ - Local mode requires: pip install -e envs/openapp_env && playwright install chromium
+ """,
+ )
+
+ parser.add_argument(
+ "--mode",
+ choices=["docker", "local"],
+ default="docker",
+ help="Run mode: 'docker' (recommended) or 'local'",
+ )
+ parser.add_argument(
+ "--num-steps", type=int, default=15, help="Number of steps to run (default: 15)"
+ )
+ parser.add_argument(
+ "--headless",
+ action="store_true",
+ default=False,
+ help="Run browser in headless mode (no visible window)",
+ )
+ parser.add_argument(
+ "--show-browser",
+ action="store_true",
+ default=False,
+ help="Show browser window (opposite of --headless, easier to remember)",
+ )
+
+ args = parser.parse_args()
+
+ # Determine headless mode: default to True unless --show-browser is used
+ headless = not args.show_browser if args.show_browser else args.headless
+
+ print("\n" + "=" * 70)
+ print("OpenApp Environment Example")
+ print("=" * 70)
+ print(f"Mode: {args.mode}")
+ print(f"Steps: {args.num_steps}")
+ print(f"Headless: {headless}")
+
+ if args.mode == "docker":
+ return run_with_docker(args.num_steps, headless)
+ else:
+ return run_local(args.num_steps, headless)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/examples/openapp_recording_demo.py b/examples/openapp_recording_demo.py
new file mode 100644
index 000000000..c5b2c40a0
--- /dev/null
+++ b/examples/openapp_recording_demo.py
@@ -0,0 +1,873 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+Demo script optimized for recording videos of OpenApp environment interactions.
+
+This script provides slower, more visible agent interactions suitable for video recording.
+
+SETUP FOR RECORDING:
+
+Terminal 1 - Start OpenApps server with visible browser:
+ cd OpenApps
+ python launch.py browsergym_env_args.headless=False
+
+Terminal 2 - Run this script:
+ export OPENAPPS_URL=http://localhost:5001
+ python examples/openapp_recording_demo.py
+
+Then use screen recording software to capture the browser window.
+
+Options:
+ --scenario: Choose demo scenario (calendar, todo, messages, shopping, all)
+ --delay: Seconds to wait between actions (default: 2)
+ --verbose: Print detailed information during recording
+"""
+
+import argparse
+import os
+import sys
+import time
+from pathlib import Path
+
+# Add src to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from openapp_env.models import OpenAppAction
+from openapp_env.server.openapp_environment import OpenAppEnvironment
+
+
+class RecordingDemo:
+ """Demo scenarios optimized for video recording."""
+
+ def __init__(self, openapps_url: str, delay: float = 2.0, verbose: bool = False):
+ """
+ Initialize recording demo.
+
+ Args:
+ openapps_url: URL of OpenApps server
+ delay: Seconds to wait between actions
+ verbose: Print detailed information
+ """
+ self.openapps_url = openapps_url
+ self.delay = delay
+ self.verbose = verbose
+ self.env = None
+
+ def setup(self):
+ """Set up the environment."""
+ print("=" * 70)
+ print("OpenApp Recording Demo")
+ print("=" * 70)
+ print(f"OpenApps server: {self.openapps_url}")
+ print(f"Action delay: {self.delay}s")
+ print(f"Verbose mode: {self.verbose}")
+ print()
+ print("āŗļø START YOUR SCREEN RECORDING NOW!")
+ print(" Recording the browser window that appears...")
+ print()
+
+ # Give user time to start recording
+ for i in range(3, 0, -1):
+ print(f" Starting in {i}...", end='\r')
+ time.sleep(1)
+ print("\n" + "=" * 70)
+ print()
+
+ # Create environment
+ self.env = OpenAppEnvironment(
+ openapps_url=self.openapps_url,
+ headless=False, # Visible browser
+ max_steps=100,
+ )
+
+ def wait(self, message: str = None):
+ """Wait between actions with optional message."""
+ if message and self.verbose:
+ print(f" {message}")
+ time.sleep(self.delay)
+
+ def step(self, action: OpenAppAction, description: str):
+ """Execute action with description."""
+ print(f"š¬ {description}")
+ if self.verbose:
+ print(f" Action: {action.action_type}")
+ if hasattr(action, 'url') and action.url:
+ print(f" URL: {action.url}")
+ if hasattr(action, 'bid') and action.bid:
+ print(f" Element: {action.bid}")
+ if hasattr(action, 'text') and action.text:
+ print(f" Text: {action.text}")
+
+ result = self.env.step(action)
+
+ if self.verbose:
+ print(f" ā Current URL: {result.url}")
+ if result.last_action_error:
+ print(f" ā ļø Error: {result.last_action_error}")
+
+ self.wait()
+ return result
+
+ def calendar_scenario(self):
+ """Demonstrate calendar interactions with meaningful actions."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Calendar Management")
+ print("=" * 70)
+ print("Demonstrating: View calendar, switch views, navigate months, view events\n")
+
+ self.wait("Resetting environment...")
+ self.env.reset()
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/10 - Navigate to home page"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar"),
+ "2/10 - Open calendar application (Calendar view)"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "3/10 - Scroll down to view calendar grid"
+ )
+
+ # Switch to Agenda view
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar?view=agenda"),
+ "4/10 - Switch to Agenda view"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "5/10 - Browse agenda events list"
+ )
+
+ # Switch back to Calendar view
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar?view=calendar"),
+ "6/10 - Switch back to Calendar view"
+ )
+
+ # Navigate to next month
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar/calendar_content/2026/1?view=calendar"),
+ "7/10 - Navigate to next month (January 2026)"
+ )
+
+ # Navigate to previous month
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar/calendar_content/2025/12?view=calendar"),
+ "8/10 - Navigate back to current month (December 2025)"
+ )
+
+ # View a specific event (if available)
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar/event/1"),
+ "9/10 - View event details"
+ )
+
+ # Return to calendar
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar"),
+ "10/10 - Return to calendar main view"
+ )
+
+ print("ā Calendar scenario complete!\n")
+
+ def todo_scenario(self):
+ """Demonstrate todo list interactions with meaningful actions."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Todo List Management")
+ print("=" * 70)
+ print("Demonstrating: Browse tasks, view edit interface, navigate todo items\n")
+
+ self.wait("Resetting environment...")
+ self.env.reset()
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/11 - Navigate to home page"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo"),
+ "2/11 - Open todo list application"
+ )
+
+ # Give viewers time to see the initial todo list
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "3/11 - Browse through todo items"
+ )
+
+ # View edit interface for first todo
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo/edit/0"),
+ "4/11 - Open edit interface for first task"
+ )
+
+ # Return to main todo view
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo"),
+ "5/11 - Return to todo list"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "6/11 - Scroll through more tasks"
+ )
+
+ # View edit interface for another todo
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo/edit/5"),
+ "7/11 - Open edit interface for another task"
+ )
+
+ # Return to main todo view
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo"),
+ "8/11 - Return to todo list"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "9/11 - Final browse through todo items"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="up"),
+ "10/11 - Scroll back to top"
+ )
+
+ # Return to home
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "11/11 - Return to home page"
+ )
+
+ print("ā Todo scenario complete!\n")
+
+ def messages_scenario(self):
+ """Demonstrate messenger with actual message sending."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Messenger - Send Message")
+ print("=" * 70)
+ print("Demonstrating: Browse conversations, send interactive message to Alice\n")
+
+ self.wait("Resetting environment...")
+ self.env.reset()
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/14 - Navigate to home page"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages"),
+ "2/14 - Open messenger application"
+ )
+
+ # View conversations list
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "3/14 - Browse conversation list"
+ )
+ # time.sleep(3)
+ # Open conversation with Alice
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages/Alice"),
+ "4/14 - Open conversation with Alice"
+ )
+
+ # Scroll to view conversation history
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "5/14 - Scroll through message history with Alice"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="up"),
+ "6/14 - Scroll back to message input"
+ )
+
+ # Click on the message input field to focus it
+ print(" ā Clicking message input field to focus it...")
+ result = self.step(
+ OpenAppAction(
+ action_type="click",
+ bid="msg-input"
+ ),
+ "7/14 - Click message input field"
+ )
+ if hasattr(result, 'last_action_error') and result.last_action_error:
+ print(f" ā ļø Error clicking input: {result.last_action_error}")
+
+ # Type the message using fill with CSS selector (directly targets the input element by HTML ID)
+ print(" ā Typing message using fill with CSS selector...")
+ result = self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#msg-input",
+ text="Hi, We are submitting the OpenApp for OpenEnv Hackathon and we are super excited about this!"
+ ),
+ "8/14 - Type message: 'Hi, We are submitting the OpenApp for OpenEnv Hackathon...'"
+ )
+ if hasattr(result, 'last_action_error') and result.last_action_error:
+ print(f" ā ļø Error typing message: {result.last_action_error}")
+
+ # Pause to let viewers see the typed message
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "9/14 - View typed message before sending"
+ )
+
+ # Send the message by pressing Enter
+ print(" ā Pressing Enter to send message...")
+ result = self.step(
+ OpenAppAction(
+ action_type="send_keys",
+ text="\n"
+ ),
+ "10/14 - Press Enter to send message"
+ )
+ if hasattr(result, 'last_action_error') and result.last_action_error:
+ print(f" ā ļø Error sending message: {result.last_action_error}")
+
+ # Pause to see Alice's response
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "11/14 - View Alice's response"
+ )
+
+ # Go back to messages list
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages"),
+ "12/14 - Return to conversations list"
+ )
+
+ # Return to home
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "13/14 - Return to home page"
+ )
+
+ # Final pause
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "14/14 - Demo complete"
+ )
+
+ print("ā Messenger scenario complete!\n")
+
+ def codeeditor_scenario(self):
+ """Demonstrate code editor by typing a PyTorch training loop."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Code Editor - PyTorch Training Loop")
+ print("=" * 70)
+ print("Demonstrating: Create file, type code interactively, save file\n")
+
+ self.wait("Resetting environment...")
+ self.env.reset()
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/15 - Navigate to home page"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/codeeditor"),
+ "2/15 - Open code editor application"
+ )
+
+ # Give viewers time to see the code editor interface
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "3/15 - View code editor interface with file tree"
+ )
+
+ # Create a new file
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/codeeditor/train.py"),
+ "4/15 - Create new file: train.py"
+ )
+
+ # The PyTorch training loop code to type
+ pytorch_code = '''import torch
+import torch.nn as nn
+import torch.optim as optim
+
+# Define a simple neural network
+class SimpleNet(nn.Module):
+ def __init__(self):
+ super(SimpleNet, self).__init__()
+ self.fc1 = nn.Linear(784, 128)
+ self.fc2 = nn.Linear(128, 10)
+ self.relu = nn.ReLU()
+
+ def forward(self, x):
+ x = self.relu(self.fc1(x))
+ return self.fc2(x)
+
+# Initialize model, loss, and optimizer
+model = SimpleNet()
+criterion = nn.CrossEntropyLoss()
+optimizer = optim.Adam(model.parameters(), lr=0.001)
+
+# Training loop
+for epoch in range(10):
+ optimizer.zero_grad()
+ outputs = model(inputs)
+ loss = criterion(outputs, labels)
+ loss.backward()
+ optimizer.step()
+ print(f"Epoch {epoch}: Loss = {loss.item():.4f}")
+'''
+
+ # Type the code line by line for dramatic effect
+ lines = pytorch_code.strip().split('\n')
+
+ # Type first few lines with imports
+ current_text = ""
+ for i, line in enumerate(lines[:4]): # imports
+ current_text += line + "\n"
+
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#editor",
+ text=current_text.strip()
+ ),
+ "5/15 - Type imports: torch, nn, optim"
+ )
+
+ # Add class definition
+ for i, line in enumerate(lines[4:10]): # class header
+ current_text += line + "\n"
+
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#editor",
+ text=current_text.strip()
+ ),
+ "6/15 - Define SimpleNet class with layers"
+ )
+
+ # Add forward method
+ for i, line in enumerate(lines[10:15]): # forward method
+ current_text += line + "\n"
+
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#editor",
+ text=current_text.strip()
+ ),
+ "7/15 - Add forward() method"
+ )
+
+ # Add model initialization
+ for i, line in enumerate(lines[15:20]): # model init
+ current_text += line + "\n"
+
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#editor",
+ text=current_text.strip()
+ ),
+ "8/15 - Initialize model, loss function, optimizer"
+ )
+
+ # Add training loop
+ for i, line in enumerate(lines[20:]): # training loop
+ current_text += line + "\n"
+
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#editor",
+ text=current_text.strip()
+ ),
+ "9/15 - Add training loop with forward/backward pass"
+ )
+
+ # Pause to let viewers see the complete code
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "10/15 - View complete PyTorch training code"
+ )
+
+ # Save the file by clicking the save button
+ self.step(
+ OpenAppAction(action_type="click", bid="Save"),
+ "13/15 - Save the file"
+ )
+
+ # Return to code editor index
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/codeeditor"),
+ "14/15 - Return to code editor to see saved file"
+ )
+
+ # Return to home
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "15/15 - Return to home page"
+ )
+
+ print("ā Code Editor scenario complete!\n")
+
+ def maps_scenario(self):
+ """Demonstrate maps with search and landmark exploration."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Maps Exploration with Search")
+ print("=" * 70)
+ print("Demonstrating: Search locations, view landmarks, explore map areas\n")
+
+ self.wait("Resetting environment...")
+ self.env.reset()
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/12 - Navigate to home page"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps"),
+ "2/12 - Open maps application"
+ )
+
+ # Give viewers time to see the initial map with landmarks
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "3/12 - View map with landmarks and search interface"
+ )
+
+ # Scroll to explore different parts of the interface
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "4/12 - Scroll to view route planning interface"
+ )
+
+ # Pause to show the route planning form
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "5/12 - View route planning controls (From/To location fields)"
+ )
+
+ # Scroll back up to the map
+ self.step(
+ OpenAppAction(action_type="scroll", direction="up"),
+ "6/12 - Scroll back to main map view"
+ )
+
+ # Use the search API to find a location (this demonstrates the search feature)
+ # Search for "San Francisco" to pan the map
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps/where?q=San+Francisco"),
+ "7/12 - Search for location: San Francisco"
+ )
+
+ # Return to maps to see the result
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps"),
+ "8/12 - Return to map view"
+ )
+
+ # Explore the map by scrolling
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "9/12 - Pan map to explore different areas"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="up"),
+ "10/12 - Pan map back"
+ )
+
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "11/12 - Final view of map with all features"
+ )
+
+ # Return to home
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "12/12 - Return to home page"
+ )
+
+ print("ā Maps scenario complete!\n")
+
+ def app_tour_scenario(self):
+ """Tour through all applications with meaningful interactions."""
+ print("\n" + "=" * 70)
+ print("SCENARIO: Complete App Tour")
+ print("=" * 70)
+ print("Demonstrating: All OpenApps applications with diverse interactions\n")
+
+ self.wait("Starting tour...")
+ self.env.reset()
+
+ # Home page
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "1/21 - Start at home page"
+ )
+
+ # === Calendar ===
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar"),
+ "2/21 - Calendar: View calendar application"
+ )
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "3/21 - Browse calendar events"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar?view=agenda"),
+ "4/21 - Switch to Agenda view"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/calendar/event/1"),
+ "5/21 - View event details"
+ )
+
+ # === Todo ===
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo"),
+ "6/21 - Todo: Open task manager"
+ )
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "7/21 - Browse todo items"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo/edit/0"),
+ "8/21 - View task edit interface"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/todo"),
+ "9/21 - Return to todo list"
+ )
+
+ # === Messenger ===
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages"),
+ "10/26 - Messenger: View conversations list"
+ )
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "11/26 - Browse conversations"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages/Alice"),
+ "12/26 - Open conversation with Alice"
+ )
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "13/26 - View message history"
+ )
+ # Send a message to Alice
+ self.step(
+ OpenAppAction(
+ action_type="fill",
+ bid="#msg-input",
+ text="Hi, We are submitting the OpenApp for OpenEnv Hackathon and we are super excited about this!"
+ ),
+ "14/26 - Type hackathon message to Alice"
+ )
+ self.step(
+ OpenAppAction(action_type="send_keys", text="\n"),
+ "15/26 - Send message (press Enter)"
+ )
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "16/26 - View Alice's response"
+ )
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/messages"),
+ "17/26 - Return to conversations list"
+ )
+
+ # === Maps ===
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps"),
+ "18/26 - Maps: Open navigation app"
+ )
+ self.step(
+ OpenAppAction(action_type="noop"),
+ "19/26 - View maps with landmarks"
+ )
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "20/26 - View route planning interface"
+ )
+
+ # Search for a location
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps/where?q=Golden+Gate+Park"),
+ "21/26 - Search: Golden Gate Park"
+ )
+
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/maps"),
+ "22/26 - Return to map view"
+ )
+
+ self.step(
+ OpenAppAction(action_type="scroll", direction="down"),
+ "23/26 - Explore map areas"
+ )
+
+ # Return home
+ self.step(
+ OpenAppAction(action_type="goto", url=f"{self.openapps_url}/"),
+ "24/26 - Return to home page"
+ )
+
+ print("ā App tour complete!\n")
+
+ def cleanup(self):
+ """Clean up and close environment."""
+ print("\n" + "=" * 70)
+ print("š¬ Recording Demo Complete!")
+ print("=" * 70)
+ print()
+ print("ā¹ļø STOP YOUR SCREEN RECORDING NOW!")
+ print()
+ print("Waiting 1 seconds before cleanup...")
+ time.sleep(1)
+
+ if self.env:
+ self.env.close()
+ print("ā Environment closed")
+ print()
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="OpenApp Recording Demo - Optimized for video recording",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ # Tour all apps (recommended for comprehensive demo)
+ python examples/openapp_recording_demo.py --scenario all
+
+ # Record calendar interactions only
+ python examples/openapp_recording_demo.py --scenario calendar
+
+ # Record todo list interactions
+ python examples/openapp_recording_demo.py --scenario todo
+
+ # Record messenger interactions
+ python examples/openapp_recording_demo.py --scenario messages
+
+ # Record maps interactions
+ python examples/openapp_recording_demo.py --scenario maps
+
+ # Record code editor with PyTorch training loop
+ python examples/openapp_recording_demo.py --scenario codeeditor
+
+ # Slower pacing for detailed recording (3 seconds between actions)
+ python examples/openapp_recording_demo.py --scenario calendar --delay 3
+
+ # Verbose output with detailed logs
+ python examples/openapp_recording_demo.py --scenario all --verbose
+
+Before running:
+ 1. Terminal 1: cd OpenApps && python launch.py browsergym_env_args.headless=False
+ 2. Terminal 2: export OPENAPPS_URL=http://localhost:5001
+ 3. Start your screen recording software
+ 4. Run this script
+ """,
+ )
+
+ parser.add_argument(
+ "--scenario",
+ choices=["calendar", "todo", "messages", "maps", "codeeditor", "all"],
+ default="all",
+ help="Demo scenario to record (default: all)",
+ )
+ parser.add_argument(
+ "--delay",
+ type=float,
+ default=2.0,
+ help="Seconds to wait between actions (default: 2.0)",
+ )
+ parser.add_argument(
+ "--verbose",
+ action="store_true",
+ help="Print detailed information during recording",
+ )
+
+ args = parser.parse_args()
+
+ # Check if OPENAPPS_URL is set
+ openapps_url = os.environ.get("OPENAPPS_URL")
+ if not openapps_url:
+ print("=" * 70)
+ print("ā ERROR: OPENAPPS_URL not set")
+ print("=" * 70)
+ print()
+ print("Please set up OpenApps server first:")
+ print()
+ print("Terminal 1 - Start OpenApps server with visible browser:")
+ print(" cd OpenApps")
+ print(" python launch.py browsergym_env_args.headless=False")
+ print()
+ print("Terminal 2 - Set URL and run this script:")
+ print(" export OPENAPPS_URL=http://localhost:5001")
+ print(" python examples/openapp_recording_demo.py")
+ print()
+ print("=" * 70)
+ return 1
+
+ # Create demo
+ demo = RecordingDemo(
+ openapps_url=openapps_url,
+ delay=args.delay,
+ verbose=args.verbose,
+ )
+
+ try:
+ # Setup
+ demo.setup()
+
+ # Run selected scenario
+ if args.scenario == "all":
+ demo.app_tour_scenario()
+ elif args.scenario == "calendar":
+ demo.calendar_scenario()
+ elif args.scenario == "todo":
+ demo.todo_scenario()
+ elif args.scenario == "messages":
+ demo.messages_scenario()
+ elif args.scenario == "maps":
+ demo.maps_scenario()
+ elif args.scenario == "codeeditor":
+ demo.codeeditor_scenario()
+
+ return 0
+
+ except KeyboardInterrupt:
+ print("\n\nā ļø Recording interrupted by user")
+ return 1
+ except Exception as e:
+ print(f"\n\nā Error during recording: {e}")
+ import traceback
+ traceback.print_exc()
+ return 1
+ finally:
+ demo.cleanup()
+
+
+if __name__ == "__main__":
+ sys.exit(main())
+ 
+
+*A web application simulation environment for OpenEnv that wraps the [OpenApps](https://github.com/facebookresearch/OpenApps) framework and BrowserGym.*
+
+
+
+