Enhanced Eliza 3D Python Starter

Overview

This is an enhanced Python starter kit for building sophisticated Hyperfy agents with advanced physics simulation, voice capabilities, action systems, and AI integration. It provides a comprehensive framework that combines cutting-edge architectural patterns with lightweight implementation and working Hyperfy integration.

This starter kit enables developers to create sophisticated Hyperfy agents using Python with enterprise-grade features and modern architectural patterns.

Features

Advanced Agent Architecture

• Enhanced Physics Engine: Advanced physics integration with ForceMode.FORCE/IMPULSE, collision processing, sophisticated raycasting
• Event-Driven Agent Base: Multi-threaded 60 FPS update loops, comprehensive lifecycle management
• Sophisticated Action System: Priority queues, dependencies, concurrency management, specialized action implementations
• Mathematical Framework: Vector3D operations, spatial calculations, geometric operations, quaternion support
• Voice Capabilities: Speech recognition and text-to-speech for natural interactions
• World State Management: Advanced object, player, and environment state tracking

Integration Features

• Working Hyperfy Integration: Live agent connection to Hyperfy worlds (visible at https://hyperfy.io/chill)
• Lightweight Architecture: Minimal Node.js bridge with Python backend
• HTTP API Bridge: RESTful endpoints for health, chat, messages, and world state
• Minimal Dependencies: Streamlined dependency management for optimal performance

Architecture

The enhanced starter kit features a sophisticated multi-layer architecture:

Python Agent ◄──HTTP API──► Node.js Bridge ◄──WebSocket──► working-agent.mjs ◄──WebSocket──► Hyperfy

Components

• Core: Advanced agent base with event-driven architecture and lifecycle management
• Physics: Professional physics engine with collision and raycasting systems
• Actions: Complete action system with priority queues, dependencies, and specialized implementations
• Utils: Vector3D math and spatial calculation framework
• Voice: Voice recognition and synthesis capabilities
• Bridge: Minimal Node.js HTTP bridge for Hyperfy connectivity

Requirements

All dependencies are managed through requirements.txt:

# Core framework dependencies
pygame==2.6.1
numpy==1.26.4
pillowcase==2.0.0
python-dotenv==1.1.0
langchain==0.0.267
pymunk==7.0.0
SpeechRecognition==3.14.3
pyttsx3==2.98
requests==2.32.3
pydantic==2.11.5
PyAudio==0.2.14
json5==0.12.0

# Enhanced features
PyJWT==2.10.1
websockets==15.0.1
aiohttp==3.12.6
openai==1.82.1

# Testing
pytest==8.3.5
pytest-asyncio==1.0.0

# Plus extensive PyObjC framework support for macOS voice capabilities

Installation

1. Clone the repository:

   git clone <repository-url>
   cd eliza-3d-python-starter

2. Create and activate virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install system dependencies (macOS):

   brew install sdl2 portaudio

4. Install Python dependencies:

   pip install -r requirements.txt

5. Install Node.js dependencies (for Hyperfy bridge):

   npm install

6. Set up environment variables: Create a .env file in the project root:

   OPENAI_API_KEY=your-api-key-here
   HYPERFY_AUTH_TOKEN=your-hyperfy-token
   

Usage

Running the Enhanced Agent

Start the Python backend:

source venv/bin/activate
python nodejs_bridge.py

Start the Hyperfy agent (in another terminal):

node working-agent.mjs

The agent will connect to Hyperfy and be visible at https://hyperfy.io/chill

Testing Advanced Features

Run comprehensive feature tests:

source venv/bin/activate
python test_enhanced_features.py

This tests all advanced features:

• Math Utils (Vector3D operations, spatial calculations)
• Config Manager (hierarchical loading, validation)
• Action System (priority queues, dependencies)
• Physics Engine (advanced physics integration, collision processing)
• Agent Base (event-driven architecture, lifecycle)
• Voice Manager (speech synthesis, recognition)

Creating Your Own Agent

1. Create a new agent class inheriting from AgentBase:

from src.core.agent_base import AgentBase

class MyAgent(AgentBase):
    def __init__(self, config):
        super().__init__("MyAgent", config)
        # Custom initialization using advanced patterns
        
    def on_start(self):
        # Called when the agent starts
        self.say("Hello, world!")
        
    def on_voice_input(self, text, confidence):
        # Handle voice input using advanced voice system
        self.say(f"You said: {text}")
        
    def update(self, delta_time):
        # Called every frame at 60 FPS
        super().update(delta_time)
        # Custom update logic

2. Use the advanced action system:

from src.actions.movement_action import MovementAction
from src.utils.math_utils import Vector3D

# Queue movement using sophisticated action system
target = Vector3D(10.0, 0.0, 5.0)
action = MovementAction(target, speed=2.0)
self.action_system.queue_action(action, priority=1)

3. Leverage the physics engine:

# Use advanced physics for collision detection
hit_result = self.physics_engine.raycast(
    start_pos, direction, max_distance=10.0
)
if hit_result:
    print(f"Hit object at {hit_result.position}")

Core Components

Enhanced Agent Base

Sophisticated agent architecture featuring:

• Event-driven system via emit_event() and register_event_handler()
• Multi-threaded 60 FPS update loops
• Lifecycle management with start/stop and cleanup
• Advanced state management with position, rotation, velocity tracking

Advanced Physics Engine

Professional physics integration featuring:

• Collision processing with contact pairs and user data
• Advanced raycasting with numpy vector normalization
• 60 FPS physics simulation threading
• Agent registration supporting objects and IDs

Comprehensive Action System

Complete action framework:

• Action base class with ActionStatus enum and ActionResult dataclass
• ActionSystem with priority queues, dependencies, and concurrency
• Specialized implementations: MovementAction, WaitAction, SayAction

Mathematical Utilities

Vector3D and MathUtils classes:

• Comprehensive 3D math operations (dot/cross products, normalization)
• Spatial calculations and transformations
• Geometric operations and collision detection
• Quaternion conversions and interpolation

Examples

Enhanced Agent Implementation

The enhanced agent demonstrates sophisticated patterns:

• Event-driven responses using the advanced event system
• Physics integration with collision and raycasting
• Action queuing with priority and dependency system
• Mathematical operations using Vector3D and MathUtils
• Configuration management with hierarchical config system

Framework Features

This enhanced starter kit provides comprehensive functionality:

Core Framework

• Physics Engine: Complete physics integration with collision processing and raycasting
• Agent Architecture: Event-driven system with 60 FPS update loops and lifecycle management
• Action System: Priority queues, dependencies, concurrency, and specialized action implementations
• Mathematical Framework: Vector3D operations and comprehensive spatial calculations
• Configuration System: Hierarchical loading with validation and environment variables

Integration Features

• Hyperfy Integration: Working WebSocket connection to Hyperfy worlds
• Lightweight Bridge: Minimal Node.js bridge with optimal performance
• HTTP API: RESTful endpoints for external integration
• Live Testing: Agent visible and interactive at https://hyperfy.io/chill

Status

• ✅ Advanced Features: All major systems implemented and tested (6/6 tests passing)
• ✅ Hyperfy Integration: Working connection and visibility
• ✅ Lightweight Architecture: Minimal dependencies while maintaining functionality
• ✅ Comprehensive Testing: All features validated and working

Contributing

Contributions are welcome! This project maintains sophisticated architecture while providing practical Hyperfy integration.

License

MIT