feat: implement scene update logic based on predicted action type

README.md

@@ -17,6 +17,12 @@ RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent C
 </a>&nbsp&nbsp🤖 <a href="https://github.com/FlagOpen/RoboBrain/">RoboBrain 1.0</a>: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete.
 </p>
 
+💬 If you have any questions, feel free to contact us via WeChat.
+<p align="center">
+  <img src="./assets/wechat.png" width="300" alt="WeChat QR Code">
+</p>
+
+
 ## 🔥 Overview
 The rise of embodied intelligence has intensified the need for robust multi-agent collaboration in industrial automation, service robotics, and smart manufacturing. However, current robotic systems struggle with critical limitations, including poor cross-embodiment adaptability, inefficient task scheduling, and inadequate dynamic error correction. While end-to-end vision-language-action (VLA) models (e.g., OpenVLA, RDT, Pi-0) exhibit weak long-horizon planning and task generalization, hierarchical VLA models (e.g., Helix, Gemini-Robotics, GR00T-N1) lack cross-embodiment compatibility and multi-agent coordination capabilities.
 To address these challenges, we present **RoboOS**, the first open-source embodied operating system based on a *Brain-Cerebellum* hierarchical architecture, facilitating a paradigm shift from single-agent to swarm intelligence. Specifically, RoboOS comprises three key components: **(1) the Embodied Cloud Model**, a multimodal large language model (MLLM) for global perception and high-level decision-making;  **(2) the Cerebellum Skill Library**, a modular, plug-and-play toolkit for seamless multi-skill execution; and  **(3) Real-Time Shared Memory**, a spatiotemporal synchronization mechanism for multi-agent state coordination. By integrating hierarchical information flow, RoboOS bridges the Embodied Brain and Cerebellum Skill Library, enabling robust planning, scheduling, and error correction for long-horizon tasks while ensuring efficient multi-agent collaboration by Real-Time Shared Memory. Moreover, we optimize edge-cloud communication and cloud-based distributed inference to support high-frequency interactions and scalable deployment.
@@ -45,10 +51,8 @@ Extensive real-world experiments across diverse scenarios (e.g., restaurant, hou
 - pip package manager
 
 ### 🎥 Deployment Video Tutorial
-<video width="100%" controls>
-  <source src="https://flagopen.github.io/RoboOS/assets/Deployment.mp4" type="video/mp4">
-  Your browser does not support the video tag.
-</video>
+> 👉 [Click here to watch the deployment tutorial](https://flagopen.github.io/RoboOS/assets/Deployment.mp4) *(Recommended before getting started)*
+
 
 ### 🚀 2. Deployment Methods
 Two deployment options are available based on your needs:
@@ -70,7 +74,7 @@ docker run -itd \
     --shm-size=500g \
     --name agent \
     --hostname flagscale-agent \
-    -v {your_local_path}/BAAI/RoboBrain2.0-7B:/path/in/container \
+    -v {your_local_path}/BAAI/RoboBrain2.0-7B:/workspace/RoboBrain2.0-7B \
     --network=host \
     -p 8888:8888 \
     -w /workspace/RoboOS \
@@ -102,7 +106,7 @@ pip install -r requirements.txt
 
 git clone https://github.com/FlagOpen/FlagScale 
 cd FlagScale
-git checkout a0687db035ba1d9c7b2661d8142ee4e8348b1459
+git checkout 3fc2037f90917227bd4aebabd9d7b330523f437c
 
 # Install in editable mode with PYTHONPATH
 PYTHONPATH=./:$PYTHONPATH pip install . --verbose --no-build-isolation 
@@ -148,6 +152,116 @@ python skill.py
 Visit the web UI at http://127.0.0.1:8888 and follow the on-screen instructions to complete configuration.
 Once finished, you can control the robot and trigger skills from the interface.
 
+
+##### ⚡️ 5. Start vLLM Model Service
+
+RoboOS requires a large language model backend to handle reasoning and tool calls.  
+We recommend using **vLLM** to serve the [RoboBrain2.0-7B](https://www.modelscope.cn/models/BAAI/RoboBrain2.0-7B/summary) model.
+
+
+#### 5.1 Install vLLM
+
+```bash
+pip install vllm
+```
+
+#### 5.2 Prepare Chat Template
+The tool_chat_template_hermes.jinja file must be provided for tool-call parsing.
+Place it in the following directory:
+
+```arduino
+RoboOS/deploy/templates/tool_chat_template_hermes.jinja
+```
+#### 5.3 Launch vLLM
+Run the following command to start the model service:
+
+```bash
+vllm serve RoboBrain2.0-7B \
+    --gpu-memory-utilization=0.9 \
+    --max-model-len=10000 \
+    --max-num-seqs=256 \
+    --port=4567 \
+    --trust-remote-code \
+    --enable-chunked-prefill \
+    --enable-auto-tool-choice \
+    --tool-call-parser hermes \
+    --chat-template RoboOS/deploy/templates/tool_chat_template_hermes.jinja 
+```
+
+### ⚙️ 6. Master & Slaver Configuration
+Before running the system, you need to configure both the **master** and **slaver** agents.  
+Each agent requires a `config.yaml` file to define model connection, audio, and logging settings.
+
+#### 6.1 Configuration Files
+- `master/config.yaml`
+- `slaver/config.yaml`
+
+A default template is provided below (you may adjust according to your environment):
+
+```yaml
+
+
+# Cloud Server (vLLM) Model Parameters
+model:
+  model_select: "/workspace/model/BAAI/RoboBrain2.0-7B"
+  model_retry_planning: 5
+  model_dict:
+    cloud_model: "/workspace/model/BAAI/RoboBrain2.0-7B"
+    cloud_type: "default"
+    cloud_api_key: "EMPTY"
+    cloud_server: "http://localhost:4567/v1/"
+    max_chat_message: 50
+
+# Redis Collaborator
+collaborator:
+  host: "127.0.0.1"
+  port: 6379
+  db: 0
+  clear: true
+  password: ""
+
+# Slaver Robot 
+robot:
+  # "local" with a fold name such as "demo_robot"
+  # "remote" with URL such as "http://127.0.0.1:8000", and run the Python script 'skill.py' on the robot itself.
+  # call_type: local
+  # path: "demo_robot_local"
+  name: demo_robot
+  call_type: remote
+  path: "http://127.0.0.1:8000"
+
+# Master Scene profile
+profile:
+  path: ./scene/profile.yaml
+
+# Slaver
+tool:
+  # Has the model undergone targeted training on tool_calls
+  support_tool_calls: false
+
+```
+
+
+#### 6.2 Key Parameters
+
++ model.cloud_server:
+Must point to your vLLM service (default: http://localhost:4567/v1/)
+
++ collaborator:
+Redis server configuration (default: 127.0.0.1:6379)
+
++ profile：
+Path to the scene profile YAML file that defines environment and task settings (e.g., ./scene/profile.yaml)
+
++ tool：
+Enable or disable tool-call support. Set `support_tool_calls: true` if your model has been trained for tool calls
++ robot:
+Two modes of calling robot tools
+
+
+⚠️ Make sure these fields are correctly configured; otherwise, RoboOS may fail to connect to vLLM, Redis, or load scene/tool profiles.
+
+
 ## 🔧 Manual Deployment (Advanced)
 If you prefer to manually run RoboOS without using the deployment web UI, follow the steps below to start the system components directly from source.
 

deploy/templates/tool_chat_template_hermes.jinja

@@ -0,0 +1,130 @@
+{%- macro json_to_python_type(json_spec) %}
+    {%- set basic_type_map = {
+    "string": "str",
+    "number": "float",
+    "integer": "int",
+    "boolean": "bool"
+} %}
+
+    {%- if basic_type_map[json_spec.type] is defined %}
+        {{- basic_type_map[json_spec.type] }}
+    {%- elif json_spec.type == "array" %}
+        {{- "list[" +  json_to_python_type(json_spec|items) + "]" }}
+    {%- elif json_spec.type == "object" %}
+        {%- if json_spec.additionalProperties is defined %}
+            {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }}
+        {%- else %}
+            {{- "dict" }}
+        {%- endif %}
+    {%- elif json_spec.type is iterable %}
+        {{- "Union[" }}
+        {%- for t in json_spec.type %}
+            {{- json_to_python_type({"type": t}) }}
+            {%- if not loop.last %}
+                {{- "," }}
+            {%- endif %}
+        {%- endfor %}
+        {{- "]" }}
+    {%- else %}
+        {{- "Any" }}
+    {%- endif %}
+{%- endmacro %}
+
+
+{{- bos_token }}
+{{- "<|im_start|>system\nYou are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> " }}
+{%- if tools is iterable and tools | length > 0 %}
+    {%- for tool in tools %}
+        {%- if tool.function is defined %}
+            {%- set tool = tool.function %}
+        {%- endif %}
+        {{- '{"type": "function", "function": ' }}
+        {{- '{"name": "' + tool.name + '", ' }}
+        {{- '"description": "' + tool.name + '(' }}
+        {%- for param_name, param_fields in tool.parameters.properties|items %}
+            {{- param_name + ": " + json_to_python_type(param_fields) }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- endif %}
+        {%- endfor %}
+        {{- ")" }}
+        {%- if tool.return is defined %}
+            {{- " -> " + json_to_python_type(tool.return) }}
+        {%- endif %}
+        {{- " - " + tool.description + "\n\n" }}
+        {%- for param_name, param_fields in tool.parameters.properties|items %}
+            {%- if loop.first %}
+                {{- "    Args:\n" }}
+            {%- endif %}
+            {{- "        " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
+        {%- endfor %}
+        {%- if tool.return is defined and tool.return.description is defined %}
+            {{- "\n    Returns:\n        " + tool.return.description }}
+        {%- endif %}
+        {{- '"' }}
+        {{- ', "parameters": ' }}
+        {%- if tool.parameters.properties | length == 0 %}
+            {{- "{}" }}
+        {%- else %}
+            {{- tool.parameters|tojson }}
+        {%- endif %}
+        {{- "}" }}
+        {%- if not loop.last %}
+            {{- "\n" }}
+        {%- endif %}
+    {%- endfor %}
+{%- endif %}
+{{- " </tools>" }}
+{{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
+' }}
+{{- "For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+" }}
+{{- "<tool_call>
+" }}
+{{- '{"name": <function-name>, "arguments": <args-dict>}
+' }}
+{{- '</tool_call><|im_end|>' }}
+{%- for message in messages %}
+    {%- if message.role == "user" or message.role == "system" or (message.role == "assistant" and message.tool_calls is not defined) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" and message.tool_calls is defined %}
+        {{- '<|im_start|>' + message.role }}
+        {%- for tool_call in message.tool_calls %}
+            {{- '\n<tool_call>\n' }}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '{' }}
+            {{- '"name": "' }}
+            {{- tool_call.name }}
+            {{- '"' }}
+            {%- if tool_call.arguments is defined %}
+                {{- ', ' }}
+                {{- '"arguments": ' }}
+                {{- tool_call.arguments|tojson }}
+            {%- endif %}
+            {{- '}' }}
+            {{- '\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>tool\n' }}
+        {%- endif %}
+        {{- '<tool_response>\n' }}
+        {{- message.content }}
+        {%- if not loop.last %}
+            {{- '\n</tool_response>\n' }}
+        {%- else %}
+            {{- '\n</tool_response>' }}
+        {%- endif %}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

master/agents/agent.py

@@ -22,6 +22,8 @@ def __init__(self, config_path="config.yaml"):
 
         self.logger.info(f"Configuration loaded from {config_path} ...")
         self.logger.info(f"Master Configuration:\n{self.config}")
+
+        self._init_scene(self.config["profile"])
         self._start_listener()
 
     def _init_logger(self, logger_config):
@@ -54,9 +56,26 @@ def _init_config(self, config_path="config.yaml"):
         with open(config_path, "r", encoding="utf-8") as f:
             self.config = yaml.safe_load(f)
 
+    def _init_scene(self, scene_config):
+        """Initialize scene object"""
+        path = scene_config["path"]
+        if not os.path.exists(path):
+            self.logger.error(f"Scene config file {path} does not exist.")
+            raise FileNotFoundError(f"Scene config file {path} not found.")
+        with open(path, "r", encoding="utf-8") as f:
+            self.scene = yaml.safe_load(f)
+
+        scenes = self.scene.get("scene", [])
+        for scene_info in scenes:
+            scene_name = scene_info.pop("name", None)
+            if scene_name:
+                self.collaborator.record_environment(scene_name, json.dumps(scene_info))
+            else:
+                print("Warning: Missing 'name' in scene_info:", scene_info)
+
     def _handle_register(self, robot_name: Dict) -> None:
         """Listen for robot registrations."""
-        robot_info = self.collaborator.retrieve_agent(robot_name)
+        robot_info = self.collaborator.read_agent_info(robot_name)
         self.logger.info(
             f"AGENT_REGISTRATION: {robot_name} \n {json.dumps(robot_info)}"
         )
@@ -170,8 +189,8 @@ def reasoning_and_subtasks_is_right(self, reasoning_and_subtasks: dict) -> bool:
                 if isinstance(subtask, dict) and "robot_name" in subtask
             }
 
-            # Retrieve list of all registered robots from the collaborator
-            robots_list = set(self.collaborator.retrieve_all_agents_name())
+            # Read list of all registered robots from the collaborator
+            robots_list = set(self.collaborator.read_all_agents_name())
 
             # Check if all workers are registered
             return worker_list.issubset(robots_list)

master/agents/planner.py

@@ -67,11 +67,12 @@ def display_profiling_info(self, description: str, message: any):
     def forward(self, task: str) -> str:
         """Get the sub-tasks from the task."""
 
-        all_robots_name = self.collaborator.retrieve_all_agents_name()
-        all_robots_info = self.collaborator.retrieve_all_agents()
+        all_robots_name = self.collaborator.read_all_agents_name()
+        all_robots_info = self.collaborator.read_all_agents_info()
+        all_environments_info = self.collaborator.read_environment()
 
         content = MASTER_PLANNING_PLANNING.format(
-            robot_name_list=all_robots_name, robot_tools_info=all_robots_info, task=task
+            robot_name_list=all_robots_name, robot_tools_info=all_robots_info, task=task, scene_info=all_environments_info
         )
 
         messages = [