Add NeurIPS2022 competition agent to agent zoo (#1838)

CHANGELOG.md

@@ -13,6 +13,8 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Engine configuration utility that uses the following locations to allow configuration of the SMARTS engine. The engine consumes the configuration files from the following locations in the following priority: `./engine.ini`, `~/.smarts/engine.ini`, `$GLOBAL_USER/smarts/engine.ini`, and `${PYTHON_ENV}/lib/${PYTHON_VERSION}/site-packages/smarts/engine.ini`.
 - Added map source uri as `map_source` inside of `hiway-v1` reset info to indicate what the current map is on reset.
 - Added NGSIM documentation.
+- Added a zoo agent, named Interaction-aware Motion Prediction, from NeurIPS2022 submission. This zoo agent runs in benchmark `driving_smarts==0.0`.
+- Added Agent Zoo documentation in ReadTheDocs.
 ### Changed
 - Made changes in the docs to reflect `master` branch as the main development branch.
 - Enabled supplying agent locator directly to benchmark runner and removed the need for an intermediary config file. Updated benchmark docs to reflect this.

docs/benchmarks/agent_zoo.rst

@@ -0,0 +1,56 @@
+.. _agent_zoo:
+
+Agent Zoo
+=========
+
+Numerous prebuilt and trained agents are hosted at `zoo/policies <https://github.com/huawei-noah/SMARTS/tree/master/zoo/policies>`_ . 
+These agents should be installed prior to being used in SMARTS environments or benchmarks.
+
+.. code-block:: bash
+
+    $ cd <path>/SMARTS
+    # Install a zoo agent.
+    $ scl zoo install <agent path>
+    # e.g., scl zoo install zoo/policies/interaction_aware_motion_prediction
+
+.. note::
+
+    To build a wheel, execute:
+
+    .. code-block:: bash
+
+        $ scl zoo build <agent path>
+        # e.g., scl zoo build zoo/policies/interaction_aware_motion_prediction
+
+Contribute agents
+-----------------
+
+.. note::
+    This section is only for contributing to the agent zoo.
+
+First, add the new agent to `zoo/policies <https://github.com/huawei-noah/SMARTS/tree/master/zoo/policies>`_. It should contain 
+
+* package setup file,
+* inference code with prebuilt model, and
+* algorithm explanation.
+
+Then, register the newly added zoo agent in `zoo/policies/__init__.py <https://github.com/huawei-noah/SMARTS/tree/master/zoo/policies/__init__.py>`_.
+
+Available zoo agents
+--------------------
+
+.. _available_zoo_agents:
+.. list-table::
+   :header-rows: 1
+
+   * - Agent locator and path
+     - Benchmark or Env
+     - Action space
+     - Source
+     - Remarks
+   * - | zoo.policies:interaction-aware-motion-prediction-agent-v0
+       | zoo/policies/interaction_aware_motion_prediction
+     - driving_smarts==0.0
+     - :attr:`~smarts.core.controllers.ActionSpaceType.TargetPose`
+     - `code <https://github.com/smarts-project/smarts-project.rl/tree/master/interaction_aware_motion_prediction>`__
+     - Contributed as part of `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`_ competition.

docs/benchmarks/benchmark.rst

@@ -3,6 +3,13 @@
 Instructions
 ============
 
+A benchmark is a standard set of rigorous environments which can be used to 
+assess and compare the performance of agents built by various researchers. 
+
+:doc:`Agent Zoo </benchmarks/agent_zoo>` contains prebuilt and trained agents 
+which could be deployed as reference ego agent in the benchmarks. Feel free to 
+mix and match compatible agents and benchmarks.
+
 Run a benchmark
 ---------------
 

docs/benchmarks/driving_smarts_2022.rst

@@ -4,7 +4,7 @@ Driving SMARTS 2022
 ===================
 
 The Driving SMARTS 2022 is a benchmark derived from the
-NeurIPS 2022 Driving SMARTS Competition.
+`NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`_ competition.
 
 This benchmark is intended to address the following requirements:
 
@@ -19,21 +19,17 @@ This benchmark is intended to address the following requirements:
 -  Added benchmarks should be versioned.
 -  Added benchmarks should be discover-able.
 
-See `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`_ page for historical context.
-
 This benchmark allows ego agents to use any one of the following action spaces.
 
 + :attr:`~smarts.core.controllers.ActionSpaceType.TargetPose`
 + :attr:`~smarts.core.controllers.ActionSpaceType.RelativeTargetPose`
 
-Compatible zoo agents
----------------------
-
-A list of SMARTS zoo agents which are compatible with this benchmark is
-provided here. A compatible zoo agent can be run as follows.
+See the list of :ref:`available zoo agents <available_zoo_agents>` which are compatible with this benchmark. A compatible zoo agent can be run as follows.
 
 .. code-block:: bash
 
     $ cd <path>/SMARTS
-    $ scl zoo install <path/to/agent policy>
+    $ scl zoo install <agent path>
+    # e.g., scl zoo install zoo/policies/interaction_aware_motion_prediction
     $ scl benchmark run driving_smarts==0.0 <agent_locator> --auto_install
+    # e.g., scl benchmark run driving_smarts==0.0 zoo.policies:interaction-aware-motion-prediction-agent-v0 --auto-install

docs/conf.py

@@ -66,6 +66,7 @@
     "setup.py",
     "scenarios",
     "smarts/ros",
+    "zoo/policies/interaction_aware_motion_prediction",
 ]
 apidoc_extra_args = [
     "--force",
@@ -89,6 +90,7 @@
     "ray",
     "tabulate",
     "tools",
+    "torch",
     "waymo_open_dataset",
 ]
 

docs/index.rst

@@ -54,6 +54,7 @@ If you use SMARTS in your research, please cite the `paper <https://arxiv.org/ab
    :caption: Benchmarks
 
    benchmarks/benchmark.rst
+   benchmarks/agent_zoo.rst
    benchmarks/driving_smarts_2022.rst
 
 .. toctree::

smarts/benchmark/entrypoints/benchmark_runner_v0.py

@@ -185,7 +185,7 @@ def format_scores_total(named_scores: List[Tuple[str, Score]], scenario_count):
     print(format_one_line_scores(named_scores))
     print()
     print("`Driving SMARTS V0` averaged result:")
-    print(format_scores_total(named_scores, len(env_args) or 1))
+    print(format_scores_total(named_scores, len(env_args)))
 
 
 def benchmark_from_configs(benchmark_config, agent_locator, debug_log=False):

zoo/README.md

@@ -1,42 +1,5 @@
 # Policy Zoo
 
-## Packaging Policies
-
-**Build policies** (and make available in zoo/policies)
-
-```bash
-# cd <project root>
-scl zoo build <policy>
-# e.g. scl zoo build zoo/policies/open-agent
-```
-
-**Installing policies** (and make available in zoo/policies)
-
-```bash
-# cd <project root>
-scl zoo install <policy>
-# e.g. scl zoo install zoo/policies/open-agent
-```
-
-**Use** (in scenarios)
-Under your scenario directory, create a requirements.txt
-```
-# we'll be serving packages from a local pip index
---extra-index-url http://localhost:8080
-open-agent==0.1.195
-```
-
-Then in your scenario.py:
-```python
-# */scenario.py (example)
-# ...
-t.SocialAgentActor(
-    name="my-open-agent",
-    agent_locator="open_agent:open_agent-v0"
-)
-# ...
-```
-
 ## Evaluating Policies
 
 ### Introduction

zoo/policies/__init__.py

@@ -1,4 +1,5 @@
-from typing import Any, Dict
+import importlib
+from pathlib import Path
 
 from smarts.core.agent_interface import AgentInterface, AgentType
 from smarts.core.controllers import ActionSpaceType
@@ -102,3 +103,33 @@ def human_keyboard_entrypoint(*arg, **kwargs):
 
 
 register(locator="human-in-the-loop-v0", entry_point=human_keyboard_entrypoint)
+
+
+def _verify_installation(pkg: str, module: str):
+    try:
+        lib = importlib.import_module(module, pkg)
+    except (ModuleNotFoundError, ImportError):
+        raise ModuleNotFoundError(
+            "Zoo agent is not installed. "
+            f"Install via `scl zoo install {str(Path(__file__).resolve().parent/pkg)}`."
+        )
+
+    return lib
+
+
+def entry_point_iamp(**kwargs):
+    pkg = "interaction_aware_motion_prediction"
+    module = ".policy"
+    lib = _verify_installation(pkg=pkg, module=module)
+
+    return AgentSpec(
+        interface=AgentInterface(
+            action=ActionSpaceType.TargetPose,
+        ),
+        agent_builder=lib.Policy,
+    )
+
+
+register(
+    locator="interaction-aware-motion-prediction-agent-v0", entry_point=entry_point_iamp
+)

zoo/policies/interaction_aware_motion_prediction/setup.cfg

@@ -0,0 +1,19 @@
+[metadata]
+name = smarts_agent_iamp_stub
+version = 0.1.0
+url = https://github.com/huawei-noah/SMARTS
+description = SMARTS zoo agent.
+long_description = Trained Interaction-aware Motion Prediction agent. See [SMARTS](https://github.com/huawei-noah/SMARTS).
+long_description_content_type=text/markdown
+classifiers=
+    Programming Language :: Python
+    Programming Language :: Python :: 3 :: Only
+    Programming Language :: Python :: 3.8
+
+[options]
+packages = find:
+include_package_data = True
+zip_safe = True
+python_requires = == 3.8.*
+install_requires = 
+    smarts_agent_iamp==0.1.0

zoo/policies/interaction_aware_motion_prediction/setup.py

@@ -0,0 +1,4 @@
+from setuptools import setup
+
+if __name__ == "__main__":
+    setup()