Release 0 5 1 (#1255)

* only load sumo road network once during scenario building

* map fixes, tests for generating od map, cleanup

* Add docstring test.

* Remove .sh files.

* Remove output folder.

* changes based on reviews

* changes based on reviews

* break into multiple lines

* reformatting

* cleanup

* use simple social agents for environment test

* bug fixes

* use simpler social agent

* use simpler social agent

* use simpler social agent

* Change message.

* bug fixes

* bug fixes

* bug fixes

* bug fix

* Map Creation Fixes (#1201)

* fix is_same_map for SumoRoadNetwork to handle net_file naming looseness

* oops

* updates from review

* minor: style

* GitHub Actions: Update requirements.txt

* fix od_newmarket map, cache width computations and extend map creation refactor to opendrive maps

* updated CHANGELOG.md

* reformatting

* changes based on reviews

* changes based on reviews

* Update CHANGELOG.md

* opendrive gym environment based on hiway environment

* Type annotation (#1208)

* GitHub Actions: Update requirements.txt

* add examples using opendrive gym environment

* reformatting

* type hint fixes (#1216)

* type hint fixes as detected by a pytype experiment

* change Set to Iterable to avoid being overly restrictive

* Delete opendrive_agent.py

* changes based on reviews

* revert minor change

* Add rendering check

* GitHub Actions: Update requirements.txt

* bug fix

* detect the version of scenario we are using instead and decide if to add the traffic to HiWayEnv.

* bug fix

* remove opendrive env

* refactoring

* refactoring

* refactoring

* reformatting and updated CHANGELOG.md

* updated CHANGELOG.md

* changes based on reviews

* reformatting

* GitHub Actions: Update requirements.txt

* Use bisect module to look up in a sorted array

* changes based on reviews

* changes based on reviews

* reformatting

* changes based on reviews

* pytype fixes

* only enable sumo traffic and social agents if all scenarios are SUMO

* remove spacing attribute in lanepoints.py

* Don't assume a file name more specific than searched for in cli/studio.py

* Don't assume a file name more specific than searched for in default_map_builder.py

* remove the odd interface of glb generation

* reformatting

* reformatting

* bug fix

* bug fix

* removed deprecated class and methods from agent.py (#1222)

* removed deprecated class and methods from agent.py

* changelog

* GitHub Actions: Update requirements.txt

* move human_in_the_loop.py into zoo

* Observation space modifier in FrameStack wrapper (#1121)

* Graceful handle of traci connection errors

* Make format

* Fix check for empty route

* Fix bugs with provider changes

* Add required property to provider

* Add provider error handling

* Add `__sim__` done

* Format

* Update changelog

* Fix missing changes

* Add unsaved file

* Push another unsaved file.

* Document new methods

* Move recovery flag configuration to SMARTS

* Apply suggestions

* Fix provider not inheriting from Provider

* Revert unimportant changes

* Note make test test_notebook timeout in CHANGELOG

* Make sure notebook tests do not time out

* Update changelog

* Remove unnecessary EmptyProvider

* Improve provider error handling

* Ensure that error handling is working

* Fix issue that causes crash with TrapManager

* Fix `SumoTrafficSimulation.recover` definition

* Add missing import

* Handle `SMARTS.reset(..)` errors

* Set default recover to re-raise exception

* Address comments.

* Misc. backfill (#1205)

* remove uses of open_agent_actor in some scenarios; limit num_cpus used in some tests

* gain updates to trajectory tracking controller

* Fix early end due to being off_road

* Ignore sumo map errors

* revert case-related changes in sstudio/type

* updates from review

* made it clearer that the error gains are not coming from the model config file

* cleanup based on review

* Driving in traffic example (#1141)

* GitHub Actions: Update requirements.txt

* Fix SMARTS.__del__() throwing at program exit

* Make  required for SMARTS

* Add doc to explain use of SMARTS

* Update changelog

* Update error demonstration

* Fix `Point.__del__`  error.

* Make SMARTS.__del__ idempotent

* GitHub Actions: Update requirements.txt

* Remove route_waypoints from RoadWaypoints (#1229)

* Smooth headings (#1219)

* type hint fixes (from pytype)

* add max_angular_velocity option

* added options to smooth headings with sliding window and ignore heading changes when below minumum speed

* comments + CHANGELOG

* updates from review

* updates from review

* minor typo fix

* changes based on reviews

* changes based on reviews

* changes based on reviews

* reformatting

* minor changes

* minor change

* GitHub Actions: Update requirements.txt

* move human-in-the-loop-v0 into zoo

* reformat human_in_the_loop.py

* expose vehicle Starts for use by imitation learning (#1227)

* refactor to expose get_vehicle_start_at_time()

* added docstring and CHANGELOG entry + fixed minor bug

* updates from review

* CHANGELOG

Co-authored-by: Rutvik Gupta <[email protected]>

* GitHub Actions: Update requirements.txt

* GitHub Actions: Update mac_requirements.txt

* remove unused imports and reformat

* GitHub Actions: Update requirements.txt

* Fix secondary traci connection error

* Update changelog

* Fix typo

* add black and isort to precommit hook

* add pylint to hooks

* format instead of only checking

* move pre-commit and pylint dependencies into [dev] option

* add instructions for setup

* update CONTRIBUTING.md

* update CONTRIBUTING.d

* update black in pre-commit

* use local black and isort in pre-commit

* remove comments

* update contributing, only format python files

* update contributing

* remove conflicts

* remove conflicts

* map building fixes (#1234)

* default map builder_fn will use map.net.xml first if it exists

* allow for hashing a non-file based map

* CHANGELOG

* typo

* typo and whitespace

* clean up sstudio for map building
- don't assume map filenames
- correctly handle shift_to_origin

* updates from review

* added/fixed type hints based on pytype

* First batch of docstrings

* Second batch of docstrings

* Third batch of docstrings

* Fourth batch of docstrings

* Fifth batch of docstrings

* remove python3.7-dev

* Sixth batch of docstrings

* add extra intro for zsh user

* Fix error

* Update README.md

* remove python3.7-dev from all setup-related files

* Seventh batch of docstrings

* GitHub Actions: Update mac_requirements.txt

* Eigth batch of docstrings

* Tenth batch of docstrings

* Eleventh batch of docstrings

* 12th batch of docstrings

* 13th batch of docstrings

* Package smarts_ros ROS node (#1243)

* move smarts_ros ROS node into smarts/ros so it can be distributed with pkg

* moved ros logging stuff to smarts.ros.logging
formatting

* set env for catkin_make

* fix typo

* copyright headers

* changelog

* bugfix: remove incorrect assert w.r.t. map_spec

* comment

* add error message to __main__.py in smarts.ros

* allow for install_path to be set for the smarts.ros node

* formatting

* updated README to ref the setup_node script

* comment change per review

* GitHub Actions: Update requirements.txt

* 14th batch of docstrings

* 15th batch of docstrings

* Fix imports

* 16th batch of docstrings

* Fix interface issues

* GitHub Actions: Update requirements.txt

* use process instead of thread to speed up the build-all option

* reformat and correct typo

* add the change to CHANGELOG

* GitHub Actions: Update requirements.txt

* GitHub Actions: Format - Black and isort.

* GitHub Actions: Update requirements.txt

* GitHub Actions: Update requirements.txt

* Auto format (#1)

* GitHub Actions: Update requirements.txt

* GitHub Actions: Format - Black and isort.

* exclude manager_pb2_grpc.py,worker_pb2_grpc.py

* Fix steering observation description

* Auto format and commit (#1252)

* GitHub Actions: Update requirements.txt

* GitHub Actions: Format

* Update studio.py

* Update studio.py

* Apply suggestions from code review

Co-authored-by: adai <[email protected]>
Co-authored-by: Saul Field <[email protected]>

* Remove duplicate packages.

* GitHub Actions: Update requirements.txt

* GitHub Actions: Format

* GitHub Actions: Update mac_requirements.txt

* GitHub Actions: Update mac_requirements.txt

* Apply suggestions from code review

Co-authored-by: adai <[email protected]>
Co-authored-by: Saul Field <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saul Field <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saul Field <[email protected]>

* Spelling check

* Fix import error

* Apply suggestions from code review

Co-authored-by: adai <[email protected]>

* Update smarts/core/smarts.py

* Apply suggestions from code review

Co-authored-by: adai <[email protected]>

* Add ros node docstrings

* Pass doctest

* Update smarts/core/utils/math.py

Co-authored-by: adai <[email protected]>

* Apply suggestions from code review

Co-authored-by: adai <[email protected]>

* Update smarts/env/custom_observations.py

Co-authored-by: adai <[email protected]>

* GitHub Actions: Format

* GitHub Actions: Update requirements.txt

* GitHub Actions: Format

* Driving in traffic - dependency version and type annotation (#1250)

* GitHub Actions: Update requirements.txt

* Update version of SMARTS

Co-authored-by: Rutvik Gupta <[email protected]>
Co-authored-by: adaickalavan <[email protected]>
Co-authored-by: Qianyi Sun <[email protected]>
Co-authored-by: Rutvik Gupta <[email protected]>
Co-authored-by: Steve <[email protected]>
Co-authored-by: Steve <[email protected]>
Co-authored-by: Qianyi Sun <[email protected]>
Co-authored-by: Qianyi Sun <[email protected]>
Co-authored-by: qianyi-sun <[email protected]>
Co-authored-by: Stephen Hockema <[email protected]>
Co-authored-by: Allan Liu <[email protected]>
Co-authored-by: Saul Field <[email protected]>

.github/workflows/ci-auto-commit-linux.yml

@@ -0,0 +1,61 @@
+name: SMARTS CI Auto Commit Linux
+
+on:
+  push:
+    branches:
+      - develop
+  pull_request:
+    branches:
+      - develop
+
+env:
+  venv_dir: .venv
+
+jobs:
+  auto-commit-linux:
+    runs-on: ubuntu-18.04
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    container: huaweinoah/smarts:v0.4.18-minimal
+    steps:
+      - name: Install packages
+        run: |
+          add-apt-repository -y ppa:git-core/ppa
+          apt-get update
+          apt-get install -y git curl ca-certificates
+          curl -sL https://deb.nodesource.com/setup_14.x | bash -
+          apt-get install -y nodejs
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          ref: develop 
+      - name: Install SMARTS
+        run: |
+          python3.7 -m venv ${{env.venv_dir}}
+          . ${{env.venv_dir}}/bin/activate
+          pip install --upgrade pip wheel
+          pip install .[camera-obs,test,train]
+      - name: Update requirements
+        run: |
+          . ${{env.venv_dir}}/bin/activate
+          pip freeze | grep -v 'smarts' | grep -v 'pkg-resources==0.0.0' > requirements.txt
+      - name: Commit requirement changes
+        uses: EndBug/add-and-commit@v7
+        with:
+          add: 'requirements.txt'
+          branch: develop
+          default_author: user_info 
+          message: 'GitHub Actions: Update requirements.txt'
+      - name: isort, Black, and prettier
+        run: |
+          . ${{env.venv_dir}}/bin/activate
+          pip install .[dev]
+          isort -m VERTICAL_HANGING_INDENT --skip-gitignore --ac --tc --profile black ./baselines/marl_benchmark/ ./cli ./envision ./examples/ ./utils/ ./scenarios/ ./smarts ./ultra ./zoo
+          black .
+          npx prettier --write ./envision/web/src
+      - name: Commit format changes
+        uses: EndBug/add-and-commit@v7
+        with:
+          add: '["*.py", "*.html", "*.js"]'
+          branch: develop
+          default_author: user_info 
+          message: 'GitHub Actions: Format'

.github/workflows/ci-dependencies-mac.yml → .github/workflows/ci-auto-commit-mac.yml

@@ -1,4 +1,4 @@
-name: SMARTS CI Dependencies Mac
+name: SMARTS CI Auto Commit Mac
 
 on:
   schedule:
@@ -10,7 +10,7 @@ env:
   venv_dir: .venv
 
 jobs:
-  update-requirements-mac:
+  auto-commit-mac:
     runs-on: macos-11
     steps:
       - name: Checkout

.github/workflows/ci-base-tests-mac.yml

@@ -2,7 +2,7 @@ name: SMARTS CI Base Tests Mac
 
 on:
   workflow_run:
-    workflows: ["SMARTS CI Dependencies Mac"]
+    workflows: ["SMARTS CI Auto Commit Mac"]
     types:
       - completed
   workflow_dispatch:

.github/workflows/ci-format.yml

@@ -0,0 +1,39 @@
+name: SMARTS CI Format
+
+on: [push, pull_request]
+
+env:
+  venv_dir: .venv
+
+jobs:
+  test-header:
+    runs-on: ubuntu-18.04
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    container: huaweinoah/smarts:v0.4.18-minimal
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Check header
+        run: |
+          cd $GITHUB_WORKSPACE
+          make header-test
+
+  test-docstring:
+    runs-on: ubuntu-18.04
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    container: huaweinoah/smarts:v0.4.18-minimal
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Check docstring
+        run: |
+          cd $GITHUB_WORKSPACE
+          pip install --upgrade pip
+          pip install pylint
+          pylint -d all \
+            -e missing-function-docstring \
+            -e missing-class-docstring \
+            -s n \
+            --ignore marl_benchmark,examples,scenarios,docs,manager_pb2_grpc.py,worker_pb2_grpc.py \
+            --msg-template='{path}: line {line}: {msg_id}: {msg}' \
+            ./smarts ./envision

.pre-commit-config.yaml

@@ -0,0 +1,15 @@
+repos:
+-   repo: local
+    hooks:
+    -   id: pylint
+        name: pylint
+        entry: pylint
+        language: system
+        types: [python]
+        args:
+            [
+            "--errors-only",
+            "-rn", # Only display messages
+            "-sn", # Don't display the score
+            ]
+

CHANGELOG.md

@@ -1,10 +1,10 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
-This changelog is to adhere to the format given at [keepachangelog](keepachangelog.com/en/1.0.0/) 
+This changelog is to adhere to the format given at [keepachangelog](keepachangelog.com/en/1.0.0/)
 and should maintain [semantic versioning](semver.org).
 
-All text added must be human-readable. 
+All text added must be human-readable.
 
 Copy and pasting the git commit messages is __NOT__ enough.
 
@@ -16,6 +16,19 @@ Copy and pasting the git commit messages is __NOT__ enough.
 ### Removed
 ### Security
 
+# [0.5.1] 2022-01-25
+### Added
+- Added `get_vehicle_start_time()` method for scenarios with traffic history data.  See Issue #1210.
+### Changed
+- If more than one qualifying map file exists in a the `map_spec.source` folder, `get_road_map()` in `default_map_builder.py` will prefer to return the default files (`map.net.xml` or `map.xodr`) if they exist.
+- Moved the `smarts_ros` ROS node from the `examples` area into the `smarts.ros` module so that it can be distributed with SMARTS packages.
+- Use `Process` to replace `Thread` to speed up the `scl scenario build-all --clean <scenario_dir>` runtime.
+### Deprecated
+### Fixed
+- Fixed a secondary exception that the `SumoTrafficSimulation` will throw when attempting to close a TraCI connection that is closed by an error.
+### Removed
+### Security
+
 ## [0.5.0] - 2022-01-07
 ### Added
 - Added Minimum FrameRate tests to measure the fps for `smart.step()` method. See Issue #455.
@@ -36,7 +49,7 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Added `smarts.core.utils.import_utils` to help with the dynamic import of modules.
 - Added `single_agent` env wrapper and unit test. The wrapper converts a single-agent SMARTS environment's step and reset output to be compliant with gym spaces.
 - Added `rgb_image` env wrapper and unit test. The wrapper filters SMARTS environment observation and returns only top-down RGB image as observation.
-- Extended the `RoadMap` API to support `OpenDRIVE` map format in `smarts/core/opendrive_road_network.py`. Added 3 new scenarios with `OpenDRIVE` maps. See PR #1186.  
+- Extended the `RoadMap` API to support `OpenDRIVE` map format in `smarts/core/opendrive_road_network.py`. Added 3 new scenarios with `OpenDRIVE` maps. See PR #1186.
 - Added a "ReplayAgent" wrapper to allow users to rerun an agent previously run by saving its configurations and inputs. See Issue #971.
 - Added `smarts.core.provider.ProviderRecoveryFlags` as flags to determine how `SMARTS` should handle failures in providers. They are as follows:
   - `NOT_REQUIRED`: Not needed for the current step. Error causes skip of provider if it should recover but cannot or should not recover.
@@ -49,12 +62,12 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Added recovery options to `smarts.core.smarts.SMARTS.add_provider()`
   - Add `recovery_flags` argument to configure the recovery options if the provider disconnects or throws an exception.
 - Added `driving_in_traffic` reinforcement learning example. An ego agent is trained using DreamerV2 to drive as far and as fast as possible in heavy traffic, without colliding or going off-road.
-- Added `smarts.core.smarts.SMARTSDestroyedError` which describes use of a destroyed `SMARTS` instance. 
+- Added `smarts.core.smarts.SMARTSDestroyedError` which describes use of a destroyed `SMARTS` instance.
 ### Changed
 - `test-requirements` github action job renamed to `check-requirements-change` and only checks for requirements changes without failing.
 - Moved examples tests to `examples` and used relative imports to fix a module collision with `aiohttp`'s `examples` module.
 - Made changes to log sections of the scenario step in `smarts.py` to help evaluate smarts performance problems. See Issue #661.
-- Introducted `RoadMap` class to abstract away from `SumoRoadNetwork` 
+- Introducted `RoadMap` class to abstract away from `SumoRoadNetwork`
   and allow for (eventually) supporting other map formats.  See Issue #830 and PR #1048.
   This had multiple cascading ripple effects (especially on Waypoint generation and caching,
   Missions/Plans/Routes and road/lane-related sensors).  These include:
@@ -68,7 +81,7 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Public `SMARTS` methods will throw `smarts.core.smarts.SMARTSDestroyedError` if `SMARTS.destroy()` has previously been called on the `SMARTS` instance.
 ### Fixed
 - Fix lane vector for the unique cases of lane offset >= lane's length. See PR #1173.
-- Logic fixes to the `_snap_internal_holes` and `_snap_external_holes` methods in `smarts.core.sumo_road_network.py` for crude geometry holes of sumo road map. Re-adjusted the entry position of vehicles in `smarts.sstudio.genhistories.py` to avoid false positive events. See PR #992. 
+- Logic fixes to the `_snap_internal_holes` and `_snap_external_holes` methods in `smarts.core.sumo_road_network.py` for crude geometry holes of sumo road map. Re-adjusted the entry position of vehicles in `smarts.sstudio.genhistories.py` to avoid false positive events. See PR #992.
 - Prevent `test_notebook.ipynb` cells from timing out by increasing time to unlimited using `/metadata/execution/timeout=65536` within the notebook for regular uses, and `pytest` call with `--nb-exec-timeout 65536` option for tests. See for more details: "https://jupyterbook.org/content/execute.html#setting-execution-timeout" and "https://pytest-notebook.readthedocs.io/en/latest/user_guide/tutorial_intro.html#pytest-fixture".
 - Stop `multiprocessing.queues.Queue` from throwing an error by importing `multiprocessing.queues` in `envision/utils/multiprocessing_queue.py`.
 - Prevent vehicle insertion on top of ignored social vehicles when the `TrapManager` defaults to emitting a vehicle for the ego to control. See PR #1043
@@ -78,7 +91,7 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Updated deprecated Shapely functionality.
 - Fixed the type of `position` (pose) fields emitted to envision to match the existing type hints of `tuple`.
 - Properly detect whether waypoint is present in mission route, while computing distance travelled by agents with missions in TripMeterSensor.
-- Fixed `test_notebook` timeout by setting `pytest --nb-exec-timeout 65536`. 
+- Fixed `test_notebook` timeout by setting `pytest --nb-exec-timeout 65536`.
 ### Deprecated
 - The `timestep_sec` property of SMARTS is being deprecated in favor of `fixed_timesep_sec`
   for clarity since we are adding the ability to have variable time steps.
@@ -89,7 +102,7 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Removed `route_waypoints` attribute from `smarts.core.sensors.RoadWaypoints`.
 
 ## [0.4.18] - 2021-07-22
-### Added 
+### Added
 - Dockerfile for headless machines.
 - Singularity definition file and instructions to build/run singularity containers.
 - Support multiple outgoing edges from SUMO maps.
@@ -102,16 +115,16 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Fix case where multiple outgoing edges could cause non-determinism.
 
 ## [0.4.17] - 2021-07-02
-### Added 
+### Added
 - Added `ActionSpace.Imitation` and a controller to support it.  See Issue #844.
 - Added a `TraverseGoal` goal for imitation learning agents.  See Issue #848.
-- Added `README_pypi.md` to update to the general user installation PyPI instructions. See Issue #828. 
+- Added `README_pypi.md` to update to the general user installation PyPI instructions. See Issue #828.
 - Added a new utility experiment file `cli/run.py` to replace the context given by `supervisord.conf`. See PR #911.
 - Added `scl zoo install` command to install zoo policy agents at the specified paths. See Issue #603.
 - Added a `FrameStack` wrapper which returns stacked observations for each agent.
 ### Changed
 - `history_vehicles_replacement_for_imitation_learning.py` now uses new Imitation action space. See Issue #844.
-- Updated and removed some package versions to ensure that Python3.8 is supported by SMARTS. See issue #266. 
+- Updated and removed some package versions to ensure that Python3.8 is supported by SMARTS. See issue #266.
 - Refactored `Waypoints` into `LanePoints` (static, map-based) and `Waypoints` (dynamic). See Issue #829.
 - Vehicles with a `BoxChassis` can now use an `AccelerometerSensor` too.
 - When importing NGSIM history data, vehicle speeds are recomputed.
@@ -139,7 +152,7 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Removed `supervisord.conf` and `supervisor` from dependencies and requirements. See Issue #802.
 
 ## [0.4.16] - 2021-05-11
-### Added 
+### Added
 - Added `sanity-test` script and asked new users to run `sanity-test` instead of `make test` to ease the setup
 process
 - Added `on_shoulder` as part of events in observation returned from each step of simulation
@@ -167,9 +180,9 @@ process
 ### Added
 - This CHANGELOG as a change log to help keep track of changes in the SMARTS project that can get easily lost.
 - Hosted Documentation on `readthedocs` and pointed to the smarts paper and useful parts of the documentation in the README.
-- Running imitation learning will now create a cached `history_mission.pkl` file in scenario folder that stores 
+- Running imitation learning will now create a cached `history_mission.pkl` file in scenario folder that stores
 the missions for all agents.
-- Added ijson as a dependency. 
+- Added ijson as a dependency.
 - Added `cached-property` as a dependency.
 ### Changed
 - Lowered CPU cost of waypoint generation. This will result in a small increase in memory usage.
@@ -186,9 +199,9 @@ the missions for all agents.
 - Fix envision error 15 by cleanly shutting down the envision worker process.
 
 ## [Format] - 2021-03-12
-### Added 
+### Added
 – Describe any new features that have been added since the last version was released.
-### Changed 
+### Changed
 – Note any changes to the software’s existing functionality.
 ### Deprecated
 – Note any features that were once stable but are no longer and have thus been removed.