Long Running Determinism Test

[Gamenot]

What we need to do here is have a long running test that runs for a few days and does checks for determinism.

• Use the minicity example
• Runs for ~1-2 days

[Adaickalavan]

1. A pull request has been put up on this issue Long-duration determinism test #479 .
2. Expected runtime is about 5 hours, as the each job in a workflow can run for up to 6 hours before being terminated by Github Actions usage limit policy.
3. Currently the the long-duration determinism test runs the intersections/2lane scenario, since the minicity scenario tends to fail with the following error message:

root@kyber-MSI-power:/src# pytest --forked ./smarts/env/tests/test_determinism.py::test_long_determinism
=================================== test session starts ===================================
platform linux -- Python 3.7.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /src
plugins: notebook-0.6.1, cov-2.10.1, forked-1.3.0, xdist-2.1.0, benchmark-3.2.3
collected 1 item                                                                          

smarts/env/tests/test_determinism.py F                                              [100%]

======================================== FAILURES =========================================
__________________________________ test_long_determinism __________________________________
def test_long_determinism():
        max_steps_per_episode=90
        episode_count=1
        capture_step=30
        scenarios="scenarios/minicity"
>       determinism(agent_spec(max_steps_per_episode), scenarios, episode_count, capture_step)

smarts/env/tests/test_determinism.py:210: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
smarts/env/tests/test_determinism.py:196: in determinism
    run(agent_spec, check_callback, scenarios, episode_count, capture_step)
smarts/env/tests/test_determinism.py:81: in run
    callback(rewards, agent_obs, dones, int(i / capture_step))
smarts/env/tests/test_determinism.py:144: in check_callback
    vehicle_state_check(evs_now, evs_prev)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

vs_now = EgoVehicleObservation(id='Agent-007-47378190', position=array([ 289.90942679, 1411.20048604,    0.        ]), bounding...lar_velocity=array([0., 0., 0.]), linear_acceleration=0.0, angular_acceleration=0.0, linear_jerk=0.0, angular_jerk=0.0)
vs_prev = EgoVehicleObservation(id='Agent-007-5be6128e', position=array([ 770.91927216, 1497.43034843,    0.        ]), bounding...lar_velocity=array([0., 0., 0.]), linear_acceleration=0.0, angular_acceleration=0.0, linear_jerk=0.0, angular_jerk=0.0)

    def vehicle_state_check(vs_now, vs_prev):
>       assert all([now == prev for (now, prev) in zip(vs_now.position, vs_prev.position)])
E       assert False
E        +  where False = all([False, False, True])

smarts/env/tests/test_determinism.py:88: AssertionError
---------------------------------- Captured stdout call -----------------------------------
╭────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────╮
│            Episode │     Sim T / Wall T │        Total Steps │        Steps / Sec │       Scenario Map │    Scenario Routes │     Mission (Hash) │             Scores │
├────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┤
│                0/1 │               0.03 │                 89 │               2.64 │           minicity │      basic.rou.xml │ -67203396157717118 │  12.02 - Agent-007 │
╰────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────╯
╭────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────╮
│            Episode │     Sim T / Wall T │        Total Steps │        Steps / Sec │       Scenario Map │    Scenario Routes │     Mission (Hash) │             Scores │
├────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┤
╰────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────╯
================================= short test summary info =================================
FAILED smarts/env/tests/test_determinism.py::test_long_determinism
=================================== 1 failed in 43.86s ====================================

[Gamenot]

This is complete, now we need to debug the determinism issues with #537 .