Integrate bug catching utilities.

[Gamenot]

Detail

In PR #619 we created a replay system to track down a bug caused in sumo. This has good utility for tracking down other bugs. It would be useful to re-evaluate these utilities.

The replay agent was partially integrated in #971

Context #619

It is not even certain that this is an issue any longer but it is necessary to confirm it to merge #1884.

• SMARTS is deterministic
  • Restore long-running determinism test
• Minimal context
  • Docker ubuntu:22.04
  • Add build dependencies
    • likely similar to the sumo 1.5 build utility https://github.com/huawei-noah/SMARTS/blob/master/utils/extras/sumo/ubuntu_compile/Dockerfile.ubuntu16.sumo1_5
    • https://sumo.dlr.de/docs/Installing/Linux_Build.html#building_the_sumo_binaries_with_cmake
  • Build the newest version of SUMO with debug options
    • https://sumo.dlr.de/docs/Installing/Linux_Build.html#building_the_sumo_binaries_with_cmake
  • Add smarts dependencies: python3.8 pip smarts
• Modernize crash reproduction
  • Create debug branch
    • Configure engine reset retries to 0
    • Disable handover to local traffic provider
    • Configure loop scenario with similar behavior to original test
  • Attempt to generate crash(es)
    • Terminal crash 1: ERROR:SMARTS:connection closed by SUMO with original context _usr_bin_sumo.1002.txt
    • Terminal crash 2: ERROR:root:TraCI has disconnected with: ... which is raised as traci.exceptions.FatalTraCIError('Route replacement failed for car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.0')
  • Host docker image

Originally posted by @Gamenot in #619 (comment)