[Help Request] Evaluating the trained RL model errors

[lurenlym]

High Level Description

I am having difficulty evaluating the trained RL model.

Version

Master branch 291417

Operating System

ubuntu 20.04

Problems

I followed driving_smarts_2023_3 and trained a rl model.

When I want to evaluate the model, output errors as followed:

<-- Starting `Driving SMARTS 2023.3` benchmark -->

This is the Driving SMARTS 2023.3 benchmark.

For context see:
    - https://smarts-project.github.io/competition/2023_driving_smarts/
    - https://codalab.lisn.upsaclay.fr/competitions/


Evaluating platoon-scenarios/sumo/platoon/merge_exit_sumo_t_agents_1...
Using configuration from: /home/sutpc/lym/SMARTS/smarts/engine.ini
:display:egldisplay(error): Failed to find or initialize a suitable EGL display connection.
Attempt to register type x11GraphicsPipe more than once!
Attempt to register type x11GraphicsWindow more than once!
Xlib:  extension "XFree86-DGA" missing on display "localhost:10.0".
Xlib:  extension "XFree86-DGA" missing on display "localhost:10.0".
Traceback (most recent call last):
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 84, in _eval_worker_local
    obs, reward, terminated, truncated, info = env.step(action)
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/gymnasium/core.py", line 361, in step
    return self.env.step(action)
  File "/home/sutpc/lym/SMARTS/smarts/env/gymnasium/wrappers/metric/metrics.py", line 130, in step
    new_costs = cost_func(
  File "/home/sutpc/lym/SMARTS/smarts/env/gymnasium/wrappers/metric/costs.py", line 411, in func
    assert min_len >= min_waypoints_length, (
AssertionError: Expected waypoints length >= 28, but got waypoints length = 25.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sutpc/miniconda3/envs/rl/bin/scl", line 8, in <module>
    sys.exit(scl())
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sutpc/miniconda3/envs/rl/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/sutpc/lym/SMARTS/cli/benchmark.py", line 79, in run
    run_benchmark(
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/__init__.py", line 109, in run_benchmark
    entrypoint(
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 221, in benchmark_from_configs
    benchmark(
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 174, in benchmark
    for name, records in iterator(
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 126, in _serial_task_iterator
    name, records = _eval_worker_local(
  File "/home/sutpc/lym/SMARTS/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 89, in _eval_worker_local
    records = env.records()
  File "/home/sutpc/lym/SMARTS/smarts/env/gymnasium/wrappers/metric/metrics.py", line 300, in records
    costs=op_dataclass(
  File "/home/sutpc/lym/SMARTS/smarts/env/gymnasium/wrappers/metric/utils.py", line 70, in op_dataclass
    new[field.name] = op(getattr(first, field.name), second)
  File "/home/sutpc/lym/SMARTS/smarts/env/gymnasium/wrappers/metric/utils.py", line 86, in divide
    return float(value / divider)
ZeroDivisionError: division by zero
WARNING:SumoTrafficSimulation:TraCI has provided a warning Vehicle 'Agent_0' is not known

[Gamenot]

Hello @lurenlym, thank you for the report, the benchmark is still a little unstable.

@Adaickalavan has also been training on the benchmark and should have better context on this particular error.

[Gamenot]

I am converting that assertion to a warning for now.

[Gamenot]

Master branch head now uses a warning instead.

[qianyi-sun]

Hi, fyi, not sure which scenario(s) you are using, in this PR https://github.com/huawei-noah/SMARTS/pull/2017/files we have some validated scenarios for cruising, turning and car following tasks, please feel free to use them.

[Adaickalavan]

Hi @lurenlym,

This issue is fixed by PR #2051.