-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI test for RL examples: Drive and VehicleFollowing #2062
Conversation
@@ -6,7 +6,7 @@ env: | |||
venv_dir: .venv | |||
|
|||
jobs: | |||
base-tests-linux: | |||
base-tests: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not against it but is there any reason to change the name of the test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the addition of more jobs in the same workflow, following the original naming style, each of the job would have been suffixed with -linux
. The suffix appeared a little repetitive, given the workflow's name of SMARTS CI Base Tests Linux
which indicates it is for Linux. In short, the change was for aesthetics. This is how the test results look like now.
-k 'not test_long_determinism' | ||
|
||
examples-rl: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How long do these two tests take to run? We would want the total to be ~2 mins or less.
Otherwise, we will need to put it on some other schedule than always run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Currently the
examples-rl
test takes ~20 minutes - By reducing the number of routes generated in the scenarios, the
examples-rl
test time may be reduced to ~12 minutes - Previously the
base-tests
test took ~53 minutes to run when it attempted to build all scenarios. - By only building the necessary scenarios, the
base-tests
now takes ~6 minutes. - For a quick benefit, we may make another PR consisting of only building the necessary scenarios which will hasten the
base-tests
, while we figure how to best do CI forexamples-rl
test.
The test times appear to be ~18 minutes and 7 minutes. I think we should find a way to run these less frequently, maybe only against certain commit messages and master. |
make build-all-scenarios
command and only build necessary scenarios in CI to reduce test time.