Add runtime tests #475

michaeldeistler · 2024-10-29T16:33:11Z

Note that the tests are disabled in the workflow: pytest -m "not runtime"

jnsbck

Awesome! Thanks for implementing this, I think this will be really useful. I had also worked on the same thing, but was pursuing sth slightly different. I have added my proof of concept to this branch. (see also my comments). Feel free to work off of it or remove it. I think having a seperate CI for regressions might be useful in the long run.

Happy to discuss this more.

.github/workflows/regression_tests.yml

tests/regression_test_runner.py

tests/test_regression.py

tests/test_runtime.py

github-actions · 2024-11-22T14:35:59Z

New Baselines

test_runtime(num_cells=1, artificial=False, connect=False, connection_prob=0.0, voltage_solver=jaxley.stone)
🆕 build_time: (0.566s).
🆕 compile_time: (18.334s).
🆕 run_time: (2.804s).

test_runtime(num_cells=1, artificial=False, connect=False, connection_prob=0.0, voltage_solver=jax.sparse)
🆕 build_time: (0.394s).
🆕 compile_time: (3.041s).
🆕 run_time: (2.451s).

test_runtime(num_cells=10, artificial=False, connect=True, connection_prob=0.1, voltage_solver=jaxley.stone)
🆕 build_time: (1.836s).
🆕 compile_time: (29.308s).
🆕 run_time: (18.758s).

test_runtime(num_cells=10, artificial=False, connect=True, connection_prob=0.1, voltage_solver=jax.sparse)
🆕 build_time: (1.256s).
🆕 compile_time: (26.581s).
🆕 run_time: (25.064s).

test_runtime(num_cells=1000, artificial=True, connect=True, connection_prob=0.001, voltage_solver=jaxley.stone)
🆕 build_time: (109.680s).
🆕 compile_time: (45.643s).
🆕 run_time: (41.482s).

test_runtime(num_cells=1000, artificial=True, connect=True, connection_prob=0.001, voltage_solver=jax.sparse)
🆕 build_time: (108.686s).
🆕 compile_time: (61.090s).
🆕 run_time: (58.329s).

…ready for review

…timetests

jnsbck · 2024-11-22T15:20:18Z

@michaeldeistler This should be ready for review. The following is implemented and works:

tests/test_regression.py contains the regression tests that are run using pytest -m regression
automates writing results to database, comparing results stored baselines and producing a report (see above).
regression_tests.yml runs the regression tests.
update_regression_baseline.yml to update the baseline
idea is to trigger this on comment
comments with updated baseline report
pushes the updated baselines to the branch in which the workflow is run.

If you want to run this locally do you can do NEW_BASELINE=1 pytest -m regression i.e. on main and then run pytest -m regression on feature.

Some thoughts I had: Even with a 20% tolerance the regression tests frequently fail for some reason. (might need to average across several runs or take the max across several runs for the baseline, but currently the regression tests take quite long. Would be good to merge #489 to speed this up).

Lemme know if anything is unclear

michaeldeistler

Wow this is amazing!!!

I don't think I understand when exactly the updating of the times is being run. On a comment to this particular PR (i.e., #475)? Or some other PR/issue? Also, what should the comment be? Anything? Who would the comment have to be by? Anyone? (BTW I am commenting now, so does this trigger the regression tests :D ?)

michaeldeistler · 2024-11-22T15:44:03Z

.github/workflows/regression_tests.yml

+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e ".[dev]"


do we really need dev?

michaeldeistler · 2024-11-22T15:45:44Z

.github/workflows/update_regression_baseline.yml

+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e ".[dev]"


dev needed?

michaeldeistler · 2024-11-22T15:48:47Z

.github/workflows/update_regression_baseline.yml

+on:
+  issue_comment: # trigger from comment; event runs on the default branch
+    types: [created]
+  # pull_request:


michaeldeistler · 2024-11-22T15:49:00Z

.github/workflows/update_regression_baseline.yml

+        git commit -m "Update regression test baselines"
+        git push origin HEAD:${{ github.head_ref }}
+
+    # Needed when workflow is triggered from a comment


michaeldeistler · 2024-11-22T15:49:36Z

.gitignore

@@ -55,6 +55,8 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
+tests/regression_test_results.json
+tests/regression_test_baselines.json


I don't get this. Why do we first put it in the gitignore just to then force add it?

michaeldeistler · 2024-11-22T15:50:55Z

tests/test_regression.py

+        json.dump(result_data, f, indent=2)
+
+
+class compare_to_baseline:


CompareToBaseline to follow PEP style

michaeldeistler force-pushed the runtimetests branch 3 times, most recently from 4696db6 to 8089980 Compare October 29, 2024 16:37

michaeldeistler requested a review from jnsbck October 29, 2024 16:37

michaeldeistler force-pushed the runtimetests branch from 8089980 to 941752f Compare October 29, 2024 17:34

jnsbck reviewed Oct 30, 2024

View reviewed changes

jnsbck linked an issue Nov 4, 2024 that may be closed by this pull request

Add runtime tests #446

Open

This comment was marked as resolved.

Sign in to view

jaxleyverse deleted a comment from github-actions bot Nov 4, 2024

This comment was marked as resolved.

Sign in to view

jnsbck force-pushed the runtimetests branch 2 times, most recently from 5b0281b to 2855ff7 Compare November 21, 2024 19:03

jaxleyverse deleted a comment from github-actions bot Nov 21, 2024

jnsbck force-pushed the runtimetests branch 4 times, most recently from eb50f1b to 8e6de5a Compare November 22, 2024 10:53

jaxleyverse deleted a comment from github-actions bot Nov 22, 2024

jnsbck force-pushed the runtimetests branch from ea5ca3d to d30d790 Compare November 22, 2024 11:31

jaxleyverse deleted a comment from github-actions bot Nov 22, 2024

jnsbck force-pushed the runtimetests branch 2 times, most recently from 8cfc276 to 824a83f Compare November 22, 2024 14:00

jnsbck added 2 commits November 22, 2024 15:02

wip: new baselines

2877102

fix: rebase

b8798c5

jnsbck force-pushed the runtimetests branch from 824a83f to b8798c5 Compare November 22, 2024 14:04

jnsbck added 2 commits November 22, 2024 15:06

fix: fix rebase

9e52fbb

fix: fix pyproject merge

7048b76

Merge 7048b76 into 11279a5

ca01667

jaxleyverse deleted a comment from github-actions bot Nov 22, 2024

$username and others added 4 commits November 22, 2024 14:35

Update regression test baselines

bd96bd8

wip: add tests and regression tests back in, this should make the PR …

7f84813

…ready for review

Merge branch 'runtimetests' of github.com:jaxleyverse/jaxley into run…

16d31b0

…timetests

enh: print report only if there is regression tests in the session

28a52ee

jnsbck self-requested a review November 22, 2024 15:20

fix: fix username env variable not working

a4f0539

This comment was marked as resolved.

Sign in to view

michaeldeistler commented Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add runtime tests #475

Add runtime tests #475

michaeldeistler commented Oct 29, 2024 •

edited

Loading

jnsbck left a comment

This comment was marked as resolved.

This comment was marked as resolved.

github-actions bot commented Nov 22, 2024

jnsbck commented Nov 22, 2024 •

edited

Loading

This comment was marked as resolved.

michaeldeistler left a comment

michaeldeistler Nov 22, 2024

michaeldeistler Nov 22, 2024

michaeldeistler Nov 22, 2024

michaeldeistler Nov 22, 2024

michaeldeistler Nov 22, 2024

michaeldeistler Nov 22, 2024

		json.dump(result_data, f, indent=2)


		class compare_to_baseline:

Add runtime tests #475

Are you sure you want to change the base?

Add runtime tests #475

Conversation

michaeldeistler commented Oct 29, 2024 • edited Loading

jnsbck left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

github-actions bot commented Nov 22, 2024

New Baselines

jnsbck commented Nov 22, 2024 • edited Loading

This comment was marked as resolved.

michaeldeistler left a comment

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler Nov 22, 2024

Choose a reason for hiding this comment

michaeldeistler commented Oct 29, 2024 •

edited

Loading

jnsbck commented Nov 22, 2024 •

edited

Loading