Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory and runtime benchmark/testing infrastructure #8858

Open
stscijgbot-jp opened this issue Oct 4, 2024 · 1 comment
Open

memory and runtime benchmark/testing infrastructure #8858

stscijgbot-jp opened this issue Oct 4, 2024 · 1 comment

Comments

@stscijgbot-jp
Copy link
Collaborator

stscijgbot-jp commented Oct 4, 2024

Issue JP-3775 was created on JIRA by Ned Molter:

Add and implement tools to ensure that pull requests do not inadvertently cause significant regressions in memory usage or runtime.

At a hack day discussion, we targeted initial implementation and testing in stcal, since it's a smaller repository but still has non-trivial memory- and runtime-hungry functions (e.g., outlier detection median calculation).  Goal is very quick adoption by JWST, and adoption by Roman once other outstanding regression test issues have been resolved.

The preferred workflow would be to have decorators that could be added to tests to flag them as memory- and/or runtime-checked.  At time of writing, codspeed is being considered for runtime, and pytest-memray is being considered for memory.  

Task list for now:

  • Add memory usage unit tests for methods where memory optimization has been attempted, e.g. the median calculator in outlier detection.
  • Test out codspeed by starting to use it in stcal
  • Figure out a good general way to report memory usage on PR branches
  • Decide whether memory usage regression tests are needed, or if benchmarking and showing the results to PR developers is good enough
@stscijgbot-jp stscijgbot-jp changed the title memory and runtime benchmark infrastructure memory and runtime benchmark/testing infrastructure Oct 4, 2024
@stscijgbot-jp
Copy link
Collaborator Author

Comment by Ned Molter on JIRA:

A PR in stcal adds a single unit test to the median calculator ensuring its memory usage does not go substantially above the expected amount.  The structure of this test, using tracemalloc and comparing the tracemalloc.get_traced_memory() high-water mark with something, should be used in cases where the expected memory usage can be calculated straightforwardly based on the input size. 

PR link here: spacetelescope/stcal#299 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant