Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3336] [Test Performance] Optimize the set_up_plugin_manager() function #9029

Closed
peterallenwebb opened this issue Nov 7, 2023 · 0 comments · Fixed by #9031
Closed

[CT-3336] [Test Performance] Optimize the set_up_plugin_manager() function #9029

peterallenwebb opened this issue Nov 7, 2023 · 0 comments · Fixed by #9031
Assignees
Labels
performance tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality

Comments

@peterallenwebb
Copy link
Contributor

peterallenwebb commented Nov 7, 2023

Short description

Profiling shows that the set_up_plugin_manager() function takes up 30% of the runtime on the main thread during integration testing. The hot path is set_up_plugin_manager() -> from_modules() -> iter_modules(). By caching the results we need after the first time iter_modules() is called, nearly all this time can be won back.

Acceptance criteria

  • The iter_modules() function is only called once in the above call tree for each test process.
  • We have documented a notable improvement in performance in GitHub or via local profile.

Impact to Other Teams

None. They may enjoy the speedier tests, though.

Will backports be required?

No.

@peterallenwebb peterallenwebb added the user docs [docs.getdbt.com] Needs better documentation label Nov 7, 2023
@github-actions github-actions bot changed the title [Test Performance] Optimize the set_up_plugin_manager() function [CT-3336] [Test Performance] Optimize the set_up_plugin_manager() function Nov 7, 2023
@peterallenwebb peterallenwebb added performance tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality and removed user docs [docs.getdbt.com] Needs better documentation labels Nov 7, 2023
@peterallenwebb peterallenwebb self-assigned this Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant