chore: make fiximports run in parallel#12985
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the fiximports script to improve performance by processing files in parallel.
- Introduces concurrent reading, processing, and writing of file contents.
- Replaces the sequential fixGoImports function with parallel pipelines using worker routines.
|
Seen #11695 (comment) ? |
|
No, but
Yes, that was my initial thought as I was working on this because it's that global that stops simple parallelism. Figured I'd try for least-change though and this yields pretty good results. |
masih
left a comment
There was a problem hiding this comment.
No blockers.
I think the logic can be simplified quite a bit using error groups.
|
Yeah, errgroups made it much better, thanks. PTAL @masih |
There was a problem hiding this comment.
Pull Request Overview
This PR addresses the performance issues in the fiximports tool by processing files in parallel. Key changes include:
- Reading file contents concurrently before processing.
- Using parallel execution to run imports.Process for multiple prefixes.
- Writing modified files concurrently after processing.
7d93e11 to
c9e5ef4
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR aims to significantly reduce the execution time of fiximports by processing files in parallel, reducing the overall runtime drastically.
- Converts file reading, processing, and writing to parallel operations.
- Leverages errgroup with a worker limit based on the number of CPUs.
fiximports has been driving me a little insane lately. For some reason it's started taking up to ~40s each execution and I can't figure out why. I tried switched from running go from a snap to just installing it from a package, I tried clearing my go build cache, I tried removing all the x/tools sources I had. Maybe I made a difference somewhere in there, but I'm still ~20s.
When measuring, the time cost is all in
imports.Process, so something in there isn't having a happy time.After a couple of iterations of trying to fix it I've landed on the following approach which gives me the best results:
imports.LocalPrefix, process all files in parallel—so we still doimports.Processtwice but we do the files in parallel and don't need to worry aboutimports.LocalPrefixmutationBefore:
After:
Almost too good to be true? Seems to work just fine though. @masih would you mind taking a good look at this when you have some time please?