This repository has been archived by the owner on Jun 30, 2023. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
jar: prefer io.ReadFull over io.ReadAll
This io.ReadAll() call is responsible for a bit more than half the allocations observed in a very highly scaled up instance of this [1]. io.ReadAll does repeated allocations [1], starting with a 512 byte buffer. We know the size of the file, so allocate it all at once. A future commit should look at a further optimization: using a buffer pool. That wasn't done in this CL because it requires dealing with the recursive call. [1]: ``` (pprof) list checkJAR Total: 213.06GB ... . 9.89MB 334: f, err := zf.Open() . . 335: if err != nil { . . 336: return fmt.Errorf("open file %s: %v", p, err) . . 337: } . . 338: defer f.Close() . 169.96GB 339: data, err := io.ReadAll(f) . . 340: if err != nil { . . 341: return fmt.Errorf("read file %s: %v", p, err) . . 342: } . . 343: br := bytes.NewReader(data) . 3.40GB 344: r2, err := zip.NewReader(br, br.Size()) ... (pprof) list ReadAll Total: 213.06GB ROUTINE ======================== io.ReadAll in third_party/go/gc/src/io/io.go 113.40GB 169.96GB (flat, cum) 79.77% of Total . . 638:func ReadAll(r Reader) ([]byte, error) { . . 639: b := make([]byte, 0, 512) . . 640: for { . . 641: if len(b) == cap(b) { . . 642: // Add more capacity (let append pick how much). 113.40GB 113.40GB 643: b = append(b, 0)[:len(b)] . . 644: } . 56.56GB 645: n, err := r.Read(b[len(b):cap(b)]) . . 646: b = b[:len(b)+n] . . 647: if err != nil { . . 648: if err == EOF { . . 649: err = nil . . 650: } ``` [1]: https://cs.opensource.google/go/go/+/master:src/io/io.go;l=638;drc=2580d0e08d5e9f979b943758d3c49877fb2324cb
- Loading branch information