improve performance of diffs #32393

bohde · 2024-10-31T16:53:26Z

This has two major changes that significantly reduce the amount of work done for large diffs:

Kill a running git process when reaching the maximum number of files in a diff, preventing it from processing the entire diff.
When loading a diff with the URL param file-only=true, skip loading stats. This speeds up loading both hidden files of a diff and sections of a diff when clicking the "Show More" button.

A couple of minor things from profiling are also included:

Reuse existing repo in PrepareViewPullInfo if head and base are the same.

The performance impact is going to depend heavily on the individual diff and the hardware it runs on, but when testing locally on a diff changing 100k+ lines over hundreds of files, I'm seeing a roughly 75% reduction in time to load the result of "Show More"

bohde · 2024-10-31T17:49:07Z

Here are quick before & after videos showcasing viewing the files in a large pull request:

Before:
https://github.com/user-attachments/assets/c910f646-a23f-4088-8e8b-a2ec5e48bd62

After:
https://github.com/user-attachments/assets/f75316cb-91a8-4e80-b24c-70dffc4b5fd2

services/gitdiff/gitdiff.go

routers/web/repo/pull.go

services/gitdiff/gitdiff.go

services/repository/files/temp_repo.go

wxiaoguang

I could vote my approval (although the error handling seems strange, I might propose some following PRs if I could get a clear picture for it).

This has two major changes that significantly reduce the amount of work done for large diffs: * Kill a running git process when reaching the maximum number of files in a diff, preventing it from processing the entire diff. * When loading a diff with the URL param `file-only=true`, skip loading stats. This speeds up loading both hidden files of a diff and sections of a diff when clicking the "Show More" button. A couple of minor things from profiling: * Reuse open git commits if possible to avoid querying the repo. * Reuse existing repo in `PrepareViewPullInfo` if head and base are the same.

wxiaoguang · 2024-11-02T04:10:11Z

We need this to correctly handle the errors: Fix git error handling #32401

* giteaofficial/main: (21 commits) Use 8 as default value for git lfs concurrency (go-gitea#32421) Fix milestone deadline and date related problems (go-gitea#32339) Only query team tables if repository is under org when getting assignees (go-gitea#32414) Refactor RepoRefByType (go-gitea#32413) Refactor template ctx and render utils (go-gitea#32422) Refactor DateUtils and merge TimeSince (go-gitea#32409) Refactor markup package (go-gitea#32399) Add some handy markdown editor features (go-gitea#32400) Make LFS http_client parallel within a batch. (go-gitea#32369) Refactor repo legacy (go-gitea#32404) Replace DateTime with proper functions (go-gitea#32402) Fix git error handling (go-gitea#32401) Fix created_unix for mirroring (go-gitea#32342) Replace DateTime with DateUtils (go-gitea#32383) improve performance of diffs (go-gitea#32393) Refactor tests to prevent from unnecessary preparations (go-gitea#32398) Add artifacts test fixture (go-gitea#30300) Fix `missing signature key` error when pulling Docker images with `SERVE_DIRECT` enabled (go-gitea#32365) Fix a number of typescript issues (go-gitea#32308) Update go dependencies (go-gitea#32389) ...

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Oct 31, 2024

pull-request-size bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 31, 2024

github-actions bot added the modifies/go Pull requests that update Go code label Oct 31, 2024

lunny added the performance/speed performance issues with slow downs label Oct 31, 2024

lunny reviewed Oct 31, 2024

View reviewed changes

services/gitdiff/gitdiff.go Outdated Show resolved Hide resolved