Performance regression on v1.9+ #7910
Labels
issue/confirmed
Issue has been reviewed and confirmed to be present or accepted to be implemented
issue/needs-feedback
For bugs, we need more details. For features, the feature must be described in more detail
performance/memory
Performance issues affecting memory use
performance/speed
performance issues with slow downs
Comments
|
Which operations did you do on gitea? And could you give any error log? |
|
I've made no changes since installing it pre v1. I generally upgrade the binary 3 days after each release. Full log: Log configuration: Git config: System Notices (5 of the same): Thank you so much for looking into this! |
|
There seems too many gitea processes imho... Can you stop gitea and check if all processes are killed actually |
|
Done. Ran My systemd file: |
|
Do you have the repo indexer enabled? If yes, it could be the cause. After it's done updating the index, CPU should go to idle levels. |
|
Does it happen on a fresh install? ( including the performance regression and the runtime errors) |
|
@silverwind I don't have it enabled - I saw the strict warning and decided to use Sourcegraph on a different machine that's more powerful. @typeless I will try to find time on the weekend to set up a secondary instance and repro. Been using the same config for a while so want to be careful to not nuke the data. |
|
@issmirnov Okay. I am particularly concerned about whether it is related to the state of the database. |
|
Whoa. So here's a wild datapoint. I restarted the service 4 days ago, and look at the memory allocation! It's 321GB. There have also been 6 BILLION memory allocations and frees.
Current Memory Usage: 75MB |
|
@issmirnov I think |
|
@lunny understood, been meaning to set up a prometheus/grafana stack. I've tried to do some digging this weekend. Haven't had a chance to spin up a fresh install. A few discoveries:
Here's the full stack trace:
The reason I bring this up is that I see 4 instances of |
|
If you only have one gitea web site, you should only see ONLY one |
One valid case in which 4 instances can happen is when Gitea is executed from a Of course all of this can happen only if the system is live, with users connecting and making requests, or during a migration process. Only one instance should be a long time runner, though. |
|
Yes, I mean only one |
|
I've been digging into this every spare chance that I get, and am unable to track down the root cause. Running gitea standalone also had high CPU usage, but I can't get reliable reproduction. In the meantime I've scaled my droplet to 2GB ram and 2vCPU, and used |
|
Gitea has built-in |
|
Thanks. I'm waiting to spin up my prometheus stack to track this. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions. |
lunny
added
issue/confirmed
|
Maybe it would make sense to add the labels |
|
Not sure if it helps, but freshly installed gitea on pi 4 constantly eats 5% cpu (no repositores created) |
lafriks
added
performance/memory
|
We have more features today and some of them are enabled defaultly. I think that's one reason why gitea spend more resources than before. |
|
how can I disable some of the features enabled defaultly? I checked the documentation and didn't find anything related to performance (apart from indexing, which is turned off by default for repositories). adding MAX_WORKERS=1 and TYPE=level to [queue] section didn't help |
If this is true, then why do I have 10 (!) gitea web processes running with the default settings? Gitea is idle almost all of the time. Once every 24 hours, it has to sync a few very small GitHub-mirrored repos, that's it. The main thread constantly uses 2% CPU and the other threads alternate in using 1%. This isn't horrible, of course, but I don't understand what could possibly cause it. Raspberry Pi 4 on Arch Linux |
|
You're probably confusing threads with processes. Enable "Hide userland threads" in htop. |
Great, that was stupid. Been using htop instead of top for less than a day now, I think it shows! 😅 Thanks for reminding me. Nevertheless, I think the CPU usage is still strange. |
|
Updating from Gitea 1.11.6 to 1.12.3 doubled the CPU use. Are there any settings I could/should optimize? I've tried disabling notifications under |
|
I used to take a look at this problem. Unfortunately, I don't have much spare time to dig further. To enable |
7 tasks
6 tasks
|
I'm not sure if it's the same problem, but my |
|
@lnicola you need to tell us what version of gitea and what OS. It would be helpful if you were to use a pprof and give us cpuprofile out instead. |
|
@zeripath I'm not familiar with |
|
In app.ini: then will grab a cpuprofile that can be examined with There are other profiles including It would also be helpful to know what are kind of machine you running gitea on? What DB? Are you using repo indexing? Are your repo stats up-to-date? Are you running out of memory? And so on - but the cpuprofile would no doubt be helpful. |
|
I think we can close this one and please open new issue since we have released 1.14rc1 which far away from 1.9 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
[x]):Description
Hey folks. My server ($5 digital ocean droplet, 1vCPU 1gb RAM) has been really struggling with the latest gitea release. Up to v1.7/1.8 or so it was completely fine. Since installing v1.9,
htopand other tools consistently show thegitea webprocess eating up a combined 80% of my CPU.Are there any performance regression tests for gitea? These recent changes are having a major impact on my entire server, whereas previously it coexisted fine with about 30 services.
Looking at the logs, I do see tons of runtime errors. Unsure if this is related.
Thanks so much - really love gitea, and would love to see the resource usage become normal again.
Screenshots
The text was updated successfully, but these errors were encountered: