Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Visiting settings/admin/logging consumes CPU/RAM and triggers out of memory crash when Live is on #1262

Open
5 of 8 tasks
Lawkss opened this issue May 25, 2024 · 23 comments

Comments

@Lawkss
Copy link

Lawkss commented May 25, 2024

⚠️ This issue respects the following points: ⚠️

Bug description

Visiting settings/admin/logging crashes Edge and Firefox.

You can scroll up and down a bit then CPU load increases dramatically. The size of the logfile is 26MB.

RAM usage increases by several GB in seconds

and then the browser tab crashes: "Out of Memory"

Screenshot 2024-05-25 193730

Firefox survives sometimes:

Screenshot 2024-05-25 193921

For temporary fix clearing the logfile helps.

Steps to reproduce

  1. Uprade NC 28 to NC 29
  2. Visit Logging page

Expected behavior

No crashes.

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

29

Operating system

RHEL/CentOS

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "vom-bruch.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.1.1",
        "overwrite.cli.url": "https:\/\/vom-bruch.com\/cloud",
        "htaccess.RewriteBase": "\/cloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        },
        "default_phone_region": "AT",
        "simpleSignUpLink.shown": false,
        "app_install_overwrite": [
            "unsplash"
        ],
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "maintenance_window_start": 1,
        "updater.release.channel": "stable"

List of activated Apps

Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.4
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.6.1
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - passman: 2.4.9
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.2
  - richdocumentscode: 24.4.201
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - spreed: 19.0.1
  - support: 1.12.0
  - survey_client: 1.17.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - encryption: 2.17.0
  - files_external: 1.21.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

@Lawkss
Copy link
Author

Lawkss commented Jul 6, 2024

this is still happening in 29.0.3

just visiting logging page will immediately ramp up CPU fan and load.

@LM-vb
Copy link

LM-vb commented Jul 23, 2024

The freezing/crashing does not happen to me any longer, since the update to 29.0.4. Can someone cross-check?
The details of an entry not showing up, see #1275, however, still happens.

@Ornanovitch
Copy link

It is still happening with NC 29.0.4 on FF on my side...

@LM-vb
Copy link

LM-vb commented Jul 24, 2024

No freezes on Chrome (127.0.6533.73), Firefox (128.0) or Safari on my side. That's weird.

@baermb
Copy link

baermb commented Sep 5, 2024

FWIW, this has been a problem for me on two different NC installations: AIO on Debian; straight install on Ubuntu. And its been happening at least since 29.0.3 and possibly before (currently: the AIO is 29.0.4 and the other is 29.0.6). I've mostly tested with Firefox 115.14.0esr and a little chrome 128.0.6613.119. The tab will freezes up either immediately or after some use, and sometimes the whole window freezes. On occasions parts of the window are clickable and parts are not, even outside of the tab.
In my case, I have access to the log files so this is less of a priority. But looking at logs in the browser is just darkly ironic.

@Ornanovitch
Copy link

I tried to analyze this problem a bit more today, and I maybe have a clue. It doesn't freeze systematically, but when it does, this is because the poll?lastReqId request asks for way too much entries (hundreds), eating all my memory. It doesn't wait for me to scroll and to limit the entries with log?offset=50&query=

@spyro1248
Copy link

spyro1248 commented Sep 22, 2024

I am experiencing the same bug. As usual, it appears to be nextcloud making some kind of enormous memory allocation.

This has been happening since at least (from memory NC26, and persists in the current version (29.0.7 at the time of writing).

The logging page (when it has worked) has always been incredibly slow and prone to crashing my browser (often taking down my entire laptop - I'm not rich and havent got much RAM.)

In spite of being lowish on RAM, No other website I browse is this abysmally sluggish, and I can normally have over 50 tabs open on the machine on various sites.

@WadohS
Copy link

WadohS commented Oct 1, 2024

i experienced the same problem, restart apache2.service make off the problem, but reappear between 2-3 minutes after restarting

@Lawkss
Copy link
Author

Lawkss commented Oct 4, 2024

still happening in nextcloud 30.0.1 RC1

@joshtrichards
Copy link
Member

I tried to analyze this problem a bit more today, and I maybe have a clue. It doesn't freeze systematically, but when it does, this is because the poll?lastReqId request asks for way too much entries (hundreds), eating all my memory. It doesn't wait for me to scroll and to limit the entries with log?offset=50&query=

Is your system generating lots of new log entries every second? Are the in-browser Log reader settings set to Debug perhaps?

The poll query only asks for new queries since the last ReqId the front-end knows about. It'll poll every 10s, but the lastReqId=xxx value shouldn't change unless there have been new entries. And the entries within the response should be minimal since, at most, they'd cover a 10s period.

Those reporting this matter:

  • What are your Log reader settings? Specifically:
    • Filters
    • Log level (note: it's possible for this to be a different number than you see in your config.php)
    • Time format
    • Live view
  • How many new log entries are you seeing your log file typically over a one minute period?

Also, in case of interactions with browser extensions, try reproducing with extensions off (and maybe in private browsing mode) just to see if it changes behavior.

@Ornanovitch
Copy link

Thanks @joshtrichards

My log reader settings is set on the Warning level as well as the filters (Warning, Error, Fatal). The local time format is checked as well as the live view.I don't have a lot of entries over a one minute period, sometime only 2 or 3! As I said, the reader doesn't freeze systematically, but when it does it doesn't looks like a number of entries problem

@joshtrichards
Copy link
Member

joshtrichards commented Oct 5, 2024

Reproduced, though I'm not sure how/what precisely triggers it at the moment. It more sort of just... started happening.

@Lawkss
Copy link
Author

Lawkss commented Oct 5, 2024

I am not generating that many entries. Sometimes non per day. It also doesnt want to display the details of log entries.

for me it always freezes almost immediately. I dont know what you mean by time format I use 24h clock system

I really hope it can be debugged.

@baermb
Copy link

baermb commented Oct 7, 2024

Because of the above thread, I took another look at logging on two instances (AIO docker NC 29.0.4 and a stand alone NC 29.07). If I turn off Live-view/Polling in Log reader's settings, both of them seem to function without an issue. If I turn it back on again, the window freezes.
From the hand full of attempts I did, I generally needed to get into the Logging settings dialog and turn the Live-view setting off fairly quickly, before the window froze. And even after that, that window would freeze and I'd need to open a new window and go to NC's Logging again. If I successfully turned polling off, the logging screen then didn't cause any issues for me (unless I was foolish enough to turn Live-view back on to test it anyway :) ).

@denppa
Copy link

denppa commented Oct 17, 2024

Same here, turning on live view causes a run away thread and freezes tab. Turning it off resolves the crash.

@Lawkss
Copy link
Author

Lawkss commented Oct 17, 2024

Seems fixed for me in 30.0.1 stable branch.

@denppa
Copy link

denppa commented Oct 18, 2024

Waiting for the docker images to update, currently using NC30.0.0. Will report back once the release is made.

@Lawkss
Copy link
Author

Lawkss commented Oct 19, 2024

the crashes are now happening again. No idea why it worked for a brief while.

@joshtrichards joshtrichards changed the title [Bug]: Visiting settings/admin/logging crashes Edge and Firefox [Bug]: Crashes browsers(s): Edge and Firefox Oct 19, 2024
@joshtrichards joshtrichards pinned this issue Oct 19, 2024
@joshtrichards
Copy link
Member

Perhaps we need to introduce a delay in the live view polling.

@punxxx-me
Copy link

Still there in 30.0.2

Tested with Chrome and Firefox

@joshtrichards joshtrichards changed the title [Bug]: Crashes browsers(s): Edge and Firefox [Bug]: Visiting settings/admin/logging consumes CPU/RAM and triggers out of memory crash sometimes Nov 8, 2024
@joshtrichards joshtrichards changed the title [Bug]: Visiting settings/admin/logging consumes CPU/RAM and triggers out of memory crash sometimes [Bug]: Visiting settings/admin/logging consumes CPU/RAM and triggers out of memory crash when Live is on Nov 8, 2024
@Cees2439867
Copy link

Still happens in 30.0.3 but not so fast as previous. I thought it was fixed, but no luck.

@Lawkss
Copy link
Author

Lawkss commented Dec 7, 2024

Still happens in 30.0.3

@Lawkss
Copy link
Author

Lawkss commented Dec 14, 2024

Edge crashes almost immediately : RESULT_CODE_HUNG on 30.0.4.

I am sorry, I am not a developer, I don't understand the problem or its complexity on why this is still a thing since May 2024.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

12 participants