Netbox gets slower over time - unless you use it

[telmich]

NetBox Edition

NetBox Community

NetBox Version

4.4.2

Python Version

3.10

Steps to Reproduce

When running netbox, in my case in k8s and waiting a bit (hours), netbox can take 10 seconds to load.

I am using /api/status/ to check the time and most of the time it loads in < 1 second, sometimes it's 10 seconds.

If I add prometheus monitoring to monitor the behaviour, the 10 second delay goes away - it stays at at about 0.5 seconds for loading.

If I disable prometheus monitoring from the outside, the 10 second delay appears sometimes during an hour.

This smells a bit like swapping out, however the nodes do not have swap. Also, all other applications on the same node run without any delay.

It even smells as if "something" is gone from "a cache" (L1, L2, ...) netbox becomes very slow.

From what I understand, calling the /api/status endpoint does not involve database calls, so it's unlikely a performance issue in postgres.

I read about similar behaviour in #16454 and #16684 however my version is much newer.

Any hints on what might be going wrong here are appreciated.

Expected Behavior

No random delays.

Observed Behavior

Delays are randomly going up to 10 seconds

[rodvand]

Have a look at https://github.com/netbox-community/netbox-docker/blob/release/docker/nginx-unit.json and https://unit.nginx.org/configuration/#configuration-proc-mgmt-prcs. We had to adjust some of these values to avoid a long "spin up" time.