Lighthouse dying because of short memory spikes

[Aracki]

Description

Few times a day our Lighthouse containers are having short but high memory spikes. Beforehand it's showing a lot of:

ERRO Unable to validate attestation error: ObservedAttestationsError(SlotTooLow { slot: Slot(1035781), lowest_permissible_slot: Slot(1035797) }), peer_id: 16Uiu2HAkzJkSWstNAD1PR915MBhyByDbw8W1DJ5q7JJkMZa3saQd, type: "aggregated", slot: Slot(1035781), beacon_block_root: 0x74c201e1dfd6cc77dbc13ab8b93a791a0fc5f76aa0fcf42971cd0d4666aff3d1"

Besides this error, nothing useful can be seen (we haven't enabled DEBUG though).

Version

sigp/lighthouse:v4.6.0

It's interesting that it's happening only with Holesky network. We have LH for Mainnet & Sepolia as well, but we don't see these issues.

[GirnaarNodes]

we are also facing the same issue

[michaelsproul]

How much memory is Lighthouse using when it OOMs? If you look at sudo dmesg -T | grep killed then it will show the resident set size (RSS).

On Holesky, Lighthouse routinely needs ~8GB of RAM. We are working on optimising this, but Holesky is inherently more resource hungry than other networks due to the higher validator count.

We've also fixed some issues for v5.0.0 which should improve memory usage, see:

• Drop gossipsub stale messages when polling ConnectionHandler.  #5175

And related issue:

• 4.6.0 beacon_node memory usage issue #5227

[michaelsproul]

The SlotTooLow error isn't related, that was a separate bug, also fixed for v5.0.0 but low impact:

• SlotTooLow errors from observed aggregates #5134

[Aracki]

Mean usage is around ~8Gi yes, but it seems it hits our memory limit of 12Gi very often.

[chong-he]

Mean usage is around ~8Gi yes, but it seems it hits our memory limit of 12Gi very often.

12GB is tough. At least 16GB is required and a 32GB is recommended, particularly on Holesky where the number of validators is high

[michaelsproul]

This PR will also help: #5270

Once that's merged you could try running unstable with --state-cache-size 2. We'll release v5.1.0 with that change quite soon.

[SimplyCorey]

I'm seeing this issue at least 2-3 times a day. I was seeing this ~weekly before I upgraded to the latest release. I've tried setting the state-cache-size flag and it doesnt appear to help. Anything I can provide here to help debug?

[michaelsproul]

@SimplyCorey I was doing some memory debugging recently, and a large amount of Lighthouse's spikey behaviour comes from doing I/O. Do you have a fast NVME SSD?

[michaelsproul]

If you DM me debug logs I can take a closer look to rule out any other problem. I'm @sproul on Discord. You can find debug logs in $datadir/beacon/logs. Best to compress them before sending.