4GB memory not enough / crasher #767
Description
Describe the bug
I set up a dedicated AWS EC2 linux instance with 4GB of RAM (listed in requirements) w/o swap to run an avalanchego (avalanche/1.1.5) validator. It bootstrapped pretty quickly and then crashed a number of hours later. I lost the scrollback in my terminal, and there wasn't anything interesting in the log.
I ran it again and watched it over the last 24 hours, and piped stderr/stdout to a file. I noticed:
- memory usage slowly climbed until 4GB:
- it then crashed.
To Reproduce
./avalanchego --log-dir ./log/ --api-keystore-enabled=false --api-health-enabled=true --api-info-enabled=true --api-keystore-enabled=false --public-ip [IP I USED] &>> ./log/output.log
Expected behavior
server running fine on a machine with 4GB of RAM per avalanche validator docs
Screenshots
An hour or so before it crashed:
My friend says his validator is also using more than 4GB of RAM, but his machine can handle it:
Operating System
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Additional context
I saw a previously closed issue (#717) from a user who ran an older version of avalanchego and then upgraded, maybe causing some DB issues. To be clear, this is the only version I've run.
from outputting to a log:
INFO [02-10|15:15:27] <C Chain> snow/engine/snowman/transitive.go#116: bootstrapping finished with VEtPRUjubZYc5gu4cxc68r8BYSQs2Z2Pnin9iaxdcqFEcNZXX as the last accepted block
INFO [02-11|20:43:44] api/info/service.go#182: Info: IsBootstrapped called with chain: X
INFO [02-11|20:45:14] api/info/service.go#182: Info: IsBootstrapped called with chain: X
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f9f979bbb20 m=8 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f9f979bbb20
stack: frame={sp:0x7f9f68dfa7b0, fp:0x0} stack=[0x7f9f685fb288,0x7f9f68dfae88)
00007f9f68dfa6b0: 0000000000000000 0000000000000000
00007f9f68dfa6c0: 0000000000000000 0000000000000000
00007f9f68dfa6d0: 0000000000000000 0000000001b8ce80
00007f9f68dfa6e0: 0000000000000000 0000000000000000
... and then about 40,000 more lines of go stack
By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.