1.4.10 crashes with `Illegal Instruction`

[ssg]

Describe the bug
The docker image for 1.4.10 quits immediately with an "Illegal instruction" error. No such problem in any of the previous versions, including 1.4.9.

To Reproduce
Run avalanchego

Expected behavior
It should run fine.

Error

/avalanchego/build # ./avalanchego
Illegal instruction (core dumped)

Operating System
Linux 3.10.105 x86_64

Additional context
/proc/cpuinfo for the container:

/avalanchego/build # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 76
model name      : Intel(R) Pentium(R) CPU  N3710  @ 1.60GHz
stepping        : 4
microcode       : 0x404
cpu MHz         : 1601.000
cache size      : 1024 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch ida arat epb invpcid_single tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips        : 3199.85
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual

By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.

[danlaine]

Hi, thanks for reporting this. Can you please provide an exact set of steps to reproduce? How are you invoking the docker image?

[ssg]

Hi @danlaine, I use the same command for all container versions:

sudo docker run -d -http-host= -it --restart unless-stopped --network host --name ava -v $HOME/.avalanchego:/root/.avalanchego avaplatform/avalanchego /avalanchego/build/avalanchego

I noticed the issue when I noticed that the container was constantly restarting. Then I attached a shell and tried to run it manually, that's when I noticed the error.

[ssg]

I was able to repro this out of the docker image.

[StephenButtolph]

Hi, how are you building the docker image that you're running against?

[ssg]

Hi @StephenButtolph, I'm using the docker image in the avaplatform account with the run command I shared above. Is there anything I'm missing?

[ssg]

The error message outside a container is slightly different:

./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
./avalanchego: /lib/libstdc++.so.6: no version information available (required by ./avalanchego)
Illegal instruction (core dumped)

[ssg]

This is the stack trace from the core dump:

#0  0x000000000051e0e6 in __static_initialization_and_destruction_0(int, int) [clone .constprop.420] ()
#1  0x000000000154cb3d in __libc_csu_init ()
#2  0x00007f0d00b9ffbf in __libc_start_main () from /lib/libc.so.6
#3  0x00000000005396aa in _start ()

[ssg]

The problematic instruction is this:

=> 0x000000000051e0e6 <+294>:   c5 f9 6f 85 d0 f0 ff ff vmovdqa -0xf30(%rbp),%xmm0

[ssg]

This is a Braswell CPU and as I understand that AVX instruction set requires at least a Sandy Bridge CPU, that's why it's started crashing. 1.4.10 seems to have AVX instructions enabled in the build, and Braswell doesn't support AVX.

[mbehrsf]

I have the same error on my Celeron N3160 CPU. Not using docker, I just downloaded and untared the binary and also tried building avalanchego myself on Ubuntu 20.04. 1.4.9 still works fine.

[aphexmunky]

I'm having this error only on an older PC i'm using. It's an AMD FX-8350 and apparently supports AVX

Program received signal SIGILL, Illegal instruction.
0x00000000012eb1ff in rocksdb::LRUCache::LRUCache(unsigned long, int, bool, double, std::shared_ptr<rocksdb::MemoryAllocator>, bool, rocksdb::CacheMetadataChargePolicy) ()

gdb points to this part specifically, full logs below

➜  avalanchego git:(master) ./scripts/build.sh           
Downloading dependencies...
Building AvalancheGo...
Building AvalancheGo binary manager...
Building Coreth @ v0.5.5-rc.1 ...
Fetching AvalancheGo v1.4.5-preupgrade.4 ...
Note: switching to 'bb216f9433d30449a83607310f1f06beba8bb1c7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Building AvalancheGo v1.4.5-preupgrade.4 ...
/home/aphex/go/src/github.com/ava-labs/avalanchego
Build Successful

➜  avalanchego git:(master) ./build/avalanchego --version
[1]    36286 illegal hardware instruction (core dumped)  ./build/avalanchego --version

(gdb) file ./build/avalanchego
Reading symbols from ./build/avalanchego...
warning: File "/usr/local/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /usr/local/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/aphex/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/aphex/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
(gdb) r
Starting program: /home/aphex/go/src/github.com/ava-labs/avalanchego/build/avalanchego 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x00000000012eb1ff in rocksdb::LRUCache::LRUCache(unsigned long, int, bool, double, std::shared_ptr<rocksdb::MemoryAllocator>, bool, rocksdb::CacheMetadataChargePolicy) ()
(gdb) bt
#0  0x00000000012eb1ff in rocksdb::LRUCache::LRUCache(unsigned long, int, bool, double, std::shared_ptr<rocksdb::MemoryAllocator>, bool, rocksdb::CacheMetadataChargePolicy) ()
#1  0x00000000012eb400 in rocksdb::NewLRUCache(unsigned long, int, bool, double, std::shared_ptr<rocksdb::MemoryAllocator>, bool, rocksdb::CacheMetadataChargePolicy) ()
#2  0x00000000012eb5de in rocksdb::NewLRUCache(rocksdb::LRUCacheOptions const&) ()
#3  0x00000000011db10e in rocksdb::BlockBasedTableFactory::InitializeOptions() ()
#4  0x00000000011db60e in rocksdb::BlockBasedTableFactory::BlockBasedTableFactory(rocksdb::BlockBasedTableOptions const&) ()
#5  0x00000000011b71cd in rocksdb::ColumnFamilyOptions::ColumnFamilyOptions() ()
#6  0x0000000000526835 in __static_initialization_and_destruction_0(int, int) [clone .constprop.448] ()
#7  0x00000000014d9c3d in __libc_csu_init ()
#8  0x00007ffff7a514f1 in __libc_start_main (main=0x5a87c0 <main>, argc=1, argv=0x7fffffffe408, init=0x14d9bf0 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffe3f8) at ../csu/libc-start.c:279
#9  0x000000000053a88e in _start ()
(gdb) 

➜  avalanchego git:(master) cat /proc/cpuinfo            
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 17
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 2
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 18
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 3
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 19
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 4
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1642.357
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 4
cpu cores	: 4
apicid		: 20
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 5
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 5
cpu cores	: 4
apicid		: 21
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 6
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1405.974
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 6
cpu cores	: 4
apicid		: 22
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor	: 7
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD FX(tm)-8350 Eight-Core Processor
stepping	: 0
microcode	: 0x6000852
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 8
core id		: 7
cpu cores	: 4
apicid		: 23
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs		: fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 8036.55
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro