Filebeat - 64Bit alignment issue on ARM

[ikofler]

I tried to compile the latest code of filebeat on my Raspbarry Pi 3 (Arch ARM Linux, armv7h). The compilation was successful, however during runtime it crashed as follows

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0xaaf58]

goroutine 25 [running]:
panic(0x4f8a08, 0x10b22008)
/usr/lib/go/src/runtime/panic.go:500 +0x33c
sync/atomic.addUint64(0x10d501cc, 0x1, 0x0, 0x1, 0x0)
/usr/lib/go/src/sync/atomic/64bit_arm.go:31 +0x68
github.com/elastic/beats/filebeat/prospector.(*Prospector).startHarvester(0x10d50140, 0x10d26c60, 0x23, 0x0, 0x0, 0x0, 0x751cd8, 0x10d23170, 0x169c6, 0x0, ...)
/home/kofi/beats/src/github.com/elastic/beats/filebeat/prospector/prospector.go:239 +0x30c
github.com/elastic/beats/filebeat/prospector.(*ProspectorLog).scan(0x10bca0f0)
/home/kofi/beats/src/github.com/elastic/beats/filebeat/prospector/prospector_log.go:232 +0x3f4
github.com/elastic/beats/filebeat/prospector.(*ProspectorLog).Run(0x10bca0f0)
/home/kofi/beats/src/github.com/elastic/beats/filebeat/prospector/prospector_log.go:77 +0xd8
github.com/elastic/beats/filebeat/prospector.(*Prospector).Run(0x10d50140, 0x0)
/home/kofi/beats/src/github.com/elastic/beats/filebeat/prospector/prospector.go:156 +0x338
github.com/elastic/beats/filebeat/crawler.(*Crawler).Start.func1(0x10d26840, 0x0, 0x0, 0x10d50140)
/home/kofi/beats/src/github.com/elastic/beats/filebeat/crawler/crawler.go:59 +0xf4
created by github.com/elastic/beats/filebeat/crawler.(*Crawler).Start
/home/kofi/beats/src/github.com/elastic/beats/filebeat/crawler/crawler.go:60 +0x498

After some googling I found out that the 64-bit atomic must be aligned to a 64-bit address on ARM. This is however not achieved in the current Prospector struct. I tried adding a uint32 padding value before this field and now it works like a charm. As confirmed by another user who also run into that problem, another workaround is to move the uint64 value to the beginning of the struct.

[urso]

related discussion: https://discuss.elastic.co/t/64bit-alignment-issue-on-arm7-raspberry-pi/68785/6

[ruflin]

Back in the thread, @andrewkroh suggested two solutions:

Make the counter a uint32 instead of uint64, or move the counter to the first value declared in the struct.

It seems the first solution worked. Did you also try the second option? Could you share here the exact code changes you made that worked?

[urso]

The solution I'd like to see is having our own atomic package with proper types + provide a tool checking for atomic fields alignments in structures based on these: https://github.com/urso/gotools/tree/master/cmd

[spacewrangler]

@ruflin, I verified moving harvesterCounter to the first field in the struct fixed it for my case (32-bit Raspbian running on ARMv8). See: spacewrangler@1584a29. I will submit a PR for it if you like.

[ruflin]

@spacewrangler It would be great if you could open a PR with that.

[tsg]

It looks like this is also affecting the x86 architecture, so marking it as blocker for 5.2

[andrewkroh]

Looks like this issue is now fixed.

master: fd64af2 #3338
5.2: ff009ed #3364