out_http: amd64.c:121: crash: Assertion `0' failed.

[edsiper]

Reported on #429, we will track this problem here:

[2018/04/07 01:44:14] [ info] [out_http] HTTP STATUS=200
[2018/04/07 01:44:15] [ info] [out_http] HTTP STATUS=200
fluent-bit: /tmp/src/lib/monkey/deps/flb_libco/amd64.c:121: crash: Assertion `0' failed.
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

[engine] caught signal
#0  0x7f6defdfa529      in  ???() at ???:0
#1  0x7f6defdf1e66      in  ???() at ???:0
#2  0x7f6defdf1f11      in  ???() at ???:0
#3  0x55994d80c389      in  ???() at ???:0
#4  0xffffffffffffffff  in  ???() at ???:0

This problem is associated to the bad use of co-routines implementation, likely there is an unexpected and explicit return from a co-routine that trigger this problem.

cc: @StevenACoffman @jgsqware @onorua

[edsiper]

@StevenACoffman @jgsqware @onorua

are you facing this crash only when the HTTP server is enabled ?

[onorua]

I did not try to disable HTTP server to be honest, as we use it for metrics exposure.

[StevenACoffman]

@edsiper We had not noticed excessive restarts when the liveness and readiness probes were omitted previously. I will disable the HTTP Server, remove the prometheus annotations, the liveness and readiness probes and start over, and let it run overnight.

[edsiper]

thanks. I was able to track down the issue in 0.13-dev, indeed the problem is in the HTTP Server, a fix will be available shortly.

note: If you see a restart when the HTTP server is off, that's a separate problem.

[jgsqware]

Yes we have Http Server setup, and we use the dev version for this purpose. Le mar. 10 avr. 2018 à 04:26, Eduardo Silva <notifications@github.com> a écrit :

…

thanks. I was able to track down the issue in 0.13-dev, indeed the problem is in the HTTP Server, a fix will be available shortly. note: If you see a restart when the HTTP server is off, that's a separate problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#557 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJrq-1ACsMLmAlHI-XYirHoGkEdUJR0ks5tnBhRgaJpZM4TLGmM> .

[edsiper]

I've fixed the issues associated to the HTTP Server that generated the crash. All please upgrade to the following Docker image:

• fluent/fluent-bit-0.13-dev:0.16

[edsiper]

FYI: https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev

[StevenACoffman]

@edsiper FYI After 24 hours running fluent/fluent-bit-0.13-dev:0.15 without the http server, I only had 4 restarts on a single pod, and one on another. The other 20 pods had no issues. Not the same issue as what you fixed, but if you are curious, all 5 restarts were this error:

[2018/04/10 03:27:50] [ info] [out_http] HTTP STATUS=200
[engine] caught signal
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

#0  0x7ffb141f2e63      in  ???() at ???:0
#1  0x55de3bed7aec      in  ???() at ???:0
#2  0x55de3c0e7415      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890
#3  0x55de3c0e7415      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:72
#4  0x55de3bed84ce      in  ???() at ???:0
#5  0x55de3bed8d1c      in  ???() at ???:0
#6  0x55de3be7b3da      in  ???() at ???:0
#7  0x55de3be78e2c      in  ???() at ???:0
#8  0x55de3be7a98c      in  ???() at ???:0
#9  0x55de3be9e99d      in  ???() at ???:0
#10 0x55de3be9f7e8      in  ???() at ???:0
#11 0x55de3be9d34c      in  ???() at ???:0
#12 0x55de3bea3721      in  ???() at ???:0
#13 0x55de3be7aecb      in  ???() at ???:0
#14 0x55de3be81a86      in  ???() at ???:0
#15 0x55de3be21ef3      in  ???() at ???:0
#16 0x7ffb140ea2e0      in  ???() at ???:0
#17 0x55de3be20449      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

I have just applied the fluent/fluent-bit-0.13-dev:0.16 changes and will monitor it and let you know.

[StevenACoffman]

@edsiper fluent/fluent-bit-0.13-dev:0.16 with HTTP_Server On is in CrashLoopBackOff for all applied containers:

[2018/04/10 20:57:33] [ info] [out_http] HTTP STATUS=200
[2018/04/10 20:57:33] [ info] [out_http] HTTP STATUS=200
[engine] caught signal
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

#0  0x55a212d8e727      in  __mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:87
#1  0x55a212d8e75e      in  mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:93
#2  0x55a212d8eb64      in  mk_http_thread_purge() at lib/monkey/mk_server/mk_http_thread.c:197
#3  0x55a212d8e8e6      in  thread_cb_init_vars() at lib/monkey/mk_server/mk_http_thread.c:104
#4  0x55a212d995e6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#5  0xffffffffffffffff  in  ???() at ???:0

Some error immediately, some run for a minute, but then they all crash with the same error.