Use external C input plugin leads to crash on flb_config_map_set ()

[Minipada]

Bug Report

Describe the bug
I am trying to run a custom C input plugin. In my use case, I have an external C library from which i get the data and this is why I don't use the Go proxy. Based on the crash log, I suspect something in the mk library is not initialized, or allocated, it could also be a compilation flag missing related to it...but can't see what it could be.

Basing myself on the fluent-bit-plugin repository, I can build my input plugin but when running it, I encounter a crash:

mk_list_foreach(m_head, map) { is what is called at line 591, that's why I suppose something is missing.

[engine] caught signal (SIGSEGV)
#0  0x55b6ef556519      in  flb_config_map_set() at src/flb_config_map.c:591
#1  0x7fd22a669b95      in  flb_input_config_map_set() at a/install/fluent_bit_vendor/include/fluent-bit/flb_time_utils.h:595
#2  0x7fd22a669dda      in  in_test_init() at _test.c:292
#3  0x55b6ef5342b5      in  flb_input_instance_init() at src/flb_input.c:1162
#4  0x55b6ef53445e      in  flb_input_init_all() at src/flb_input.c:1216
#5  0x55b6ef56ad4c      in  flb_engine_start() at src/flb_engine.c:717
#6  0x55b6ef50d3e6      in  flb_lib_worker() at src/flb_lib.c:650
#7  0x7fd229e59b42      in  ???() at ???:0
#8  0x7fd229eeb9ff      in  ???() at ???:0
#9  0xffffffffffffffff  in  ???() at ???:0
Aborted (core dumped)

I had a conversation with Pat and hhatake on slack, see here https://fluent-all.slack.com/archives/C0CTQGHKJ/p1670454494554759 . They gave me some pointers and thanks to them, I could get a minimal example where it fails.

To Reproduce
I gathered all instructions and workspace here https://github.com/brisa-robotics/fluent-bit-bug/
For completion, I will write it here:

On host:

$ git clone git@github.com:brisa-robotics/fluent-bit-bug.git --submodules
$ docker run \
    --name fluent-bit-external-input-c-plugin \
    -v ${PWD}:/ws \
    --workdir=/ws \
    -it \
    ubuntu:jammy

In the container

$ apt update -qq
$ apt install -y cmake make g++ vim tree pkg-config libyaml-dev flex bison git libssl-dev
$ cd fluent-bit/build
$ cmake \
    -DBUILD_TESTING=Off \
    -DCMAKE_INSTALL_PREFIX=/usr \
    -DFLB_EXAMPLES=Off \
    -DFLB_SHARED_LIB=On \
    -DFLB_PROXY_GO=On ..
$ make -j
# Workaround to bypass the include that can't be found.
# This can be fixed later more permanently by adding it in the `include/CMakeLists.txt` in the fluent-bit repo.
$ cp -r /ws/fluent-bit/build/lib/monkey/include/monkey/mk_core/ /ws/fluent-bit/include/

Then let's compile our custom plugin

cd /ws/fluent-bit-plugin
mkdir build
cd build
cmake -DFLB_SOURCE=/ws/fluent-bit -DPLUGIN_NAME=in_dummy2 ../
make

And finally, running it:
```bash
/ws/fluent-bit/build/bin/fluent-bit -e ./flb-in_dummy2.so -i dummy2 -o stdout

Leading to the same error 🚀

Fluent Bit v2.0.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/12/08 13:39:07] [ info] [fluent bit] version=2.0.7, commit=fa7fd13c1f, pid=23255
[2022/12/08 13:39:07] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/12/08 13:39:07] [ info] [cmetrics] version=0.5.7
[2022/12/08 13:39:07] [ info] [ctraces ] version=0.2.5
[2022/12/08 13:39:07] [ info] [input:dummy2:dummy2.0] initializing
[2022/12/08 13:39:07] [engine] caught signal (SIGSEGV)
[2022/12/08 13:39:07] [ info] [input:dummy2:dummy2.0] storage_strategy='memory' (memory only)
#0  0x5604725d22e0      in  flb_config_map_set() at src/flb_config_map.c:591
#1  0x7fbeb82f21b8      in  ???() at ???:0
#2  0x7fbeb82f2740      in  ???() at ???:0
#3  0x7fbeb82f2b1e      in  ???() at ???:0
#4  0x5604725b00f3      in  flb_input_instance_init() at src/flb_input.c:1162
#5  0x5604725b029c      in  flb_input_init_all() at src/flb_input.c:1216
#6  0x5604725e6b13      in  flb_engine_start() at src/flb_engine.c:717
#7  0x560472589224      in  flb_lib_worker() at src/flb_lib.c:629
#8  0x7fbeb8390b42      in  ???() at ???:0
#9  0x7fbeb8421bb3      in  ???() at ???:0
#10 0xffffffffffffffff  in  ???() at ???:0
Aborted (core dumped)

Expected behavior
Load the dummy data and display it on the terminal

Your Environment

• Version used: version=2.0.7
• Configuration: -DBUILD_TESTING=Off -DCMAKE_INSTALL_PREFIX=/usr -DFLB_EXAMPLES=Off -DFLB_SHARED_LIB=On -DFLB_PROXY_GO=On
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version: Ubuntu 22 in docker
• Filters and plugins: input: dummy as shared lib, output stdout

[nokute78]

I think the struct flb_input_instance is different between fluent-bit and fluent-bit-plugin.

I tried to print the address of chunk_trace_ctxt , but it failed.

[ 50%] Building C object in_dummy2/CMakeFiles/flb-in_dummy2.dir/dummy2.c.o
/fluent-bit-plugin/in_dummy2/dummy2.c: In function 'in_dummy_init':
/fluent-bit-plugin/in_dummy2/dummy2.c:232:22: error: 'struct flb_input_instance' has no member named 'chunk_trace_ctxt'
  232 |   printf("a=%p\n", in->chunk_trace_ctxt);
      |                      ^~
make[2]: *** [in_dummy2/CMakeFiles/flb-in_dummy2.dir/build.make:76: in_dummy2/CMakeFiles/flb-in_dummy2.dir/dummy2.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:98: in_dummy2/CMakeFiles/flb-in_dummy2.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Dockerfile:

FROM ubuntu:jammy

RUN apt update -qq && apt install -y cmake make g++ vim tree pkg-config libyaml-dev flex bison git libssl-dev
RUN git clone https://github.com/fluent/fluent-bit.git && cd /fluent-bit/build && cmake .. && make
RUN cd / && git clone https://github.com/brisa-robotics/fluent-bit-plugin && cd fluent-bit-plugin && git checkout in_bug && mkdir build && cp -r /fluent-bit/build/lib/monkey/include/monkey/mk_core/ /fluent-bit/include/
RUN cd /fluent-bit-plugin/build && cmake -DFLB_SOURCE=/fluent-bit -DPLUGIN_NAME=in_dummy2 ../ && make 

[Minipada]

I used flb_input_get_property("param", in); instead of a config map to make it work. But imo it is very confusing for users, not sure it was the right way or the flb_input_instance needs a fix

[nokute78]

I sent a patch #6611 to fix this issue.