Enable execution for multi instances by cosmo0920 · Pull Request #54 · chronosphereio/calyptia-plugin

cosmo0920 · 2023-10-10T09:32:24Z

This work is built on top of #51. So, it needs to rebase off once #51 is merged.

cosmo0920 · 2023-10-16T03:15:08Z

I'd suffered that the SEGV issue on hot-reloading. After applied the hot-reloading fix PR on the corresponding callback PR, the most of the SEGV issues are gone. Still SEGV on input_log_print_novar is persist but it becomes less occurring.

pwhelan · 2023-10-23T18:59:22Z

Now that #51 has been merged can we get this rebased for testing? I might be able to debug the crashes in the log functions.

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

To work on the test cases, we should execute them sequentialy not concurrently. Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

Without using Mutex.TryLock(), this golang interface causes the main fluent-bit process stuck on pause. This shouldn't be desired behavior. Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cosmo0920 · 2023-10-24T02:07:08Z

I rebased off the current main.

pwhelan · 2023-10-24T18:49:36Z

I ran the core-http-loader with the following configuration:

go.mod:

module github.com/calyptia/core-fluent-bit-plugin-http-loader

go 1.21.0

toolchain go1.21.1

require (
	github.com/Masterminds/sprig/v3 v3.2.3
	github.com/alecthomas/assert/v2 v2.3.0
	github.com/calyptia/api v1.4.7
	github.com/calyptia/plugin **v1.1.2-0.20231024020218-cbc15ad4bf73**
	github.com/go-playground/form/v4 v4.2.1
	github.com/itchyny/gojq v0.12.13
	github.com/peterbourgon/diskv/v3 v3.0.1
	github.com/peterhellberg/link v1.2.0
	github.com/stretchr/testify v1.8.4
	github.com/vmihailenco/msgpack/v5 v5.3.5
	golang.org/x/oauth2 v0.11.0
)

...

With a master version of fluent-bit:

Fluent Bit v2.2.0
Git commit: bf17059bf673326cbdd738b1e66f36ab51f48d3d

And with the following YAML configuration:

---
service:
    plugins_file: ./plugins.conf
    log_level: error

pipeline:
    inputs:
        - name: http_loader
          url: http://localhost:8888/test.json
          pull_interval: 1s
          data_dir: data1
          out: records
        - name: http_loader
          url: http://localhost:8888/test1.json
          pull_interval: 1s
          data_dir: data2
          out: records
        - name: http_loader
          url: http://localhost:8888/test2.json
          pull_interval: 1s
          data_dir: data3
          out: records
        - name: http_loader
          url: http://localhost:8888/test3.json
          pull_interval: 1s
          data_dir: data4
          out: records
    outputs:
        - name: stdout
          match: "*"
          format: json_lines

I then sent the SIGHUP signal various times to the fluent-bit process without any ill effect:

bash$ ./fluent-bit-master -Y -c test.yaml 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/24 15:44:16] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:16] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:16] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:16] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:17] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:17] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:17] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:17] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:18] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:18] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:18] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:18] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:19] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:19] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:19] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:19] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:20] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:20] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:20] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:20] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:21] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:21] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:21] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:21] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:22] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:22] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:22] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:22] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:23] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:23] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:23] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:23] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:23] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:23] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:23] [ info] [reload] stop everything of the old context
2023/10/24 15:44:23 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:23 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:23 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:23 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:24] [ info] [reload] start everything
[2023/10/24 15:44:24] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:24] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:24] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:24] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:25] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:25] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:25] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:25] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:25] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:25] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:25] [ info] [reload] stop everything of the old context
2023/10/24 15:44:25 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:25 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:25 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:25 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:26] [ info] [reload] start everything
[2023/10/24 15:44:26] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:26] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:26] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:26] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:27] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:27] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:27] [ info] [reload] stop everything of the old context
2023/10/24 15:44:27 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:27 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:27 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:27 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:27] [ info] [reload] start everything
[2023/10/24 15:44:27] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:27] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:27] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:27] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:28] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:28] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:28] [ info] [reload] stop everything of the old context
2023/10/24 15:44:28 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:28 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:28 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:28 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:28] [ info] [reload] start everything
[2023/10/24 15:44:28] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:28] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:28] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:28] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:29] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:29] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:29] [ info] [reload] stop everything of the old context
2023/10/24 15:44:29 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:29 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:29 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:29] [ info] [reload] start everything
[2023/10/24 15:44:29] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:29] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:29] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:29] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:30] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:30] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:30] [ info] [reload] stop everything of the old context
2023/10/24 15:44:30 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:30 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:30 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:30 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:30] [ info] [reload] start everything
[2023/10/24 15:44:30] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:30] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:30] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:30] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:31] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:31] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:31] [ info] [reload] stop everything of the old context
2023/10/24 15:44:31 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:31 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:31 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:31 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:31] [ info] [reload] start everything
[2023/10/24 15:44:31] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:31] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:31] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:31] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:32] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:32] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:32] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:32] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:32] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:32] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:32] [ info] [reload] stop everything of the old context
2023/10/24 15:44:32 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:32 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:32 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:32 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:33] [ info] [reload] start everything
[2023/10/24 15:44:33] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:33] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:33] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:33] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:34] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:34] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:34] [ info] [reload] stop everything of the old context
2023/10/24 15:44:34 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:34 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:34 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:34 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:34] [ info] [reload] start everything
[2023/10/24 15:44:34] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:34] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:34] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:34] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:35] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:35] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:35] [ info] [reload] stop everything of the old context
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:35] [ info] [reload] start everything
[2023/10/24 15:44:35] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:35] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:35] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:35] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:35] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:35] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:35] [ info] [reload] stop everything of the old context
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:35 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:36] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:36] [error] reloading in progress, aborting.
[2023/10/24 15:44:36] [ info] [reload] start everything
[2023/10/24 15:44:36] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:36] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:36] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:36] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:36] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:36] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:36] [ info] [reload] stop everything of the old context
2023/10/24 15:44:36 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:36 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:36 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:36 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:37] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:37] [error] reloading in progress, aborting.
[2023/10/24 15:44:37] [ info] [reload] start everything
[2023/10/24 15:44:37] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:37] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:37] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:37] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:38] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:38] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:38] [ info] [reload] stop everything of the old context
2023/10/24 15:44:38 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:38 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:38 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:38 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:38] [ info] [reload] start everything
[2023/10/24 15:44:38] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:38] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:38] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:38] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:39] [engine] caught signal (SIGHUP)
[2023/10/24 15:44:39] [ info] reloading instance pid=178262 tid=0x7ffff6dffac0
[2023/10/24 15:44:39] [ info] [reload] stop everything of the old context
2023/10/24 15:44:39 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:39 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:39 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:39 goroutine will be stopping: name="http_loader"
[2023/10/24 15:44:39] [ info] [reload] start everything
[2023/10/24 15:44:39] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:39] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:39] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:39] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:40] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:40] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:40] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:40] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:41] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:41] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:41] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:41] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:42] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:42] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:42] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:42] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:43] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:43] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:43] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:43] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:44] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:44] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:44] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:44] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:45] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:45] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:45] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:45] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:46] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:46] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:46] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:46] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:47] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:47] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:47] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
[2023/10/24 15:44:47] [error] Error collecting data: fetch error: do request: Get "http://localhost:8888/test3.json": dial tcp [::1]:8888: connect: connection refused
^C[2023/10/24 15:44:47] [engine] caught signal (SIGINT)
2023/10/24 15:44:47 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:47 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:47 goroutine will be stopping: name="http_loader"
2023/10/24 15:44:47 goroutine will be stopping: name="http_loader"

My configuration may not have caused the same segmentation faults because I was using multiple instances of the same plugin? @cosmo0920. If not, then it seems that the plugin is working as intended.

I recommend adding an integration test specifically testing scenarios that used to cause segmentation faults in earlier versions of the plugin, but that is out of scope for this actual pull request.

LGTM

👍

cosmo0920 · 2023-10-25T01:07:06Z

My configuration may not have caused the same segmentation faults because I was using multiple instances of the same plugin? @cosmo0920. If not, then it seems that the plugin is working as intended.

In my confirmations, I used in_gdummy plugin in multiple times. However, I adopted this PR to Golang 1.20. This could be affected for the different behavior. Thanks a lot for your confirmation.

niedbalski

LGTM

cosmo0920 force-pushed the cosmo0920-enable-execution-for-multi-instances branch 2 times, most recently from f82dab5 to ac82d67 Compare October 12, 2023 08:04

cosmo0920 added 3 commits October 24, 2023 11:02

cshared: plugin: Enable executions for multi instances

130b24e

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: plugin: Add switch for enabling multiinstances or not

e91be82

To work on the test cases, we should execute them sequentialy not concurrently. Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: Make non-blocking for checking existence of the channel

cbc15ad

Without using Mutex.TryLock(), this golang interface causes the main fluent-bit process stuck on pause. This shouldn't be desired behavior. Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cosmo0920 force-pushed the cosmo0920-enable-execution-for-multi-instances branch from ac82d67 to cbc15ad Compare October 24, 2023 02:02

cosmo0920 marked this pull request as ready for review October 24, 2023 02:07

niedbalski approved these changes Oct 25, 2023

View reviewed changes

niedbalski merged commit 8dbb935 into main Oct 25, 2023

cosmo0920 deleted the cosmo0920-enable-execution-for-multi-instances branch October 25, 2023 14:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable execution for multi instances#54

Enable execution for multi instances#54
niedbalski merged 3 commits intomainfrom
cosmo0920-enable-execution-for-multi-instances

cosmo0920 commented Oct 10, 2023

Uh oh!

cosmo0920 commented Oct 16, 2023 •

edited

Loading

Uh oh!

pwhelan commented Oct 23, 2023

Uh oh!

cosmo0920 commented Oct 24, 2023

Uh oh!

pwhelan commented Oct 24, 2023

Uh oh!

cosmo0920 commented Oct 25, 2023

Uh oh!

niedbalski left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cosmo0920 commented Oct 10, 2023

Uh oh!

cosmo0920 commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwhelan commented Oct 23, 2023

Uh oh!

cosmo0920 commented Oct 24, 2023

Uh oh!

pwhelan commented Oct 24, 2023

Uh oh!

cosmo0920 commented Oct 25, 2023

Uh oh!

niedbalski left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cosmo0920 commented Oct 16, 2023 •

edited

Loading