Adds support for Content-Encoding: gzip to in_http

[erhudy]

This change adds support for gzipped content to in_http. I discovered this by accident when attempting to send gzipped JSON from another fluent-bit, and found that fluent-bit responded with an invalid JSON error. With this PR, fluent-bit will now appropriately handle gzipped content on the HTTP input.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[erhudy]

Valgrind output:

==103116== Memcheck, a memory error detector
==103116== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==103116== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==103116== Command: bin/fluent-bit -i http -o stdout
==103116== 
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/07 22:18:08] [ info] [fluent bit] version=2.1.7, commit=02a1b386d1, pid=103116
[2023/07/07 22:18:08] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/07 22:18:08] [ info] [cmetrics] version=0.6.3
[2023/07/07 22:18:08] [ info] [ctraces ] version=0.3.1
[2023/07/07 22:18:08] [ info] [input:http:http.0] initializing
[2023/07/07 22:18:08] [ info] [input:http:http.0] storage_strategy='memory' (memory only)
[2023/07/07 22:18:09] [ info] [sp] stream processor started
[2023/07/07 22:18:09] [ info] [output:stdout:stdout.0] worker #0 started
[0] http.0: [[1688782691.385651035, {}], {"hello"=>"what"}]
[0] http.0: [[1688782720.546720821, {}], {"hello"=>"gzipped what"}]
^C[2023/07/07 22:18:49] [engine] caught signal (SIGINT)
[2023/07/07 22:18:49] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/07 22:18:50] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/07 22:18:50] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/07 22:18:50] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==103116== 
==103116== HEAP SUMMARY:
==103116==     in use at exit: 0 bytes in 0 blocks
==103116==   total heap usage: 1,709 allocs, 1,709 frees, 3,672,189 bytes allocated
==103116== 
==103116== All heap blocks were freed -- no leaks are possible
==103116== 
==103116== For lists of detected and suppressed errors, rerun with: -s
==103116== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Commands I ran to test:

non-gzipped:

curl -d '{"hello": "what"}' -H "Content-Type: application/json" http://localhost:9880

gzipped:

echo '{"hello": "gzipped what"}' | gzip > body.gz;  curl --data-binary @body.gz -H "Content-Type: application/json" -H "Content-Encoding: gzip" http://localhost:9880

Invalid content-encoding header:

echo '{"hello": "gzipped what"}' | gzip > body.gz;  curl --data-binary @body.gz -H "Content-Type: application/json" -H "Content-Encoding: gozip" http://localhost:9880

(this results in no output in the fluent-bit log and returns error: invalid 'Content-Encoding' to the caller)

[erhudy]

No config file is required to test this: I just ran bin/fluent-bit -i http -o stdout and used the default settings to test. I don't think this needs to be configurable behavior because it transparently handles gzipped input that previously failed, with no change to existing input.

[edsiper]

@erhudy would you please fix the conflict ?

[erhudy]

@erhudy would you please fix the conflict ?

Reconciled.

[edsiper]

@leonardo-albertovich pls review

[erhudy]

Anything you need from me on this PR right now? I've had this in production at my company since I made this PR and in almost 4 months it's just been doing the thing I expect it to do.

[cosmo0920]

The logic seems good but I added some concerns to proceed the PR.

[patrick-stephens]

Could you rebase @erhudy as it looks like there were some issues running actions (possibly a Github outage at the time)?

[patrick-stephens]

@leonardo-albertovich are you ok with this?

[erhudy]

Could you rebase @erhudy as it looks like there were some issues running actions (possibly a Github outage at the time)?

Rebased.

[patrick-stephens]

There we go, got unit tests running now.

[erhudy]

@patrick-stephens it looks like 2/3 macOS test suites failed. I don't see an indication that it's related to my changes, but I don't have permission to rerun just those actions.

[erhudy]

Added fluent/fluent-bit-docs#1260 for documentation change.

[cosmo0920]

I don't see an indication that it's related to my changes, but I don't have permission to rerun just those actions.

Currently, macOS tests are flaky. Feel free to ignore them. Instead, please pay attention to Ubuntu unit tests and Windows builds.

[edsiper]

thank you!

please make sure to adjust commits for further contributions: https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes