[improve] [broker] Servlet support response compression

[hangc0276]

Fixes #16321

Main Issue: #xyz

PIP: #xyz

Motivation

The original PR is #18102, after communicating with the author @xuesongxs, I opened a new PR to handle it.

I tested the Prometheus metric with 1000 topics and enabled the compression to get 33x data saving.

# curl -H 'Accept-Encoding: gzip' -o kk0.log.gz http://localhost:8080/metrics/ 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  927k    0  927k    0     0  1910k      0 --:--:-- --:--:-- --:--:-- 1941k
# curl -o kk1.log http://localhost:8080/metrics/  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 33.2M    0 33.2M    0     0  73.1M      0 --:--:-- --:--:-- --:--:-- 74.4M
# ll -lrt |grep kk
-rw-r--r--     1 hangc  staff   928K Nov 28 18:46 kk0.log.gz
-rw-r--r--     1 hangc  staff    33M Nov 28 18:46 kk1.log

Modifications

Add gzip compression by default and the whether the response is compressed or not is based on the client request header Accept-Encoding: gzip.
The feature will improve the metric cross-zone traffic a lot especially Pulsar broker deployed on Cloud.

This is a new feature and it will be released on the feature branch.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[hangc0276]

@asafm @lhotari Please help take a look if you have time, thanks a lot.

[lhotari]

@hangc0276 please address the review comments. it would be great to finally get this merged after a few years of reviews :)

[hangc0276]

@hangc0276 please address the review comments. it would be great to finally get this merged after a few years of reviews :)

Sure, I will address it.

all resloved

[lhotari]

Cherry-picking to branch-3.0 and branch-3.2 since the lack of compression is causing issues when metrics are enabled and there are a large amount of active topics.

[lhotari]

Btw. It looks like this change could be improved. Instead of adding multiple GzipHandler instances, it would be better to add a single instance and provide a configuration option for configuring excluded paths, just in case the compression causes problems for some cases.

[lhotari]

Early WIP PR for improvement: #22370