Compression enabled without client asking for it via Accept-Encoding

[lidel]

Compression should only be used if client explicitly asked for it via Accept-Encoding.

How to reproduce

• curl https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v produces with no accept-encoding and response with content-encoding: br
  • 👉 there should be no compression when accept-encoding is missing.
  • ⚠️ This behavior breaks all clients without brotli support.
• curl https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v --compressed produces request with accept-encoding: deflate, gzip, zstd and response with content-encoding: zstd

🖥️ Click to show Request-response example

$ curl https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v
* Host delegated-ipfs.dev:443 was resolved.
* IPv6: 2604:1380:45e3:7000::1
* IPv4: 147.75.49.247
*   Trying 147.75.49.247:443...
* Connected to delegated-ipfs.dev (147.75.49.247) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.delegated-ipfs.dev
*  start date: Jan  4 00:00:00 2024 GMT
*  expire date: Apr  3 23:59:59 2024 GMT
*  subjectAltName: host "delegated-ipfs.dev" matched cert's "delegated-ipfs.dev"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: delegated-ipfs.dev]
* [HTTP/2] [1] [:path: /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/2
> Host: delegated-ipfs.dev
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< content-encoding: br
< content-type: application/json
< vary: Accept-Encoding
< vary: Origin
< date: Sun, 11 Feb 2024 23:35:57 GMT
< content-length: 347
< age: 53
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< cache-control: max-age=60
< x-cache-status: HIT
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

Why this is a problem

Among other things:

• breaking all clients without brotli support 🙈
• breaking setups without explicit ask for compression, where plain text json was expected as output (parsing blob as plain text)
• bad dx: it is impossible to get plain text output via basic curl unless --compressed is explicitly passed

Is this a someguy bug?

No, compression is opt-in and works correctly when using it directly:

🖥️ Click to show Request-response example

# without compression: plaintext works fine
$ curl http://127.0.0.1:8091/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy  -v
*   Trying 127.0.0.1:8091...
* Connected to 127.0.0.1 (127.0.0.1) port 8091
> GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/1.1
> Host: 127.0.0.1:8091
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< Vary: Origin
< Date: Mon, 12 Feb 2024 00:06:12 GMT
< Content-Length: 841
<
* Connection #0 to host 127.0.0.1 left intact
{"Providers":[{"Addrs":["/ip6/::1/udp/4001/quic-v1","/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/127.0.0.1/tcp/4001","/ip4/127.0.0.1/udp/4001/quic-v1","/ip4/192.168.50.102/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/udp/4001/quic-v1","/ip6/::1/tcp/4001","/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/tcp/4001"],"ID":"12D3KooWBdmLJjhpgJ9KZgLM3f894ff9xyBfPvPjFNn7MKJpyrC2","Schema":"peer"},{"ID":"12D3KooWQgrExcg6dkCdiTER3G3ARe14PZ4cLhinKtRcLHsvnk1Q","Schema":"peer"}]}%                                                                                                       

# opt-in compression: also works fine
$ curl http://127.0.0.1:8091/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v --compressed
*   Trying 127.0.0.1:8091...
* Connected to 127.0.0.1 (127.0.0.1) port 8091
> GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/1.1
> Host: 127.0.0.1:8091
> User-Agent: curl/8.5.0
> Accept: */*
> Accept-Encoding: deflate, gzip, zstd
>
< HTTP/1.1 200 OK
< Content-Encoding: zstd
< Content-Type: application/json
< Vary: Accept-Encoding
< Vary: Origin
< Date: Mon, 12 Feb 2024 00:06:31 GMT
< Content-Length: 389
<
{"Providers":[{"Addrs":["/ip4/192.168.50.102/udp/4001/quic-v1","/ip6/::1/tcp/4001","/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/46.21.210.45/tcp/27897","/ip4/46.21.210.45/udp/27897/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/127.0.0.1/udp/4001/quic-v1","/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/tcp/4001","/ip4/192.168.50.102/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip6/::1/udp/4001/quic-v1","/ip4/46.21.210.45/udp/27897/quic-v1","/ip4/127.0.0.1/tcp/4001"],"ID":"12D3KooWBdmLJjhpgJ9KZgLM3f894ff9xyBfPvPjFNn7MKJpyrC2","Schema":"peer"},{"ID":"12D3KooWQgrExcg6dkCdiT* Connection #0 to host 127.0.0.1 left intact
ER3G3ARe14PZ4cLhinKtRcLHsvnk1Q","Schema":"peer"}]}%

[lidel]

@ns4plabs assigning to you because I confirmed this is not a problem with someguy, this seems to be an infra thing (either reverse proxy is ignoring vary: Accept-Encoding, adding own compression, or maybe we had old responses cached and need to purge them?)

[ns4plabs]

Confirmed that this is indeed an issue with the reverse proxy. It wasn't processing the content-encoding headers correctly despite having the option turned on to take into account vary: Accept-Encoding (Ref.: https://docs.haproxy.org/2.9/configuration.html#6.2.1-process-vary)

For now I disabled caching in HAProxy until a better solution is put in place - https://github.com/ipshipyard/waterworks-infra/commit/da3cc18bdbd349a3008405bb355ce50e02168688