Add support for Brotli compression

[vbuberen]

📷 Screenshots

📄 Context

There is a very old issue with such feature request #202. Since we are now on OkHttp 4 it is time to add Brotli support.

📝 Changes

• Added new dependency OkHttp Brotli
• Updated OkHttpUtils to support uncompress operations for Brotli content

🚫 Breaking

Nothing breaking expected

🛠️ How to test

Run sample and check brotli transaction response.

⏱️ Next steps

I would like to add tests to this feature, but not sure how to do it properly to match our test suit. Would gladly discuss it here or in Slack and appreciate help/directions.

Closes #202

[MiSikora]

Great addition! I left small comments. Also, one minor thing, we should mention dependencies and the feature in the changelog.

As for the question about testing – unfortunately package that we depend on does not offer BrotliOutputStream so it is not that straightforward. I see two ways we can test this.

1. Depend on some other package for tests only and use it to compress the data like in gzip tests. I found this with some quick Google search but there might be better options.
2. Use hardcoded Brotli bytes and use them for requests and responses in tests and verify plain–text in transactions.

[MiSikora]

I wonder if we should add a list of supported compression types in ChuckerInterceptor KDoc. I'm saying that because BinaryDecoder mentions that it the body will be gunzipped but now we also understand Brotli. This way in BinaryDecoder KDoc we could just mention that body will be uncompressed and to see which compression types are supported see ChuckerInterceptor.

[vbuberen]

Good catch. We should mention in KDoc or add something in our Readme, because now it is not clear for end users that Chucker supports compressed payloads, so even having Gzip is something unclear.

[vbuberen]

Also, one minor thing, we should mention dependencies and the feature in the changelog.

🤦🏻 Forgot about it despite planning to add a few hours ago. Switched to that other issue with bodies and forgot. Will fix it.

2. Use hardcoded Brotli bytes and use them for requests and responses in tests and verify plain–text in transactions.

So far I looked at this direction due to having no classes available to create Brotli compression ourselves.
Might take inspiration from https://github.com/square/okhttp/blob/master/okhttp-brotli/src/test/java/okhttp3/brotli/BrotliInterceptorTest.kt

[MiSikora]

One more thing that we have to consider is sharing cURL. Should we add support for Brotli there as well?

[cortinico]

One more thing that we have to consider is sharing cURL. Should we add support for Brotli there as well?

Potentially yes, but I don't think is a hard blocker for this PR.

[vbuberen]

One more thing that we have to consider is sharing cURL. Should we add support for Brotli there as well?

Yes, we should add add there as well. Can do in a follow-up PR to not block other opened PRs.

[cortinico]

Can we merge this?

[vbuberen]

Can we merge this?

Not yet. Want to do a few minor updates later today. Updated the PR since decided to revive the development a little bit)

[vbuberen]

Now it can be checked and merged if everything is fine.

I decided not to test uncompression since it would mean testing brotli library, which is already tested quite well.

[MiSikora]

I decided not to test uncompression since it would mean testing brotli library, which is already tested quite well.

I think a sanity test that a Brotli compressed response is understood by Chucker would be helpful. It's easy to make a mistake during refactoring or some other changes and introduce a bug this way.

[vbuberen]

Added such test, which replicates the same test in Brotli library. @cortinico @MiSikora Please take another look.

[MiSikora]

I approved accidentally because I read test code wrongly. What I meant by test is to set Brotli encoded body in MockWebServer and then check if it is uncompressed by ChuckerInterceptor. Something similar to this.

chucker/library/src/test/java/com/chuckerteam/chucker/api/ChuckerInterceptorTest.kt

Lines 75 to 107 in e8d20b5

	@ParameterizedTest
	@EnumSource(value = ClientFactory::class)
	fun `gzipped response body is gunzipped for Chucker`(factory: ClientFactory) {
	val bytes = Buffer().apply { writeUtf8("Hello, world!") }
	val gzippedBytes = Buffer().apply {
	GzipSink(this).use { sink -> sink.write(bytes, bytes.size) }
	}
	server.enqueue(MockResponse().addHeader("Content-Encoding: gzip").setBody(gzippedBytes))
	val request = Request.Builder().url(serverUrl).build()
	val client = factory.create(chuckerInterceptor)
	client.newCall(request).execute().readByteStringBody()
	val transaction = chuckerInterceptor.expectTransaction()
	assertThat(transaction.isRequestBodyEncoded).isFalse()
	assertThat(transaction.responseBody).isEqualTo("Hello, world!")
	}
	@ParameterizedTest
	@EnumSource(value = ClientFactory::class)
	fun `gzipped response body is gunzipped for consumer`(factory: ClientFactory) {
	val bytes = Buffer().apply { writeUtf8("Hello, world!") }
	val gzippedBytes = Buffer().apply {
	GzipSink(this).use { sink -> sink.write(bytes, bytes.size) }
	}
	server.enqueue(MockResponse().addHeader("Content-Encoding: gzip").setBody(gzippedBytes))
	val request = Request.Builder().url(serverUrl).build()
	val client = factory.create(chuckerInterceptor)
	val responseBody = client.newCall(request).execute().readByteStringBody()!!
	assertThat(responseBody.utf8()).isEqualTo("Hello, world!")
	}

Dismissed due to my error. See comment for more info.

[vbuberen]

Ok, understood. Agree, makes sense to test in ChuckerInterceptorTest as well

[vbuberen]

@MiSikora I am not sure I understand how to test the case as it is done for Gzip:

chucker/library/src/test/java/com/chuckerteam/chucker/api/ChuckerInterceptorTest.kt

Lines 93 to 107 in e8d20b5

	@ParameterizedTest
	@EnumSource(value = ClientFactory::class)
	fun `gzipped response body is gunzipped for consumer`(factory: ClientFactory) {
	val bytes = Buffer().apply { writeUtf8("Hello, world!") }
	val gzippedBytes = Buffer().apply {
	GzipSink(this).use { sink -> sink.write(bytes, bytes.size) }
	}
	server.enqueue(MockResponse().addHeader("Content-Encoding: gzip").setBody(gzippedBytes))
	val request = Request.Builder().url(serverUrl).build()
	val client = factory.create(chuckerInterceptor)
	val responseBody = client.newCall(request).execute().readByteStringBody()!!
	assertThat(responseBody.utf8()).isEqualTo("Hello, world!")
	}

For Chucker it is clearly uncompressed and I added a test case for it. But for consumer - should we get the same uncompressed result with plain text there as well as it is in Gzip case? Quite confused on how to test it right, since in case of simple replication of such Gzip case I am getting that responseBody being encoded string I used initially. Not sure it should be like that. Could you help with that?

[MiSikora]

Quite confused on how to test it right, since in case of simple replication of such Gzip case I am getting that responseBody being encoded string I used initially. Not sure it should be like that. Could you help with that?

IMO that's a good thing. Gzip is built-in into OkHttp so users get decoded responses. Brotli, on the other hand, is not. So this test shows that ChuckerInterceptor does not interfere with the user's configuration and they need to install a Brotli interceptor themselves to read responses.

[vbuberen]

Understood, thanks.

So we are adding a second test which in contrary to gzip should check that the response stays untouched and still compressed with brotli, right?

[MiSikora]

Correct.