Other Transfer-Encodings, such as Gzip, Compress, Deflate, etc

[seanmonstar]

No description provided.

[reem]

Compression specifically could be left up to downstream things to implement. Not sure this is the domain of an http library.

[seanmonstar]

Looking through the spec, it does seem like its an optional thing, as opposed to chunked which the spec says a server MUST understand chunked to 1.1 compatibile.

[untitaker]

Some servers send me a gzip-compressed response without me asking for it, so practically I must always be prepared for such responses.

Unfortunately the RFC's language isn't very clear what the server is supposed to do if no Accept-Encoding header is sent. From RFC 2616:

If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding.

https://tools.ietf.org/html/rfc2616#section-14.3

From the newer RFC 7231:

A request without an Accept-Encoding header field implies that the user agent has no preferences regarding content-codings. Although this allows the server to use any content-coding in a response, it does not imply that the user agent will be able to correctly process all encodings.

https://tools.ietf.org/html/rfc7231#section-5.3.4

[untitaker]

See also https://stackoverflow.com/questions/8364640/how-to-properly-handle-a-gzipped-page-when-using-curl

[seanmonstar]

A header of "Accept-Encoding: identity" means the user agent is
specifically requesting no compression. Does that help your case?

On Sat, Jan 23, 2016, 11:22 AM Markus Unterwaditzer <
[email protected]> wrote:

See also
https://stackoverflow.com/questions/8364640/how-to-properly-handle-a-gzipped-page-when-using-curl

—
Reply to this email directly or view it on GitHub
#2 (comment).

[untitaker]

Probably, though I'd still prefer if Hyper handled this transparently for me (even if behind a featureflag).

Would setting this as a default header (on Hyper's side) be acceptable?

[seanmonstar]

I'm not so certain that this is a decision that hyper should default to. As you noticed in 7231, sending no header is valid, and implies no preference. If a user of hyper hasn't stated otherwise, I don't think we can assume a preference.

A higher level Client wrapper would probably be the place to be more opinionated, methinks.

[untitaker]

This has bitten me again. This time a different library, @SimonSapin's kuchiki provides a parse_html().from_http(url) interface, but doesn't explicitly set a header. The result is a garbage DOM.

If a user of hyper hasn't stated otherwise, I don't think we can assume a preference.

If a user of hyper hasn't stated otherwise, I don't think we can assume they'd thought about it.

[seanmonstar]

Again, does asking for identity fix the issue?

[untitaker]

Sorry, it seems to be an issue with the server in this case.

EDIT: Yes, in the previous instance, setting Accept-Encoding: identity helped.

[SimonSapin]

(@untitaker I’ve extended kuchiki’s from_http to also except an hyper::client::Response so that you can set headers and make the request yourself. In that case, the point of using it from_http instead of read_from with a Response as a generic std::io::Read is that it gets the charset from Content-Type, if any.)

[Trolldemorted]

@seanmonstar is it possible to set the headers once per client, so that we don't have do add them with every request?

[seanmonstar]

@Trolldemorted not in hyper. Such a feature does exist in reqwest, though!