Add option to not lowercase headers

[hissssst]

Current behaviour:
Headers are always translated to lowercase

Desired behaviour:
Have connection option to not translate request and response headers to lowercase in HTTP 1.x

This option would be useful to trick passive http-client fingerprinting
I can implement the PR

[whatyouhide]

I am open to supporting this at the connection or at the request level, I think. @ericmj thoughts?

[ericmj]

If the only use case is avoid fingerprinting I am not sure it is worth it. There are likely plenty of other ways to detect mint and it does not seem feasible to avoid all of them.

[chulkilee]

Related discussion: #172

Things to clarify

• request headers / response headers / both headers
• it only applies to HTTP 1.x (since HTTP 2 requires lowercase header)

[hissssst]

@ericmj
You may be right but as far as I know, HTTP 1.x fingerprinting (in p0f or nmap) is done by

1. User-Agent header
2. Order of headers
3. Case of headers
4. Special headers (e.g DNT)

[ericmj]

We send some headers in the same order as well.

Can you explain your use-case for wanting to avoid finger printing?

[hissssst]

@ericmj
Some big social media websites do fingerprinting to distinguish mobile agents from pc agents. For example https://vk.com blocks some features for mobile agents. I am trying to write crawler for this website and I am always being redirected to the mobile version with crawler on mint. Raw netcat requests work fine and I think the problem is with user-agent fingerprinting

[whatyouhide]

You can customize the user-agent header. Did you try that?

[hissssst]

@whatyouhide
Yes I've tried. I've copy-pasted request from firefox
Headers in code

  [
    {"User-Agent", "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"},
    {"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"},
    {"Accept-Language", "en-US,en;q=0.5"},
    {"Connection", "keep-alive"},
    {"Referer", "https://vk.com/feed"},
    {"Cookie", "cookies"},
    {"Upgrade-Insecure-Requests", "1"},
    {"Sec-Fetch-Dest", "iframe"},
    {"Sec-Fetch-Mode", "navigate"},
    {"Sec-Fetch-Site", "same-origin"},
    {"TE", "Trailers"}
  ]

received request

GET / HTTP/1.1
host: localhost:4000
user-agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
connection: keep-alive
referer: https://vk.com/feed
cookie: cookies
upgrade-insecure-requests: 1
sec-fetch-dest: iframe
sec-fetch-mode: navigate
sec-fetch-site: same-origin
te: Trailers

[whatyouhide]

So we don't change the order of headers and set the user-agent correctly, so it seems like the only difference would be the downcasing.

[ericmj]

We have a specific order of headers when mint sets them itself. For example: host, content-length, transfer-encoding etc.

@hissssst I am hesitant to make these kind of changes because it will always be a back and forth between services finding new ways of detecting clients and clients trying to circumvent them. We still don't know if the changes you suggested will actually prevent vk.com from detecting mint.

[hissssst]

@ericmj

specific order of headers

That is right but these headers can be set explicitly

I understand you hesitation but I already have the working fork which prooves my words. I also argee that this option would be useless for 99% of users.
I think that this issue can be closed

[JacquiManzi]

I also agree that this option would be useless for 99% of users.

Just chiming in to add that if someone switches to Mint from a library that previously did not modify the casing of headers, they will inadvertently break backwards compatibility for users that might be relying on headers being case-insensitive. This could have serious implications for anyone switching to Mint that might not be expecting lower-case headers for HTTP1.

[hissssst]

Reopening this issue for case of @JacquiManzi

[ericmj]

Changing libraries is indeed a backwards incompatible action. It is not the goal of Mint to be compatible with pre-existing HTTP clients, especially if you are relying on behavior that does not follow the HTTP specification.

If you can give examples on applications, services, APIs, etc. that require case-sensitive headers then we can discuss around them specifically. I will reopen if we have such examples to discuss.

[JacquiManzi]

especially if you are relying on behavior that does not follow the HTTP specification.

I might be misunderstanding what you mean by this, but the specification for HTTP/1.1 headers are case-insensitive.

[whatyouhide]

Just chiming in to add that if someone switches to Mint from a library that previously did not modify the casing of headers, they will inadvertently break backwards compatibility

@JacquiManzi what @ericmj means is that if you switch HTTP library, you shouldn't expect backwards compatibility to be maintained. You should adapt your code to use the new library and not just drop it in and expect it to behave exactly the same, if that makes sense.

the specification for HTTP/1.1 headers are case-insensitive.

Which is exactly why we lowercase them and it shouldn't make a difference, because the spec is case-insensitive.

Hopefully this makes things clearer.

[JacquiManzi]

@whatyouhide that makes sense, thank you for the clarification.

[PragTob]

Hey there @whatyouhide @ericmj - first thanks for this library & everything, as always! 💚

We're currently running into a similar issue - in our case, it's a 3rd party we don't control though that treats the headers as case sensitive. In particular, it's a let's say big/old organization we need to integrate with and a required header from them that is lowerCamelCase.

We'd be happy to provide an MR implementing this (given some guidance on what level you feel would be most appropriate for this) if you were considering allowing this change. Just an option to opt out of the lower casing behavior if one needs to.

[whatyouhide]

I would be okay with an option to do this, provided we make it clear it should not be used (because it goes against the spec). Something like case_sensitive_headers_that_violate_http_spec: true. @ericmj you ok with that?

[PragTob]

Your call, I'd rather leave the _that_... part for the docs and not the option name but your call 😁

[PragTob]

@ericmj small ping :)

[sneako]

Should there be separate options for requests and responses? For example, I know the headers my application sends will be valid (and it could possibly be a minor perf boost to skip downcasing), but I would probably still want to make sure response headers are downcased

[PragTob]

My goal would be to pass it on request initiation explicitly, so it'd only affect requests sent out.

[ericmj]

Unfortunately if we add a case_sensitive_headers option we would have to deal with additional complexity in every place we do a header lookup and possibly take a performance penalty by doing extra unnecessary down casings. There are lots of places we lookup headers by their exact name that would no longer work if headers are not lower case:

• https://github.com/elixir-mint/mint/blob/main/lib/mint/core/util.ex#L121
• https://github.com/elixir-mint/mint/blob/main/lib/mint/http1.ex#L975
• https://github.com/elixir-mint/mint/blob/main/lib/mint/http1.ex#L990-L1014
• https://github.com/elixir-mint/mint/blob/main/lib/mint/http1.ex#L1022
• https://github.com/elixir-mint/mint/blob/main/lib/mint/http1.ex#L849-L869

I'm not sure we should add this complexity to handle the rare case of servers not following the specification.

I think a better alternative would be to add an option that forces the casing of headers or adds a rewrite rule/function for headers. It would be self contained to only where we encode headers and wouldn't have a performance penalty for users that don't need it.

encode_headers_camel_case: boolean()
rewrite_header_name: (String.t() -> String.t()) | nil

[whatyouhide]

@ericmj what if we keep the original headers around alongside the downcased headers? I think using the downcased headers in Mint itself is the way to go, but we could hand over the original headers to the user, at least for responses.

[ericmj]

Is there a need to do this for response headers? Unless Mint is used for proxying I don't see a need for it.

We could do it for request headers but I would prefer to rewrite at encoding time than keeping both the downcase and original header names everywhere. Not a strong opinion but if we are doing things outside of the specification I would prefer to keep it as contained as possible.

[whatyouhide]

@ericmj the problem with rewriting itself is that you don't really have a way to know what the server sent, for example if you're implementing a proxy. And yes, for response headers, that would be needed if you're using Mint for proxying or for writing some weird tool that tries to break servers or whatnot, who knows. I think Mint is low-level enough that it makes sense to keep things compliant with the spec, but also allow users to do these non-conforming things when they want.

[ericmj]

the problem with rewriting itself is that you don't really have a way to know what the server sent, for example if you're implementing a proxy

Why is this a problem, the rewriting function will receive the header name and know what the server sent?

And yes, for response headers, that would be needed if you're using Mint for proxying or for writing some weird tool that tries to break servers or whatnot, who knows

Do we need this for response headers? It seemed like @PragTob and @hissssst only needed it for request headers. If there is no user need for it then I don't think we should implement it, specially since it's for something outside the specification.

[ericmj]

To clarify the proposal for this options was only for the request headers.

rewrite_header_name: (String.t() -> String.t()) | nil

[hissssst]

I am here to confirm that I needed this option only for request headers, but I think that this option could be useful for response headers too.

Case of the headers is one of many factors accountable for application fingerprinting.

[whatyouhide]

@ericmj and I talked about this and we decided that yes, having an option to not lowercase request headers is something we want in Mint.

Let's call the option case_sensitive_headers_that_violate_http_spec: true | false, using false by default. Even if we set the option to true, we'll want to still lowercase headers and use that for all comparisons (such as setting content-length if not present, and so on).

Anybody willing to work on a PR for this? 😊

[wojtekmach]

Just some additional feedback, In Req v0.4, I'm automatically downcasing all header names. Furthermore, functions like Req.Request.get_header/2, Req.Request.put_header/3, etc do that too.

So unless someone does something really custom (and manually updates the underlying request.headers map. wink wink), Req will send all header names as downcase so I think it would be nice to have an option not to do the downcase again in Mint.

[whatyouhide]

@wojtekmach we'd love some help to bring #399 over the finish line in case you have some bandwidth 😉

[ericmj]

Implemented in #399.