Spring adding 'chunked' transfer encoding even if this header already exists [SPR-16985]

[spring-projects-issues]

Pradeep Gadi opened SPR-16985 and commented

In our Middleware project we forward rest response from another service as it is to angular UI. The response we are getting to the Middleware already having transfer-encoding →chunked header. while returning the same response from our middleware to angular UI spring adding transfer-encoding again. see below response

content-type →application/json;charset=UTF-8
date →Fri, 29 Jun 2018 07:02:06 GMT
transfer-encoding →chunked, chunked

 

Because of this F5 not able to send the response properly. Spring should not add transfer-encoding If this header already present in the response.

 

---

Affects: 5.0.5

Issue Links:

• HTTP Response should not contain both Transfer-Encoding and Content-Length headers [SPR-15212] #19776 HTTP Response should not contain both Transfer-Encoding and Content-Length headers

[spring-projects-issues]

Juergen Hoeller commented

From a quick review, I can't find any place where we explicitly set the "Transfer-Encoding" header in Spring's processing, in particular not to "chunked". Does this maybe come through the HTTP server or some HTTP client library here?

Brian Clozel, assigning this to you for further consideration...

[spring-projects-issues]

Pradeep Gadi commented

I have added a filter and tested the same, this issue is not because of the Httpserver. I am not sure how to figure it out whether the issue is with HTTP client or Spring. Could you please provide me any suggestion on how to proceed with this issue.

[spring-projects-issues]

Brian Clozel commented

Hi Pradeep Gadi,

Could you provide us with a sample application? I can't seem to reproduce this at the moment.

[BSchwetzel]

We stumbled upon this just yesterday and opened an issue against traefik because of this (traefik/traefik#8060). Basically what we do is using a Spring Boot application as a proxy, as we don't want to deal with CORS and Same-Origin-Policy and all that stuff. Thankfully, our test case for this is rather simple:

package org.my-organization.test;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.client.RestTemplate;

@Controller
@RequestMapping("/status")
public class StatusController {

    private RestTemplate restTemplate;

    @Autowired
    public StatusController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping(value = "/test")
    public ResponseEntity getStatus() {
        return restTemplate.getForEntity("http://another-spring-boot-service:8080/actuator/health", String.class);
    }
}

Now if we curl that controller:

$  curl -v http://localhost:8080/status/test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
> GET /status/test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.75.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Transfer-Encoding: chunked
< Date: Wed, 14 Apr 2021 08:14:11 GMT
< Keep-Alive: timeout=60
< Connection: keep-alive
< Content-Type: application/json
< Transfer-Encoding: chunked
<
{ [20 bytes data]
100    15    0    15    0     0     78      0 --:--:-- --:--:-- --:--:--    78{"status":"UP"}
* Connection #0 to host localhost left intact

As you can see we have Transfer-Encoding present twice, which breaks traefik (by breaking golang's net/http it seems: https://github.com/golang/go/blob/59bfc18e3441d9cd0b1b2f302935403bbf52ac8b/src/net/http/transfer.go#L634-L637)

We are using Spring Boot 2.4.4 / Spring 5.3.5

[bclozel]

Hi @BSchwetzel

In the meantime, we received out-of-band feedback on issues similar to yours. I'll try to summarize our findings in this comment.

As far as I understand, golang introduced this restriction in the http layer to prevent HTTP smuggling attacks - but this can only apply to requests received by servers/proxies, not responses sent by servers. So while this change is enforcing a new restriction, it's not meant to protect you here. This change not only applies to traefik, but to many proxies/servers using golang.

We've only found that problem in Spring applications with the exact use case you're describing: a controller uses an HTTP client to retrieve a response and returns it as the response of the controller handler method without any modification. We consider this use case as problematic for several reasons:

1. Spring provides a way to set response headers using ResponseEntity, but does not enforce HTTP validation here, since it's the job of the web server. Some server implementations are more lenient than others and might reject or ignore combinations. In this case, servers like Tomcat or Netty might choose to enforce stricter rules as well in the future (see Refuse adding invalid HTTP 2.0 headers apache/tomcat#277 and https://bz.apache.org/bugzilla/show_bug.cgi?id=64144, for example). Some Servlet containers are already "fixing" the response for you if it contains both Transfer-Encoding and Content-Length.

2. This is not specific to Spring, as you could do the same thing with the HTTP Servlet response directly - in that case, Spring would have no way of enforcing rules or cleaning the response.

3. Applications are not in a position to set the Transfer-Encoding response header because it has not the necessary visibility to set that correctly, nor the ability to then send a correctly formatted chunked body.

So while sending duplicate chunked values are illegal anyway, we don't think that Spring is in a position to fix that. It's up to servers to decide what they want to do with this.

Now we've had similar discussions with developers using the same usage pattern (controllers proxying requests). The chunked header value is a symptom of a much bigger problem: copying headers in and out can cause many HTTP and security issues. HTTP headers can carry information about the state of the connection, security restrictions or options negotiated between client and server. This implementation is a typical example of possible HTTP smuggling vulnerabilities and I'd encourage you to consider this in your application.

At a minimum, headers should be properly filtered (in and out). I understand that we're probably looking at a very restricted use case, but implementing a general purpose proxy/gateway is not easy and there are many more problems with that. Maybe Spring Cloud Gateway can be helpful.

[anwarmd]

We are also seeing same issue, "Transfer-Encoding: chunked" in response and using following SPring boot version 2.5.2.

1. How you fixed this issue and what Filter used?
2. Any one tried out Spring Cloud Gateway and is it working without chunked issue?

[bclozel]

@anwarmd this issue is about getting duplicate Transfer-Encoding: chucked response headers. If you're seeing this problem, this is very likely due to your application setting those headers directly in a ResponseEntity returned by a controller: this should not be done by the application, you'll need to fix the problem in your codebase.

[anwarmd]

@bclozel Can you please suggest me on

1. How can we remove one of the header ?
2. Can we override any filter to remove one of the headers ?
3. We might be getting from downstream Services and any period of time we would like to make it one encoding ?
4. What context does spring add encoding ?