Doc: gzip support in RestTemplate [SPR-7874]

[spring-projects-issues]

Stephan Oudmaijer opened SPR-7874 and commented

The org.springframework.web.client.RestTemplate does not support gzip encoding. Please add an option to add headers to the request or provide built-in support for gzip encoding.

Now I have added a hack in my own GzipRestTemplate version:

public class GzipRestTemplate extends RestTemplate {

   @Override
    protected <T> T doExecute(URI url, HttpMethod method, RequestCallback requestCallback, ResponseExtractor<T> responseExtractor) throws RestClientException {
        Assert.notNull(url, "'url' must not be null");
        Assert.notNull(method, "'method' must not be null");
        ClientHttpResponse response = null;
        try {
            ClientHttpRequest request = createRequest(url, method);
            if( request.getHeaders() != null ) {
                request.getHeaders().add("Accept-Encoding", "gzip,deflate"); // <- hack
            }

---

Affects: 3.0.5

Referenced from: commits 3d645cf

8 votes, 13 watchers

[spring-projects-issues]

Michel Zanini commented

GZip support was added in Android Rest Template, you guys could backport it from there.

[spring-projects-issues]

Michel Zanini commented

You can port this implementation from this classes:

https://github.com/SpringSource/spring-android/blob/master/spring-android-rest-template/src/main/java/org/springframework/http/client/AbstractClientHttpRequest.java
https://github.com/SpringSource/spring-android/blob/master/spring-android-rest-template/src/main/java/org/springframework/http/client/AbstractClientHttpResponse.java

[spring-projects-issues]

Andrew Swan commented

This improvement isn't just about adding the Accept-Encoding header; that's easily done using the existing API, by changing this:

Foo myFoo = restOperations.getForObject(myUrl, Foo.class);

.. to this:

HttpHeaders headers = new HttpHeaders();
headers.set("Accept-Encoding", "gzip,deflate");
HttpEntity requestEntity = new HttpEntity(headers);
Foo myFoo = restOperations.exchange(myUrl, GET, requestEntity, Foo.class).getBody();

No, the main improvement we need is for RestTemplate to be able to detect a compressed response and decompress it using the correct algorithm.

[spring-projects-issues]

Andrew Swan commented

Thanks for getting onto this, guys. Can I request please that the scope of this ticket includes support for both gzip and deflate compression?

[spring-projects-issues]

Balint Kurnasz commented

Hi guys, a simple solution can be applied without backporting anything. Simply implement a proxy message converter and use that. How i did this is something like:
note that the interest thing is that 3 little line in read() method

/**
 * A simple implementation of the {@link HttpMessageConverter} interface which has the ability to determine that a given
 * response is compressed with Gzip or not. If the response is compressed a new HttpInputMessage will be delegated to
 * the underlying converter. That HttpInputMessage contains a {@link GZIPInputStream} rather than a regular InputStream
 *
 * @author balint
 * @version 1.0
 * @see GZIPInputStream
 * @since 07 08 2013
 */
public class GzipHttpMessageConverterProxy implements HttpMessageConverter<Object> {
    /** Original converter */
    private HttpMessageConverter<Object> converter;

    /**
     * Initialize a new GzipHttpMessageConverterProxy for a given HttpMessageConverter.
     *
     * @param converter converter to be proxied
     */
    @SuppressWarnings("unchecked")
    public GzipHttpMessageConverterProxy(HttpMessageConverter<?> converter) {
        super();
        Assert.notNull(converter);
        this.converter = (HttpMessageConverter<Object>) converter;
    }

    /**
     * {@inheritDoc}
     *
     * @param clazz     the class to test for readability
     * @param mediaType the media type to read, can be null if not specified. Typically the value of a Content-Type header.
     * @return true if data can be read otherwise false
     */
    @Override
    public boolean canRead(Class clazz, MediaType mediaType) {
        return converter.canRead(clazz, mediaType);
    }

    /**
     * {@inheritDoc}
     * Delegates through
     *
     * @return delegated method return value
     */
    @Override
    public boolean canWrite(Class clazz, MediaType mediaType) {
        return converter.canWrite(clazz, mediaType);
    }

    /**
     * {@inheritDoc}
     * Simply delegates to the given converter
     *
     * @return delegated method return value
     */
    @Override
    public List<MediaType> getSupportedMediaTypes() {
        return converter.getSupportedMediaTypes();
    }

    /**
     * {@inheritDoc}
     * In addition this method checks if the response is compressed or not. If it is, a new {@link HttpInputMessageProxy HttpInputMessageProxy}
     * HttpInputMessage will be delegated to the proxying converter otherwise a simple delegation is invoked
     *
     * @return delegated method return value
     */
    @Override
    public Object read(Class clazz, HttpInputMessage inputMessage) throws IOException, HttpMessageNotReadableException {
        HttpInputMessage inputMessageProxy = inputMessage;

        if (isCompressed(inputMessage.getHeaders())) {
            inputMessageProxy = new HttpInputMessageProxy(new GZIPInputStream(inputMessage.getBody()), inputMessage.getHeaders());
        }
        return converter.read(clazz, inputMessageProxy);
    }

    /**
     * {@inheritDoc}
     * <p/>
     * Delegates through
     */
    @Override
    public void write(Object o, MediaType contentType, HttpOutputMessage outputMessage) throws IOException, HttpMessageNotWritableException {
        converter.write(o, contentType, outputMessage);
    }

    /**
     * Determine if the request/response header has gzip compression set. If GZIP enabled true will be returned otherwise false
     *
     * @param headers headers to check for the compression
     * @return true if compressed otherwise false
     */
    private boolean isCompressed(HttpHeaders headers) {
        for (String headerKey : headers.keySet()) {
            final List<String> headerValues = headers.get(headerKey);
            if (headerValues.contains("gzip")) {
                return true;
            }
        }
        return false;
    }

    /**
     * Simple proxy class which proxies a given input stream and a given HttpHeaders object. This class is used when the
     * original HttpInputMessage has been changed to delegate a new {@link GZIPInputStream} to the underlying converter.
     *
     * @see HttpInputMessage
     */
    private class HttpInputMessageProxy implements HttpInputMessage {
        private InputStream is;
        private HttpHeaders headers;

        private HttpInputMessageProxy(InputStream is, HttpHeaders headers) {
            this.is = is;
            this.headers = headers;
        }

        @Override
        public InputStream getBody() throws IOException {
            return is;
        }

        @Override
        public HttpHeaders getHeaders() {
            return headers;
        }
    }
}

And you can use it like this:

List<HttpMessageConverter<?>> messageConverters = new ArrayList<>(restTemplate.getMessageConverters());

restTemplate.getMessageConverters().clear();
for (HttpMessageConverter<?> messageConverter : messageConverters) {
    restTemplate.getMessageConverters().add(new GzipHttpMessageConverterProxy(messageConverter));
}

With this I am able to achieve a solution which makes it possible to ship my library without any custom dependency and deep modifications in spring fw. You can improve this as you want, think this is a clear OO solution with the Proxy design pattern.

[spring-projects-issues]

Juergen Hoeller commented

Arjen, what's your take on this? Seems easy enough to do for 4.0 RC1 still...

[spring-projects-issues]

Arjen Poutsma commented

I would prefer to solve it at the ClientHttpRequest/Response level.

That said, I don't think we need to write this feature ourselves, since Apache HttpClient already supports it via the DecompressingHttpClient. Adding some documentation about this wrapper class would be enough IMO.

[spring-projects-issues]

Arjen Poutsma commented

Fixed by adding documentation on DecompressingHttpClient.

[spring-projects-issues]

John Mark commented

I have tried using the DecompressingHttpClient as Arjen proposed, but it does not seem to work properly with RestTemplate's jackson message converter (MappingJackson2HttpMessageConverter). The message converter appears to be receiving the gzip'ed bytes and does not know what to do with them.

[spring-projects-issues]

John Mark commented

Ignore my earlier comment, RestTemplate is able to handle everything nicely with using Apache's DecompressingHttpClient. I had been trying to manually set the Accept-Encoding parameter and that was messing it up. Turns out Apache handles everything automatically.

I should also add that DecompressingHttpClient has been deprecated in HttpComponents 4.3. HttpClientBuilder (which by default supports compression) should be used instead.