Support serving of pre-GZIP encoded files

[DamianEdwards]

Pre-GZIPping files is seemingly becoming more popular. This involves running a tool ahead of deployment that creates GZIPped copies of suitable files in the site, e.g. site.js => site.js.gzip. Then the file serving aspect of the web server will serve the GZIPped file when appropriate.

[tugberkugurlu]

👍 for this one.

[nikmd23]

I'd also like to point out that this shouldn't be about "pre-gziping", but rather pre-compressing in general, no matter what the encoding format is.

For example, Microsoft and Google have both played with improved encoding schemes (peerdist and sdch). At one point Chrome even included sdch in its Accept-Encoding header. Of course these two examples are a bit crumby since neither caught on, but new compression techniques come up all the time.

What this means is, in reality, I might want to pre-compile a file into multiple formats and have the proper one selected based on the Accept-Encoding. My content directory could have several "gzip compatible" files encoded with zopfli (*.gz), the same files encoded in sdch (*.sdch) as well as the original uncompressed files for really old browsers.

I think of this as being very analogous to keeping images in both *.jpg and *.webp formats, and serving the appropriate ones at the appropriate times.

[resnyanskiy]

I found simple workaround based on URL redirect supported in HTTP:

class Startup
{
    private StaticFileOptions StaticFileOptions
    {
        get
        {
            return new StaticFileOptions
            {
                OnPrepareResponse = OnPrepareResponse
            };                
        }
    }

    private void OnPrepareResponse(StaticFileResponseContext context)
    {
        var file = context.File;
        var request = context.Context.Request;
        var response = context.Context.Response;

        if (file.Name.EndsWith(".gz"))
        {
            response.Headers[HeaderNames.ContentEncoding] = "gzip";
            return;
        }

        if (file.Name.IndexOf(".min.", StringComparison.OrdinalIgnoreCase) != -1)
        {
            var requestPath = request.Path.Value;
            var filePath = file.PhysicalPath;

            if (IsDevelopment)
            {
                if (File.Exists(filePath.Replace(".min.", ".")))
                {
                    response.StatusCode = (int)HttpStatusCode.TemporaryRedirect;
                    response.Headers[HeaderNames.Location] = requestPath.Replace(".min.", ".");
                }
            }
            else
            {
                var acceptEncoding = (string)request.Headers[HeaderNames.AcceptEncoding];
                if (acceptEncoding.IndexOf("gzip", StringComparison.OrdinalIgnoreCase) != -1)
                {
                    if (File.Exists(filePath + ".gz"))
                    {
                        response.StatusCode = (int)HttpStatusCode.MovedPermanently;
                        response.Headers[HeaderNames.Location] = requestPath + ".gz";
                    }
                }
            }
        }
    }

   public void Configure(IApplicationBuilder application)
   {
        application
            .UseDefaultFiles()
            .UseStaticFiles(StaticFileOptions)
    }
}

I used Wikipedia as reference. This approach also allows to use not minified files in development environment with no need to change links on client side.

I found that currect implementation of 'aspnet/StaticFiles' lacks of two feature:

1. Declarative setup of 'Content-Encoding' header. I suppose it can be implemented very easy - by adding second dictionary to FileExtensionContentTypeProvider.
2. Redirect support. Current design doesn't provide the ability to cancel invoke 'IHttpSendFileFeature.SendFileAsync/StreamCopyOperation.CopyToAsync'.

@davidfowl fyi.

[nikmd23]

In support of my comment above, both FireFox and Chrome will be shipping support for Brotli compression soon.

[jods4]

👍

Actually, even a good story for serving gzipped static files (with a cache, of course, since it's static compression) is currently lacking. I'm saying this in the "conventional", "IIS-kind" of way, as opposed to the build-time gzip generation.
Or did I miss something?

For those coming here from Google with dynamic content compression needs, you might try this gist:
https://gist.github.com/PinpointTownes/ac7059733afcf91ec319

[JohannesRudolph]

Would it be beyond the scope of this to suggest that StaticFiles should optionally allow cached "on-demand" compression of files? (I.e. gzip on first request and put result to file system cache, serve gzip requests from there)

[Cowlephant]

Really looking forward to being able to do this. I can go through all the trouble of bundling, minifying, tree shaking and gzipping to get things down to an absolute minimum size... but I can't serve those .js.gz files and it's problematic.

I'm also strugglin to get IIS to gzip itself as well, but that's for another area entirely.

[neyromant]

dotnet/aspnetcore#1584

https://github.com/AnderssonPeter/CompressedStaticFiles

[dfaivre]

With Angular (2) creating *.gz files when doing --prod builds, it would be great to have this out of the box.

[RehanSaeed]

Other related use cases are serving pre-brotli compressed (.br) files and also serving WebP (.webp) files instead of PNG/JPG/etc.

[Tratcher]

@JohannesRudolph you should now be able to combine ResponseCaching, ResponseCompression, and StaticFiles to achieve dynamic compression and caching of static files. @JunTaoLuo this would be a good combo to test.

[Tratcher]

Can folks clarify if they expect the original url to contain the compression extension (e.g. ".gz")? Some of the above samples do and some don't. I assume content negotiation based on accept headers would be the more general case.

[JohannesRudolph]

Great! Content-negotiation is (from my experience) the far more often-used and preferred method.

…

On Fri, Feb 17, 2017 at 2:59 PM, Chris R ***@***.***> wrote: Can folks clarify if they expect the original url to contain the compression extension (e.g. ".gz")? Some of the above samples do and some don't. I assume content negotiation based on accept headers would be the more general case. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH8NyZXuqJehp3rBf7VkFTq_XulZ8gYks5rdafHgaJpZM4CQ_LY> .

[dfaivre]

@Tratcher -- I would not expect the original URL to contain the extension.

Also, looks like the Angular CLI will no longer output the "*.gz" files, so @Tratcher's ResponseCaching and ResponseCompression solution may be sufficient in that case?

[firecube]

@Tratcher do you have an example of combining those three bits of middleware to suit this scenario?

Thank you.

[Tratcher]

@JunTaoLuo can you do a sample?

[firecube]

A small example would be really appreciated here if possible. Thanks in advance.

[JunTaoLuo]

You can take a look at the sample I have created at https://github.com/JunTaoLuo/MiddlewaresSample which uses ResponseCaching, ResponseCompression and StaticFiles to create, cache and serve different representations of the same resource.

Here's the sample output where I made 6 requests to http://localhost:5000/Foo.txt. The first was sent with no Accept-Encoding header, the second contained Accept-Encoding: gzip and the third had Accept-Encoding: custom. Then I made the first three requests again and checked that they were served from cache.

Hosting environment: Production
Content root path: C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 206.5195ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 23.3728ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 3.5304ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 7.703ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 1.4551ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 1.3005ms 200 text/plain

[Tratcher]

Hmm, no logs for the compression middleware...

[RudeySH]

@muratg why was this closed?

[joeaudette]

I came across this because I've started using webpack to pre-gzip my js and css.
@resnyanskiy your code set me on the right path, but I wanted to avoid the redirect.

The solution I came up with was implementing a custom IFileProvider based on the code from CompositeFileProvider.

I set it up like this:

app.UseStaticFiles(new StaticFileOptions()
{
    OnPrepareResponse = GzipMappingFileProvider.OnPrepareResponse,
    FileProvider = new GzipMappingFileProvider(
	loggerFactory,
	true,
	Environment.WebRootFileProvider
	)
});

it uses a convention, if the requested file ends with .min.js or .min.css it will look for the same file name with .gz on the end, and if found it returns that. Then later I got the idea to add logic to try to create the .gz file if it does not exist, and then return that on success else return the original file.

It seems to be working well, would appreciate any feedback on the implementation, found here:
https://github.com/joeaudette/cloudscribe/blob/master/src/cloudscribe.Web.Common/StaticFiles/GzipMappingFileProvider.cs

My solution is using standard .min.js and .min.css urls, but the .gz file is served, I'm not using .gz in my urls.

One known issue is that this solution is bypassing content negotiation, and just giving you gzip whether you like it or not, but really not a big issue in my view for real browsers.

[Tratcher]

That approach sounds like it would mess up the content-length and etag headers.

[joeaudette]

@Tratcher could you elaborate on how that would get messed up? The IFileProvider is passing up the IFileInfo about the gz file so it would have the correct content length of the gz file, isn't that what it should have?

We still have an opportunity to tweak the headers in OnPrepareResponse if there is something messed up, but I'm trying to understand what would be messed up and why.

[Tratcher]

Content-length and gzip are rarely used together because the implications are really confusing. I'll need to verify, but I think the content-length is supposed to be the uncompressed length rather than the compressed length.

StaticFiles also uses the length to calculate the etag, so your pre compressed file will have a different etag than the uncompressed version, even if the contents are the same.

StaticFiles also has built in support for range headers, which refer to offsets in the uncompressed file. This won't work with compressed files.

Managing pre compression in the file provider is inadequate, it needs to be built into StaticFiles to make the above scenarios work correctly (or at least be bypassed correctly).

[joeaudette]

@Tratcher that makes sense for dynamic compression, but the issue here is pre-compressed static files and I think we would want content-length for any static file wouldn't we? and it should be used to calculate the etag I would think.

In my scenario the .gz file is created by the webpack build process, while my FileProvider can be configured to generate the .gz file that is a secondary concern, mainly I am serving static files that are already gzipped.

In my scenario without content negotiation the browser is only going to get the already compressed static file, the uncompressed file would not be returned unless the compressed file does not exist and could not be created. If the source file is modified newer than the compressed file I am regenerating the compressed file.

[joeaudette]

@Tratcher so are you saying that for my pre-gzipped files I should remove the Content-Length and Accept-Range headers?

[Tratcher]

Never mind on content-length, I re-checked the spec and verified that content-length and transfer-encoding don't mix, but content-encoding is OK.

I still think the Range requests will be broken though, those offsets should be for the un-compressed representation. So yes, at least remove the Accept-Ranges header. I haven't found any spec references to confirm this yet.

[joeaudette]

@Tratcher thanks! I am by no means an expert, but I'm still a little doubtful that I should remove the Accept-Ranges:bytes header. Other samples I have found for pre-gzipped files seem to include that header.
I would think the offsets sent by the browser would mainly be to recover from partial downloads and get the rest of the file and that would be true even if the file is pre-gzipped.

I found this which seems to indicate chrome and firefox for example would store the file in its content-encoding format ie .gz and will send range requests for those files to complete a failed partial download.

https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0112.html

[Tratcher]

I expect a problem due to the differences between static and dynamic compression. With dynamic compression a range request would be processed against the un-compressed content and then the result would be compressed. With static compression the range request is processed against the compressed content. How is the client supposed to tell the difference? There's no indication in the response that you used static or dynamic compression, except that dynamic compression often uses chunked rather than content-length.

Maybe it doesn't matter so long as the behavior is consistent on a per resource basis.

[joeaudette]

@Tratcher I found this on SO, which seems to indicate content-encoding is a property of the entity whereas transfer-encoding is a property of the message, so it would seem logical if dynamic compression used transfer-encoding and pre compressed static files would use content-encoding
https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip

[Tratcher]

The edit on that answer is really telling..
"Edit: That is, unless you want to serve gzipped content to clients that only understand Content-Encoding. Which, unfortunately, seems to be most of them. But be aware that you leave the realms of the spec"

[joeaudette]

yes I saw that, you notice I said it would be logical, not that is how it is done, it sounds like in practice dynamic compression does it wrong, they should use transfer-encoding but they don't because of browsers :-D

but nevertheless I think it will be ok to keep the Accept-Ranges header
from my previous link I suspect it would work the same way with dynamic compression the offsets would be based on the compressed file not the decompressed because the dynamic compression does use content-encoding header so the client doesn't know it is dynamic. I don't think it will use offsets based on the the decompressed file, the previous link it mentions chrome and ff cache it compressed and make range requests based on the file, IE would cache the file decompressed but would only make range requests for uncompressed files, if I understood it correctly. It would seem weird if a browser used offsets that were not against the actual file it downloaded but a version transformed after download. It has to use the content-length of what it downloaded and calculate offsets based on how much of that it got already I would think.

[herecydev]

Adding my implementation into the mix. Use case, client calls /foo.js and the WebRootPath has foo.js.gz or foo.js.br and prefers the compressed versions.

public class CompressionFileProvider : IFileProvider
	{
		private readonly IFileProvider _fileProvider;
		private readonly IHttpContextAccessor _httpContextAccessor;
		private readonly string _root;

		public CompressionFileProvider(IHostingEnvironment hostingEnvironment, IHttpContextAccessor httpContextAccessor)
		{
			_fileProvider = hostingEnvironment.WebRootFileProvider;
			_httpContextAccessor = httpContextAccessor;
			_root = hostingEnvironment.WebRootPath;
		}

		public IDirectoryContents GetDirectoryContents(string subpath)
			=> _fileProvider.GetDirectoryContents(subpath);

		public IFileInfo GetFileInfo(string subpath)
		{
			if (_httpContextAccessor.HttpContext.Request.Headers.TryGetValue("Accept-Encoding", out var encodings))
			{

				if (encodings.Any(encoding => encoding.Contains("br")))
				{
					var compressedEncoding = _fileProvider.GetFileInfo(subpath + ".br");
					if (compressedEncoding.Exists)
						return compressedEncoding;
				}

				if (encodings.Any(encoding => encoding.Contains("gzip")))
				{
					var compressedEncoding = _fileProvider.GetFileInfo(subpath + ".gz");
					if (compressedEncoding.Exists)
						return compressedEncoding;
				}
			}

			return _fileProvider.GetFileInfo(subpath);
		}

		public IChangeToken Watch(string filter)
			=> _fileProvider.Watch(filter);
	}

public static class ApplicationBuilderExtensions
    {
		public static IApplicationBuilder UseCompressedStaticFiles(this IApplicationBuilder applicationBuilder, IHostingEnvironment hostingEnvironment, IHttpContextAccessor httpContextAccessor)
		{
			return applicationBuilder.UseStaticFiles(new StaticFileOptions
			{
				FileProvider = new CompressionFileProvider(hostingEnvironment, httpContextAccessor),
				OnPrepareResponse = ctx =>
				{
					var headers = ctx.Context.Response.Headers;

					if (ctx.File.Name.EndsWith(".br"))
						headers.Add("Content-Encoding", "br");
					else if (ctx.File.Name.EndsWith(".gz"))
						headers.Add("Content-Encoding", "gzip");
				}
			});
		}
    }

[aspnet-hello]

This issue was moved to dotnet/aspnetcore#2458