Implement brotli compression data-format for faster loading of websites.

[nmabhinandan]

References:

• Introducing Brotli: a new compression algorithm for the internet - Google OpenSource blog
  http://google-opensource.blogspot.fr/2015/09/introducing-brotli-new-compression.html
• Intent to Ship: Brotli (Accept-encoding: br on HTTPS connection) - Chromium mailing-list
  https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/brotli/blink-dev/JufzX024oy0/WEOGbN43AwAJ
• Brotli IETF draft proposal
  https://www.ietf.org/id/draft-alakuijala-brotli-08.txt
• Brotli reference implementation by Google
  https://github.com/google/brotli

[mholt]

Yeah, brotli looks cool. Waiting for a good pure Go library that implements brotli.

[mholt]

Going to mark this "good for newbies" since implementing an algorithm from spec is a great exercise for anyone who knows Go.

Here's one that was started but is incomplete: https://godoc.org/github.com/dsnet/compress/brotli

[humboldtux]

I would like to work on this.

[mholt]

Awesome, let the author know and it looks like this would be a good place to start: https://github.com/dsnet/compress/blob/master/brotli/writer.go

[humboldtux]

Which author, @mholt ?

[mholt]

I mean @dsnet ;)

[dsnet]

Hello.

The current state of my brotli package is that the Reader is 100% functional. There are still many performance tweaks to be made, but I am not aware of any bugs that make it non-compliant with the RFC draft. In fact, as I was implementing it a few months back, a few PRs were made to the draft to clarify certain points.

As for the Writer, there is still work to be done. I have completed an implementation of a bit writer and a entropy encoder. The major remaining piece, is writing a LZ77-style string matcher. For that reason, I'm actually currently working on finishing my own implementation of a RFC 1951 DEFLATE compressor. The components built to implement DEFLATE will make implementing Brotli significantly easier.

Given that I work on my compression library as a personal side project, I don't anticipate getting this all done for a few months. Follow the project for updates on progress. Also, I can ping this issue when the Writer is in a functional state.

[humboldtux]

Thank you for explanations @dsnet

Should we put this on hold @mholt?

[mholt]

Yeah, I guess. But at least now Joe knows there is great interest in his work - so please do let us know, Joe, when you're ready for others to help or use it.

Or if anyone finds another pure Go brotli package before then, we could look at that too.

[mathwhiz1212]

I would like to see this included in caddy as well.

[mholt]

@dsnet By the way, I just realized I don't think I thanked you for chiming in a few months back. Thank you! 😄 Looking forward to your progress in the coming weeks/months. Looks like really great code so far.

[faddat]

I am not entirely sure this is the kind of thing you're looking for but:

https://github.com/Nelson69/izip/

[mholt]

No, unfortunately that package requires heavy amounts of cgo.

[faddat]

Gotcha-- well, I'll keep my eyes open :)

[wendigo]

Anyone want to implement golang version of brotli from scratch?

[faddat]

out of curiosity, what is gained by purging one's go of cgo?

Pride? performance? Stability?

Thanks :)

Jacob Gadikian
E-mail: [email protected]
SKYPE: faddat
Phone/SMS: +84 167 789 6421

On Thu, Oct 13, 2016 at 12:21 AM, Mateusz Gajewski <[email protected]

wrote:

Anyone want to implement golang version of brotli from scratch?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#525 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGz6iTV8to4EllN8cVbxWpLS4v-HdqrKks5qzRcWgaJpZM4HJhl_
.

[dsnet]

@faddat

It is incredibly hard to get cgo to work properly with stability on all platforms and across versions of Go. Even carefully reading the documentation, its easy to miss something. [0] [1] [2]

In my job, one of my tasks is to do the release of the next version of Go. Issues coming from cgo are probably among the most common failures (and also the most painful to debug). What worked in one version may not work in the next version because the cgo code made subtle assumptions about the runtime environment that happened to be true in one version, but not the next. After a release, it is common to see bug reports (some from popular projects) about cgo breaking after a release and IIRC a majority of them is due to faulty assumptions.

As an example of an upcoming change in Go1.8, SSA is getting smarter and no longer keeps variables live until the end of the function. This can cause failure in cgo when a pointer is passed to C, but ends up being the last reference of it. The GC may come and clear the backing memory. It is the programmer's responsibility to call runtime.KeepAlive to ensure that the variable is live in the duration of the cgo call (see golang/go#15843). Changes like this are not covered by the compatibility agreement.

Fundamentally, some packages must depend on cgo, but hopefully they have been battle tested by many users and carefully reviewed by experts.

(This isn't a smack on cgo, but to say that foreign function interfacing is always hard in any language)

[MisterDuval]

Hi folks,
I found this package: https://github.com/kothar/brotli-go
Could it be usefull in Brotli implementation?

[elcore]

Hello @devil1591,

sorry but no :( -- We want to use clean/pure Go (see above)

[mholt]

Well, since a brotli reader has been implemented in pure Go, Caddy now serves up pre-compressed brotli (and gzip) files if they are on disk with a .br extension. This went out with 0.9.4. I still don't know of a brotli writer in pure Go and I'm not sure if we'd want that anyway since, AFAIK, brotli compresses way slower than gzip. So maybe what we've got for now is good enough. Thanks everyone! And especially thanks to @dsnet. Happy New Year.

[Bechrissed]

Hi Matt, i'm not sure if it's the best choice to not look further into a pure go brotli writer. Akamai has written a nice article about brotli versus gzip. It seems brotli is only slower then gzip if you use them both with default settings. When using quality level 4 brotly is compresses 21% faster with 9% better compression on average. For more info please read: https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html

[wendigo]

@Bechrissed If you run caddy to serve static files you can always serve pre-compressed files (.br for brotli and .gz for gzip - brotli is preferred if both are present). If you use caddy as a frontend for other backends (fastcgi/proxy) you can always use brotli there with whatever implementation you would like. From my perspective - compressing files in the flight is bad solution and should be avoided.

[Bechrissed]

Yes, that might be true. I'll look for a build solution. Thanks

[Bechrissed]

I'm working on a fly package to generate the brotli compressed files. Will release when finished.

[vladbondarenko]

If you run something on backend that can brotli support (like nginx + ngx_brotli module) you can use caddy and caddy will send your Accept-Encoding header to backend. Backend will send compressed on-fly with brotli (content-encoding:br) and caddy will proxy it to you.

No need any support for this in proxy mode.

[Jabher]

@wendigo

(.br for brotli and .gz for gzip - brotli is preferred if both are present)

why not to prefer the one with the smaller file size?

[vladbondarenko]

what about proxy mode? how will you check which is smaller? brotli is smaller in most cases so it should be prefered.

[faradaytrs]

Yeah it would be cool to have brotli built-in support, at least as a plugin.

[faradaytrs]

Hi is a pure Go encoder and decoder @mholt , https://github.com/andybalholm/brotli
Any way to implement that? I know your opinion about brotli, that sometimes it can be even slower, but maybe it's better to let user to choose?

[whitestrake]

Not really an opinion - the facts are pretty well covered further up the thread regarding speed, no need to rehash. We'd love to see a PR, but I suspect Matt will be too busy with Caddy 2 efforts to look into this himself.

[mholt]

Thanks @faradaytrs - will consider that. @whitestrake you are right, but I might as well look into putting it into Caddy 2

[mholt]

We added it to Caddy 2, but it's unbearably slow when used with network requests. Might not make it a standard module for now. (But, hey, Caddy 2 has a good zstandard encoder instead.)