1.19.30 - Compression change from Flate to Snappy

[stevarino]

Looking over GopherTunnel's 1.19.30 work I'm seeing that Bedrock has switched compression protocols from Flate to Snappy. Opening this issue as a venue to discuss strategy.

Looking over the current implementation, I think this will only affect transforms/framer.js and transforms/encryption.js as I think DeFlate is basically just zlib? But I'm honestly not sure and this is just after a quick 5 minute browse.

As for replacements, I think the standard snappy package would work great.

Implementation wise, I'm not sure how to go about this. I'm thinking something like if (protocol < 552) return new FlateCompression() else return new SnappyCompression()? Or would a constant somewhere in minecraft-data be preferable?

[extremeheat]

Looks like there are changes to the login procedure and the addition of a new server configurable compression algorithm.

Unfortunately there is no standard library option here for snappy. We need to cut back on usage of native dependencies as they can be a hassle to deal with and also don't work in the browser. That starts with replacing raknet-native with a functional pure JS alternative, with the optional ability to use the native backend for better performance.

If we do use the native snappy over NAPI we will also need some sort of fallback JS implementation to polyfill the native module. Another option is to consider WASM packages that can implement snappy as we're not dependent on special system calls.

Will look into some of the performance considerations later.

[extremeheat]

Comparing packages, with a mix of small/moderate/large data...

snappyjs x 17.67 ops/sec ±1.31% (48 runs sampled)
snappy native (napi) x 52.32 ops/sec ±29.67% (47 runs sampled)
snappy-wasm x 2.25 ops/sec ±32.03% (9 runs sampled)
Fastest is snappy native (napi)

large files removed

snappyjs x 1,255 ops/sec ±1.41% (91 runs sampled)
snappy native (napi) x 2,672 ops/sec ±4.27% (80 runs sampled)
snappy-wasm x 25.66 ops/sec ±47.67% (6 runs sampled)
Fastest is snappy native (napi)

so it seems that the snappy-wasm is surprisingly slow, snappy native + fallback for snappyjs seem like the way to go

benchmark script

var Benchmark = require('benchmark');
var SnappyJS = require('snappyjs')
var SnappyRS = require('snappy')
var SnappyWASM = require('snappy-wasm')
const crypto = require('crypto')

let datas = []
for (let i = 0; i < 20; i++) { datas.push(crypto.randomBytes(100)) } // 100b
for (let i = 0; i < 20; i++) { datas.push(crypto.randomBytes(10000)) } // 10kb
for (let i = 0; i < 20; i++) { datas.push(crypto.randomBytes(1000000)) } // 1mb

var suite = new Benchmark.Suite;
// add tests
suite.add('snappyjs', function() {
	for (const data of datas) SnappyJS.uncompress(SnappyJS.compress(data))
})
.add('snappy native (napi)', function() {
	for (const data of datas) SnappyRS.uncompressSync(SnappyRS.compressSync(data))
})
.add('snappy-wasm', function() {
	for (const data of datas) SnappyWASM.decompress_raw(SnappyWASM.compress_raw(data))
})
// add listeners
.on('cycle', function(event) {
  console.log(String(event.target));
})
.on('complete', function() {
  console.log('Fastest is ' + this.filter('fastest').map('name'));
})
.run()