Optimizes images and deletes stale and duplicated files

[lsaether]

closes #911

This PR does two things:

• Deletes any duplicate or unused images.
• Optimizes pictures.

[lsaether]

@Swader Could you take a look at this and let me know what you think? Most of the pictures definitely get smaller, but there are some that actually got bigger in filesize (not sure why that's happening).

Clone this branch locally and run yarn then node scripts/img/optimize.js optimize. This will generate *.testing files of the output of the script next to the original image in the docs/assets folder. Can you inspect the output and let me know what you think. A few of the images I looked at really decreased in quality and made me reconsider whether we should set the parameters differently from what you suggest in #911`.

[Swader]

The file size increase can happen if the image was pre-optimized in a good tool before, like Photoshop or Illustrator. This does a very good job so any "compression" is actually inflation. It's a case by case thing I suppose, and JPG cannot be optimized with a PNG optimizer tool, or vice versa, so it should be a tool per file type. The arguments are def up for tweaking, yeah. Will check specific output and see what can be done, thanks for getting this rolling.

[lsaether]

The question to take this PR forward is what quality of images we will be happy with, then we can set the image optimizer settings appropriately and replace all the images in the assets folder. Here is one example:

Before (257.7 KiB)

After Optimization w/ Resizing (82.3 KiB)

After Optimization w/o Resizing (143.3 KiB)

So based on this one example we get approximately a size reduction of 44% from just optimization while optimization and resizing gives us 68% gain from the original with a noticeable loss of quality. My preference would be to just do the optimization on the images without resizing.

[Swader]

Agreed, bigger images also lend themselves well to the PDF export

[Swader]

I get a bunch of nulls as output as the script works and a zlib error when testing:

null
null
null
null
null
(node:18406) UnhandledPromiseRejectionWarning: Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:180:17)
    at Inflate._processChunk (/home/swader/repos/polkadot-wiki/node_modules/pngjs/lib/sync-inflate.js:110:28)
    at zlibBufferSync (/home/swader/repos/polkadot-wiki/node_modules/pngjs/lib/sync-inflate.js:151:17)
    at inflateSync (/home/swader/repos/polkadot-wiki/node_modules/pngjs/lib/sync-inflate.js:155:10)
    at module.exports (/home/swader/repos/polkadot-wiki/node_modules/pngjs/lib/parser-sync.js:85:20)
    at Object.exports.read [as image/png] (/home/swader/repos/polkadot-wiki/node_modules/pngjs/lib/png-sync.js:10:10)
    at Jimp.parseBitmap (/home/swader/repos/polkadot-wiki/node_modules/@jimp/core/dist/utils/image-bitmap.js:196:53)
    at Jimp.parseBitmap (/home/swader/repos/polkadot-wiki/node_modules/@jimp/core/dist/index.js:431:32)
    at /home/swader/repos/polkadot-wiki/node_modules/@jimp/core/dist/index.js:373:15
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
(node:18406) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:18406) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. swader@DESKTOP-0OEAV37:~/repos/polkadot-wiki$

[Swader]

Some more verbose errors:

[Error: pngload: libpng read error
pngload: libpng read error
vips2png: unable to write to target docs/assets/NPoS/article-2.png.testing.testing
]
[Error: vips2png: unable to write to target docs/assets/NPoS/nominate.png.testing.testing
]
[Error: pngload: libpng read error
vips2png: unable to write to target docs/assets/NPoS/staking-keys_stash_controller.png.testing.testing
]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
null
null
[Error: Input file contains unsupported image format]
null
[Error: Input file contains unsupported image format]
null
[Error: pngload: libpng read error
vips2png: unable to write to target docs/assets/accounts/polkadot_js_01.png.testing.testing
]

Debugging...

[Swader]

Ok, heavy modifications are needed:

• the loops should detect the image extension and append that, rather than default to PNG. So something like:

  for (const path of images) {
    let ext = path.split(".").pop();
    sharp(path)
      ...
      .toFile(path + ".r." + ext, (err) => {
        ...
  }

  await Promise.all(
    images
      .map((img) => { let ext = path.split(".").pop(); img + ".r." + ext })
      ....
  );
};

• the optimization can be further lowered to 60%, seems safe
• the sharp lib should have the withoutEnlargement flag so it does make small images bigger: .resize(800, 600, { fit: "inside", withoutEnlargement: true })
• the loops should check to process only image file: jpg, jpeg, png. Gifs can be ignored. This will speed up the processing because it won't get hung up on invalid files, folers, etc (source of many of the errors above).
• the optimizer is missing a write() call so it never really did that in this current code
• and perhaps most importantly, the loop should, before doing a write, check the filesize of the source and the destination and replace source with dest ONLY if dest is smaller.

That last point would be the only optimization that would really help us achieve results. In my tests, all my PNGs saved from screenshots of Windows' snap tool are optimized to the bone. In fact, see here:

The files that end with r.png are just resized. Those with .o.png are also optimized. Not a single one filesize improvement and the JIMP optimizer made it worse in every case.

[laboon]

Is this still being worked on?

[Swader]

@lsaether what's up with this? Is it too much work to make it securely up to date with master at this point?