Vips CLI specify file format with pipes

[felixbuenemann]

In imagemagick I can read from and write to pipes and specify their format:

cat in.jpg | convert JPG:- -resize 50% PPM:- | jpeg-recompress --ppm -q medium - out.jpg

In the vips cli tools I can use /dev/stdin or /dev/stdout, but there doesn't seem to be a way to indicate the file format, which limits how much you can accomplish using pipes in vips.

It would be great if vips allowed to specify/override the loader/saver for input and output images.

Example syntax:

cat in.jpg | vips resize -[load=jpeg] -[save=ppm] 0.5 | jpeg-recompress --ppm -q medium - out.jpg

This would also allow chaining multiple vips cli operations without paying the disk i/o for writing to .v files.

[jcupitt]

Hi Felix, this has been talked about a bit, there's even an experimental branch with support for streams:

https://github.com/jcupitt/libvips/tree/load-from-stream

It can run this C program:

https://gist.github.com/jcupitt/ff21f5b8d1dd6ba54c6f

It reads a series of images from a socket, processes them, and sends them off to another socket, all without having to load the whole image. The aim was to support AWS buckets, but it would work for command-line pipes too, with a small amount of extra work.

There was a long chat about it on this issue:

lovell/sharp#30

I gave up for a bit, then it bitrotted, then I tried to clean it up ready to merge to master, and hit some problems. It needs a bit more work still :/

[felixbuenemann]

I wasn't talking about streaming multiple images, although that would be a cool feature.

The main idea is to be able to set the loader or saver (or roughly the file type), when it can not be inferred from the file extension. I'm sure there are more issues for filetypes that require seeking, but eg. jpeg, png, ppm would work fine. This would also be helpful, when there are multiple implementations that can emit or load the same file type. For example PDF could be loaded with either pdfload or magickload.

[jcupitt]

I think you'd need the streaming stuff for /dev/stdin to work. For example, this fails:

$ cat k2.jpg | vips jpegload /dev/stdin x.jpg
VipsJpeg: Not a JPEG file: starts with 0x87 0xe5
...

The file loaders expect to be able to open their input several times. The streaming stuff would add a read buffer that would make this possible -- the many images from a stream thing is just a bonus.

[jcupitt]

This is added in 8.9. You can write eg.

aws s3 cp s3://mybucket/input.jpg - | \
  vips thumbnail_source [descriptor=0] .jpg 128 | \
    aws s3 cp - s3://mybucket/output.jpg

The release notes have more information:

https://libvips.github.io/libvips/2019/12/11/What's-new-in-8.9.html