save to bytes and got this error: io.UnsupportedOperation: fileno

[CoinCheung]

What did you do?

save an jpeg image to bytes string.

What did you expect to happen?

Obtain the string successfully.

What actually happened?

Script crashes

What are your OS, Python and Pillow versions?

• OS: ubuntu 22.04
• Python: 3.10
• Pillow: 11.0.0

--------------------------------------------------------------------
Pillow 11.0.0
Python 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0]
--------------------------------------------------------------------
Python executable is /opt/miniconda3/envs/py310/bin/python3
System Python files loaded from /opt/miniconda3/envs/py310
--------------------------------------------------------------------
Python Pillow modules loaded from /opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL
Binary Pillow modules loaded from /opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 11.0.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.12.1
--- LITTLECMS2 support ok, loaded 2.12
--- WEBP support ok, loaded 1.3.2
--- JPEG support ok, compiled for 9.0
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.13
--- LIBTIFF support ok, loaded 4.5.1
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------```

import PIL
from PIL import Image
import io
import time

impth = 'n12762896_263.JPEG'
img = Image.open(impth)

buf = io.BytesIO()
img.save(buf, format='jpeg', quality=90, subsampling=0, optimize=True)

Error message is:

  File "/opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL/Image.py", line 2605, in save
    save_handler(self, fp, filename)
  File "/opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL/JpegImagePlugin.py", line 843, in _save
    ImageFile._save(
  File "/opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL/ImageFile.py", line 558, in _save
    _encode_tile(im, fp, tile, bufsize, None, exc)
  File "/opt/miniconda3/envs/py310/lib/python3.10/site-packages/PIL/ImageFile.py", line 593, in _encode_tile
    raise _get_oserror(errcode, encoder=True) from exc
OSError: broken data stream when writing image file

This only happens with certain images, here is an image that can cause this error:

[radarhere]

If you're looking for an immediate solution, I find that removing optimize=True works.

Otherwise, the problem would seem to be at

Pillow/src/PIL/JpegImagePlugin.py

Line 843 in 6ea7dc8

bufsize = im.size[0] * im.size[1]

where the buffer size is not large enough.

I have two questions.

1. Do you have any images that show this problem that we would be able to add to our test suite and distribute under the Pillow license? Ideally, as small an image as possible.
2. Since you have multiple images that display this problem, are you able to test to see if a change works with all of them? Changing the above line to

bufsize = int(1.2 * im.size[0] * im.size[1])

[CoinCheung]

https://github.com/CoinCheung/eewee/releases/download/0.0.0/error_images.tar.zst

Here is about 30 images that I found have this problem.

[radarhere]

So my 1.2 change does work for all of those images.

Are any of those images able to be added to our test suite and distributed under the Pillow license?

[CoinCheung]

Hi,

So my 1.2 change does work for all of those images.

I fear that 1.2 would not be enough since I cannot afford to re-run my program for many times, I just change that to 5.

Are any of those images able to be added to our test suite and distributed under the Pillow license?

Yes, these images are open to download for research purpose, I think all of these images can be added to test suite.

[radarhere]

I fear that 1.2 would not be enough since I cannot afford to re-run my program for many times, I just change that to 5.

I'm not sure what you were trying to say in this comment.

I'm saying that when I locally ran your code with all of the images that you have provided, if I change bufsize = im.size[0] * im.size[1] to int(1.2 * im.size[0] * im.size[1]) within Pillow, then your code passes on my machine.

You don't necessarily need to check the code any further.

[CoinCheung]

I mean that I changed it to 5 * im.size[0] * im.size[1]. I have plenty of images, and it costs me about 2 days to process them all. The provided 30 images are only part of these error images. I fear that this problem would come again on other un-tested images, so I just use ratio of 5 instead of 1.2 on bufsize.

[radarhere]

I find that when I process all the images you have supplied, I also get 'Suspension not allowed here'. That led me back to #5448, where, rather than modifying Pillow's internals, one solution is to increase ImageFile.MAXBLOCK.

If I start out the script with

from PIL import Image, ImageFile
ImageFile.MAXBLOCK = 168533

then all of the images that you've supplied pass. I can't speak for images that I don't have access to, but there you go. I just tested with trial and error until I found that number. If you'd like to increase it higher to allow for other potential images, then you can.

It is larger than our default ImageFile.MAXBLOCK value, so I don't want to increase it permanently - that kind of defeats the purpose of having a limit.

Is this sufficient for you as a workaround?

[CoinCheung]

Thanks!! This solution works!