Classic jupyter notebook fails if first output > 3MB

[jleibs]

Using a moderately sized image can result in a crash in the kernel:

Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/jleibs/rerun/venv/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 206, in _handle_events
    handler_func(fileobj, events)
  File "/home/jleibs/rerun/venv/lib/python3.10/site-packages/tornado/iostream.py", line 702, in _handle_events
    self._handle_write()
  File "/home/jleibs/rerun/venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
    self._write_buffer.advance(num_bytes)
  File "/home/jleibs/rerun/venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
    assert 0 < size <= self._size
AssertionError

However, this doesn't not happen deterministically. If you first log a smaller image. For example only an output sized 2.5MB, and then output the larger size afterwards, everything is fine. It seems like we're overflying something in tornado.

Additionally this is only an issue with notebook classic. Jupyter lab and VS code do not share the issue.

[jleibs]

Looks like others have hit various versions of this: https://discourse.jupyter.org/t/jupyter-notebook-zmq-message-arrived-on-closed-channel-error/17869

One of the suggested workarounds:

pip install tornado==6.1 'jupyter-client<=8.1'

does seem to work for us.

[teh-cmc]

Likely fixed by #2092; can you confirm @jleibs ?