Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2

[mkoeppe]

(from #31280 comment:42)

CC: @antonio-rojas @isuruf @kiwifb @dimpase @orlitzky

Component: interfaces

Issue created by migration from https://trac.sagemath.org/ticket/32147

[mkoeppe]

comment:1

(#31280 comment:46):
If we are lucky, this is just pexpect/ptyprocess@f673a54

[mkoeppe]

comment:2

Just catching PtyProcessError in addition to ExceptionPexpect on our side does not seem to fix the problem.

[tornaria]

comment:6

Just to mention that I've been using system ptyprocess-0.7.0 (via #29665) for a few weeks without any issue.

What's the exact issue? I haven't seen any report of what goes wrong on macos when using an updated ptyprocess.

[mkoeppe]

comment:7

It looks like this:

$ ./sage -pip install -U ptyprocess
Requirement already satisfied: ptyprocess in ./local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages (0.5.1)
Collecting ptyprocess
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Installing collected packages: ptyprocess
  Attempting uninstall: ptyprocess
    Found existing installation: ptyprocess 0.5.1
    Uninstalling ptyprocess-0.5.1:
      Successfully uninstalled ptyprocess-0.5.1
Successfully installed ptyprocess-0.7.0
egret:~/s/sage/sage-rebasing/worktree-gcc11 (mkoeppe-9.5.beta9+updates $%>)$ ./sage -tp src/sage/interfaces/
too many failed tests, not using stored timings
Running doctests with ID 2022-01-19-10-06-35-c84011cc.
Using --optional=build,ccache,cryptominisat,debugpy,e_antic,homebrew,igraph,jupymake,normaliz,pip,pycryptosat,pynormaliz,sage,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Sorting sources by runtime so that slower doctests are run first....
Doctesting 61 files using 8 threads.
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/gap.py
**********************************************************************
File "src/sage/interfaces/gap.py", line 644, in sage.interfaces.gap.Gap_generic._eval_line
Failed example:
    a = gap(3)
Exception raised:
    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 511, in _execute_line
        E.sendline(line)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 578, in sendline
        return self.send(s + self.linesep)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 569, in send
        return os.write(self.child_fd, b)
    OSError: [Errno 5] Input/output error

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 658, in _eval_line
        (normal, error) = self._execute_line(line, wait_for_prompt=wait_for_prompt,
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 513, in _execute_line
        raise RuntimeError("Error evaluating %s in %s"%(line, self))
    RuntimeError: Error evaluating \$sage7:=3;; in Gap

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1166, in _start
        Expect._start(self, "Failed to start GAP.")
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 444, in _start
        self.quit()  # in case one is already running
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 626, in quit
        self._close()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 558, in _close
        self._expect.close(force=force)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 328, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 204, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:3268)
        self.fileobj.close()
    OSError: [Errno 5] Input/output error

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 1470, in __init__
        self._name = parent._create(value, name=name)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 513, in _create
        self.set(name, value)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1328, in set
        self._eval_line(cmd, allow_use_file=True)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 701, in _eval_line
        self._start()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1170, in _start
        self.quit()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 626, in quit
        self._close()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 558, in _close
        self._expect.close(force=force)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 328, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 201, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:3177)
        self.write(str_to_bytes(self.quit_string))
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ptyprocess/ptyprocess.py", line 568, in write
        return self._writeb(s, flush=flush)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ptyprocess/ptyprocess.py", line 558, in _writeb
        n = self.fileobj.write(b)
    ValueError: write to closed file

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.gap.Gap_generic._eval_line[8]>", line 1, in <module>
        a = gap(Integer(3))
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 302, in __call__
        result = self._coerce_from_special_method(x)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 330, in _coerce_from_special_method
        return (x.__getattribute__(s))(self)
      File "sage/structure/sage_object.pyx", line 718, in sage.structure.sage_object.SageObject._gap_ (build/cythonized/sage/structure/sage_object.c:6249)
        return self._interface_(G)
      File "sage/structure/sage_object.pyx", line 694, in sage.structure.sage_object.SageObject._interface_ (build/cythonized/sage/structure/sage_object.c:5783)
        X = I(s)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 295, in __call__
        return cls(self, x, name=name)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 1475, in __init__
        raise TypeError(*x.args)
    TypeError: write to closed file
[......]
----------------------------------------------------------------------
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/gap.py  # 69 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/mwrank.py  # 6 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/expect.py  # 40 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/quit.py  # 4 doctests failed
----------------------------------------------------------------------
Total time for all tests: 39.7 seconds
    cpu time: 42.7 seconds
    cumulative wall time: 130.5 seconds
Features detected for doctesting: 

[mkoeppe]

comment:8

Attachment: ptyprocess-failures.txt

[dimpase]

I cannot reproduce some of this with python 3.11.5 on macOS and system-wide installed ptyprocess 0.7.0 (and pexpect 0.4.8), used via --enable-system-site-packages.

dima@oucl13243 sage % ./sage --pip freeze | grep ptypr     
ptyprocess==0.7.0
dima@oucl13243 sage % uname -a
Darwin oucl13243.lan 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64
dima@oucl13243 sage % ./sage -tp src/sage/interfaces/

----------------------------------------------------------------------
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/mwrank.py  # 2 doctests failed
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/expect.py  # 38 doctests failed
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/quit.py  # 4 doctests failed

some of it depends upon the random seed. E.g. the following reliably passes. Or, perhaps, it only matters that it's not
run in parallel with other tests?

dima@oucl13243 sage % ./sage -tp --random-seed=192325368743415462395845369501584124267 src/sage/interfaces/quit.py
Running doctests with ID 2023-09-26-11-57-44-882b3f70.
Git branch: HEAD
Git ref: 9.2.rc2-27519-gf00562ee4e
Running with SAGE_LOCAL='/Users/dima/software/sage/local' and SAGE_VENV='/Users/dima/software/sage/local/var/lib/sage/venv-python3.11'
Using --optional=homebrew,ipympl,jupymake,notedown,pip,pycosat,pysingular,rst2ipynb,sage,sage_spkg,texttable
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_jones_numfield,database_knotinfo,dvipng,fpylll,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,graphviz,imagemagick,ipython,jupymake,kenzo,latte_int,lrcalc_python,lrslib,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,pyparsing,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,sympy,tdlib
Doctesting 1 file using 2 threads.
sage -t --warn-long 88.6 --random-seed=192325368743415462395845369501584124267 src/sage/interfaces/quit.py
    [21 tests, 1.44 s]
----------------------------------------------------------------------
All tests passed!

[mkoeppe]

Yes, repro of this bug is difficult. Didn't succeed when I tried just now.

[dimpase]

try

gap.eval("quit;")
gap.quit()

this reproduces the bug every time on my Intel macOS.

[mkoeppe]

Perfect, thanks. Meanwhile I was also able to provoke the doctester into reproducing the failures.