Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Base types #76

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions src/stdio_mgr/compat.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
r"""``stdio_mgr.compat`` *code module*.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could also be called backports, and possibly _backports

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want both backports and compat... backports would be for things like the AbstractContextManager, whereas compat would be helper stuff for coping with pytest, colorama, readline, etc.?


``stdio_mgr.compat`` provides backports of Python standard library.

**Author**
John Vandenberg ([email protected])

**File Created**
6 Sep 2019

**Copyright**
\(c) Brian Skinn 2018-2019
jayvdb marked this conversation as resolved.
Show resolved Hide resolved

**Source Repository**
http://www.github.com/bskinn/stdio-mgr

**Documentation**
See README.rst at the GitHub repository
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These documentation links in all source files should point to the new RtD docs: https://stdio-mgr.readthedocs.io/


**License**
The Python-2.0 License.
bskinn marked this conversation as resolved.
Show resolved Hide resolved

**Members**

"""
import abc

# AbstractContextManager was introduced in Python 3.6
# and may be used with typing.ContextManager.
# See https://github.com/jazzband/contextlib2/pull/21 for more complete backport
try:
from contextlib import AbstractContextManager
except ImportError: # pragma: no cover
jayvdb marked this conversation as resolved.
Show resolved Hide resolved
# Copied from _collections_abc
def _check_methods(cls, *methods):
mro = cls.__mro__
for method in methods:
for base in mro:
if method in base.__dict__:
if base.__dict__[method] is None:
return NotImplemented
break
else:
return NotImplemented
return True

# Copied from contextlib
class AbstractContextManager(abc.ABC):
"""An abstract base class for context managers."""

def __enter__(self):
"""Return `self` upon entering the runtime context."""
return self

@abc.abstractmethod
def __exit__(self, exc_type, exc_value, traceback):
"""Raise any exception triggered within the runtime context."""
return None
bskinn marked this conversation as resolved.
Show resolved Hide resolved

@classmethod
def __subclasshook__(cls, subclass):
"""Check whether subclass is considered a subclass of this ABC."""
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem like the best description of, or name for, this method -- it appears to be checking whether subclass fully implements the abstract definition of AbstractContextManager

(Again, this may be how coredevs wrote it?)

if cls is AbstractContextManager:
return _check_methods(subclass, "__enter__", "__exit__")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why include __enter__ here when it's not @abc.abstractmethod?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return NotImplemented
47 changes: 7 additions & 40 deletions src/stdio_mgr/stdio_mgr.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"""

import sys
from contextlib import ExitStack, suppress
from contextlib import suppress
from io import (
BufferedRandom,
BufferedReader,
Expand All @@ -38,12 +38,11 @@
TextIOWrapper,
)

# AbstractContextManager was introduced in Python 3.6
# and may be used with typing.ContextManager.
try:
from contextlib import AbstractContextManager
except ImportError: # pragma: no cover
AbstractContextManager = object
from stdio_mgr.triple import AutoIOTriple, IOTriple
from stdio_mgr.types import MultiCloseContextManager

_RUNTIME_SYS_STREAMS = AutoIOTriple([sys.__stdin__, sys.__stdout__, sys.__stderr__])
_IMPORT_SYS_STREAMS = AutoIOTriple([sys.stdin, sys.stdout, sys.stderr])


class _PersistedBytesIO(BytesIO):
Expand Down Expand Up @@ -272,24 +271,7 @@ class SafeCloseTeeStdin(_SafeCloseIOBase, TeeStdin):
"""


class _MultiCloseContextManager(tuple, AbstractContextManager):
"""Manage multiple closable members of a tuple."""

def __enter__(self):
"""Enter context of all members."""
with ExitStack() as stack:
all(map(stack.enter_context, self))

self._close_files = stack.pop_all().close

return self

def __exit__(self, exc_type, exc_value, traceback):
"""Exit context, closing all members."""
self._close_files()


class StdioManager(_MultiCloseContextManager):
class StdioManager(MultiCloseContextManager, IOTriple):
r"""Substitute temporary text buffers for `stdio` in a managed context.

Context manager.
Expand Down Expand Up @@ -346,21 +328,6 @@ def __new__(cls, in_str="", close=True):

return self

@property
def stdin(self):
"""Return capturing stdin stream."""
return self[0]

@property
def stdout(self):
"""Return capturing stdout stream."""
return self[1]

@property
def stderr(self):
"""Return capturing stderr stream."""
return self[2]

def __enter__(self):
"""Enter context, replacing sys stdio objects with capturing streams."""
self._prior_streams = (sys.stdin, sys.stdout, sys.stderr)
Expand Down
100 changes: 100 additions & 0 deletions src/stdio_mgr/triple.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
r"""``stdio_mgr.types`` *code module*.

``stdio_mgr.types`` provides context managers for convenient
interaction with ``stdin``/``stdout``/``stderr`` as a tuple.

**Author**
John Vandenberg ([email protected])

**File Created**
6 Sep 2019

**Copyright**
\(c) Brian Skinn 2018-2019

**Source Repository**
http://www.github.com/bskinn/stdio-mgr

**Documentation**
See README.rst at the GitHub repository

**License**
The MIT License; see |license_txt|_ for full license terms

**Members**

"""
from io import TextIOBase

from stdio_mgr.types import MultiItemIterable, TupleContextManager


class IOTriple(TupleContextManager, MultiItemIterable):
"""Base type for a type of stdin, stdout and stderr stream-like objects.

While it is a context manager, no action is taken on entering or exiting
the context.

Used as a context manager, it will close all of the streams on exit.
however it does not open the streams on entering the context manager.

No exception handling is performed while closing the streams, so any
exception occurring the close of any stream renders them all in an
unpredictable state.
"""

def __new__(cls, iterable):
"""Instantiate new tuple from iterable containing three streams."""
items = list(iterable)
assert len(items) == 3, "iterable must be three items" # noqa: S101

return super(IOTriple, cls).__new__(cls, items)

@property
def stdin(self):
"""Return stdin stream."""
return self[0]

@property
def stdout(self):
"""Return stdout stream."""
return self[1]

@property
def stderr(self):
"""Return stderr stream."""
return self[2]


class TextIOTriple(IOTriple):
"""Tuple context manager of stdin, stdout and stderr TextIOBase objects."""

_ITEM_BASE = TextIOBase

# pytest and colorama inject objects into sys.std* that are not real TextIOBase
# and fail the assertion of this class

def __new__(cls, iterable):
"""Instantiate new tuple from iterable containing three TextIOBase streams."""
self = super(TextIOTriple, cls).__new__(cls, iterable)
if not self.all_(lambda item: isinstance(item, cls._ITEM_BASE)):
raise ValueError(
"iterable must contain only {}".format(cls._ITEM_BASE.__name__)
)
return self


class FakeIOTriple(IOTriple):
"""Tuple context manager of stdin, stdout and stderr-like objects."""


class AutoIOTriple(IOTriple):
"""Tuple context manager which will create FakeIOTuple or TextIOTuple."""

def __new__(cls, iterable):
"""Instantiate new TextIOTuple or FakeIOTuple from iterable."""
items = list(iterable)
if any(not isinstance(item, TextIOTriple._ITEM_BASE) for item in items):
return FakeIOTriple(items)
else:
return TextIOTriple(items)
122 changes: 122 additions & 0 deletions src/stdio_mgr/types.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
r"""``stdio_mgr.types`` *code module*.

``stdio_mgr.types`` provides misc. types and classes.

**Author**
John Vandenberg ([email protected])

**File Created**
6 Sep 2019

**Copyright**
\(c) Brian Skinn 2018-2019

**Source Repository**
http://www.github.com/bskinn/stdio-mgr

**Documentation**
See README.rst at the GitHub repository
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually thinking about going to multiple source files.

I've been thinking about it for a while also, and types.py isnt enough.

Possibly

  • iotypes.py or io.py for classes that do IO or are IOBase subclasses, and
  • maybe winio.py for any Windows specific logic we may accumulate, and
  • sys.py and/or consoleio.py for any logic around detecting python's console state (e.g. buffered/unbuffered detection),
  • stdiotypes.py or stdio.py for the tuple context-manager class hierarchy of re-usable components, leaving
  • stdio_mgr.py for finalised classes that implement high-level end-user-consumable tools.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that sort of structure makes sense to me.

sys.py and/or consoleio.py for any logic around detecting python's console state (e.g. buffered/unbuffered detection),

Also in here any logic for dealing with cases when the sys.stdfoo have already been wrapped/mocked by something else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have stayed with types.py for the tuple context-manager class hierarchy, as this is the core type hierarchy. Many of them are not strictly for tuples, or for context managers, or for stdio ;-).

Thoughts about using triples in module / class names?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts about using triples in module / class names?

As in Triple would be the specific sub-case of Tuple where it contains three items? Definitely good with that.


Separately, as I pore over these PRs, the value of #60 is becoming steadily clearer. I'm on board to fully type the thing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started this change, but running out of time so havent finished addressing PR comments.


**License**
The MIT License; see |license_txt|_ for full license terms

**Members**

"""
import collections.abc
from contextlib import ExitStack, suppress

try:
from contextlib import AbstractContextManager
except ImportError: # pragma: no cover
from stdio_mgr.compat import AbstractContextManager


class MultiItemIterable(collections.abc.Iterable):
"""Iterable with methods that operate on all items."""

@staticmethod
def _invoke_name(item, name):
item = getattr(item, name)
if callable(item):
bskinn marked this conversation as resolved.
Show resolved Hide resolved
return item()
return item
bskinn marked this conversation as resolved.
Show resolved Hide resolved

def _map_name(self, name):
"""Perform attribute name on all items."""
for item in self:
item = self._invoke_name(item, name)
yield item

def map_(self, op):
"""Return generator for performing op on all items."""
if isinstance(op, str):
return self._map_name(op)

return map(op, self)

def suppress_map(self, ex, op):
"""Return generator for performing op on all item, suppressing ex."""
for item in self:
with suppress(ex):
yield self._invoke_name(item, op) if isinstance(op, str) else op(item)

def all_(self, op):
"""Perform op on all items, returning True when all were successful."""
return all(self.map_(op))

def suppress_all(self, ex, op):
"""Perform op on all items, suppressing ex."""
return all(self.suppress_map(ex, op))

def any_(self, op):
"""Perform op on all items, returning True when all were successful."""
return any(self.map_(op))


class TupleContextManager(tuple, AbstractContextManager):
"""Base for context managers that are also a tuple."""

# This is needed to establish a workable MRO.


class ClosingStdioTuple(TupleContextManager, MultiItemIterable):
"""Context manager of streams objects to close them on exit.

Used as a context manager, it will close all of the streams on exit.
however it does not open the streams on entering the context manager.

No exception handling is performed while closing the streams, so any
exception occurring the close of any stream renders them all in an
unpredictable state.
"""

def close(self):
"""Close all streams."""
return list(self.map_("close"))

def safe_close(self):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is available for use by a user, but is not used by default?

Do you have it in mind to have an instantiation argument to (or functional-programming modification of) the cm to cause it to use safe_close on context exit, instead of plain close?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On any layer above this, the following will achieve that.

def close(self):
    self.safe_close()

"""Close all streams, ignoring any ValueError."""
return list(self.suppress_map(ValueError, "close"))

def __exit__(self, exc_type, exc_value, traceback):
"""Exit context, closing all members."""
self.close()
return super().__exit__(exc_type, exc_value, traceback)


class MultiCloseContextManager(TupleContextManager):
"""Manage multiple closable members of a tuple."""

def __enter__(self):
"""Enter context of all members."""
with ExitStack() as stack:
all(map(stack.enter_context, self))

self._close_files = stack.pop_all().close

return self

def __exit__(self, exc_type, exc_value, traceback):
"""Exit context, closing all members."""
self._close_files()
Loading