Skip to content

Conversation

@dchigarev
Copy link
Collaborator

@dchigarev dchigarev commented Dec 6, 2023

What do these changes do?

This PR moves the following IO function to the modin.pandas.io module:

  • from_pandas()
  • from_arrow()
  • from_dataframe()
  • from_non_pandas()
  • to_pandas()
  • to_numpy()

The try_cast_to_pandas() function was not moved as IMO it's more of a utility function rather than a pure IO. However, if reviewers think differently, I still can move it to the IO module.

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Move all IO functions to the modin.pandas.io module #6805
  • tests are passing
  • module layout described at docs/development/architecture.rst is up-to-date

from modin.error_message import ErrorMessage
from modin.logging import disable_logging
from modin.pandas import Categorical
from modin.pandas.io import from_non_pandas, from_pandas, to_pandas

Check notice

Code scanning / CodeQL

Cyclic import

Import of module [modin.pandas.io](1) begins an import cycle.
def DataFrame(cls):
"""Get ``modin.pandas.DataFrame`` class."""
if cls._dataframe is None:
from .dataframe import DataFrame

Check notice

Code scanning / CodeQL

Cyclic import

Import of module [modin.pandas.dataframe](1) begins an import cycle.
from modin.config import PersistentPickle
from modin.logging import disable_logging
from modin.utils import MODIN_UNNAMED_SERIES_LABEL, _inherit_docstrings, to_pandas
from modin.pandas.io import from_pandas, to_pandas

Check notice

Code scanning / CodeQL

Cyclic import

Import of module [modin.pandas.io](1) begins an import cycle.
from .iterator import PartitionIterator
from .series_utils import CategoryMethods, DatetimeProperties, StringMethods
from .utils import _doc_binary_op, cast_function_modin2pandas, from_pandas, is_scalar
from .utils import _doc_binary_op, cast_function_modin2pandas, is_scalar

Check notice

Code scanning / CodeQL

Cyclic import

Import of module [modin.pandas.utils](1) begins an import cycle.
Signed-off-by: Dmitry Chigarev <[email protected]>
Signed-off-by: Dmitry Chigarev <[email protected]>
Signed-off-by: Dmitry Chigarev <[email protected]>
"""
return modin_obj._to_pandas()

def deprecated_func(*args: tuple[Any], **kwargs: dict[Any, Any]) -> Any:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use @functools.wraps(func) here or assign some attributes like name, doc to deprecated_func from a deprecated function to be called?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't do this. The code building a decorator is being executed at the time of the module's initialization, which means, that we cannot do circular imports at this point. See the simplified example:

def create_deprecated(msg):
    # the code in this function is being executed at the time of module initialization,
    # meaning that the following line will trigger the circular import error:
    # from modin.pandas.io import to_pandas

    # can't use this as we haven't imported 'to_pandas' yet
    # @functols.wrap(to_pandas)
    def wrapper(*args, **kwargs):
        # we can only import actual 'to_pandas' here as this code is being executed
        # when the function is called and all the modules are initialized
        from modin.pandas.io import to_pandas
        return to_pandas(*args, **kwargs)

    return wrapper
   
to_pandas = create_deprecated("to_pandas")

@YarShev
Copy link
Collaborator

YarShev commented Dec 6, 2023

Left a few comments. Other than that, LGTM.

Comment on lines 90 to 91
@classmethod
@property
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

@dchigarev dchigarev Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to using this as a classmethod-constructor

it didn't work, will find another solution

UPD: ended up writing a custom-written decorator allowing class properties

Signed-off-by: Dmitry Chigarev <[email protected]>
Signed-off-by: Dmitry Chigarev <[email protected]>
Copy link
Collaborator

@anmyachev anmyachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@YarShev YarShev merged commit 4b090f2 into modin-project:master Dec 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move all IO functions to the modin.pandas.io module

3 participants