feat(python): Add support for Iceberg

[Fokko]

Resolves #6227

Using PyIceberg:

• Implements field-id-based schema resolution
• Integration with Catalogs, among others, Hive, Glue, DynamoDB, REST, and SqlCatalog.
• Both partition and metrics-based data file pruning; will only load files that are relevant to the expression. Reducing the data being read and the number of calls to the object store.
• Support for AWS, Azure, and GCS is in the next release.
• Using the PyArrow backend.

[stinodego]

Hi @Fokko, looks like you've done quite some work here! Could you please make sure the CI passes? Then we'll take a closer look.

Let us know if you need any help. I'll put this on draft for now.

[Fokko]

@stinodego Thanks!

Could you please make sure the CI passes? Then we'll take a closer look.

Yes, I'm on top of it. PyIceberg is still at Pydantic <2. I have a PR ready on the PyIceberg side to bump it to 2, then we need to do a release, and then it should pass the CI.

[stinodego]

Why do we need a second _scan_pyarrow_dataset_impl? Can we not reuse the existing implementation?

[stinodego]

@Fokko Could you come back to me on this one?

[Fokko]

Of course. Here we do something different than in the original one. In the original, the Python eval function is used to convert the string containing Python to an actual Python class, and that's being passed into the delta library:

    if predicate:
        # imports are used by inline python evaluated by `eval`
        from polars.datatypes import Date, Datetime, Duration  # noqa: F401
        from polars.utils.convert import (
            _to_python_datetime,  # noqa: F401
            _to_python_time,  # noqa: F401
            _to_python_timedelta,  # noqa: F401
        )
    
        _filter = eval(predicate)

What we do here is that we take the string, we convert it into an abstract syntax tree, and that's being traversed to convert it into a PyIceberg expression. The reason why I did this is that the PyArrow expression doesn't have any Python methods to traverse the expression (the same goes for the Polars expression, otherwise I could just traverse that one as well). I've added this to the docstring as well 👍🏻

[stinodego]

Good stuff. I'm on holiday next week. Feel free to merge once pyiceberg 0.5.0 has been released and the PR has been updated accordingly.

[Fokko]

@stinodego @alexander-beedie PyIceberg 0.5.0 has been released, and the PR is good to go :)

[alexander-beedie]

Alright... here we go ;)

[Fokko]

Awesome, thanks all! 🚀

[alexander-beedie]

Awesome, thanks all! 🚀

It's a great addition; looking forward to seeing it in action 👍

[Fokko]

I did a small demo at PyData Amsterdam last week. Once it is released, I'll work on a blogpost! 👍🏻

[luancaarvalho]

@Fokko Do you have the demo at PyData Amsterdam recorded ?

[Fokko]

@luancaarvalho The demo is recorded, but not yet online. I think they still have to do some final editing.