Python: Table scan returning deleted data

[erikcw]

Apache Iceberg version

1.1.0 (latest release)

Query engine

Other

Please describe the bug 🐞

I originally mentioned raised this issue in #6567. After deleting rows from a table (in my case with Athena), pyiceberg is still returning parquet files with those records from a table scan. Shouldn't those files no longer be in the current manifest and hence not returned my the table scan?

-- Executed in Athena
DELETE FROM iceberg_test WHERE uid = '200441';

select count(uid) from "iceberg_test"
where uid = '200441';

-- Returns 0.

    # Glue catalog type.
    catalog = load_catalog("default")
    table = catalog.load_table("testing.iceberg_test")

    scan = table.scan(
        row_filter=NotEqualTo("uid", "200441"),  # Doesn't seem to make a difference with or without this line.
        selected_fields=("uid"),
    )
    files = [task.file.file_path for task in scan.plan_files()]
    # files all contain the deleted value.

[Fokko]

It looks like the delete strategy is merge-on-read (more information on the strategies can be found here), and this requires PyIceberg to filter out the deleted rows when reading the data.

[Fokko]

This is already fixed:

It throws an error now. The upcoming 0.4.0 release will support actual positional deletes.