Check if schema is compatible in add_files API

[sungwy]

Fixes: #869

Made this mandatory instead of optional

[HonahX]

@syun64 Thanks for working on this. It looks great!

[HonahX]

My understand is that now if we enable downcast-ns-timestamp-to-us-on-write, we allow user to add parquet files with TIMESTAMP_NANOS type data. My concern here is that we may add parquet files that not align with spec, which states that timestamp/timstamptz type should map to TIMESTAMP_MICROS. Shall we be more restrictive when checking the parquet file that will be directly added to the table?

[sungwy]

Thank you for the catch @HonahX - I think you are right. Adding a nanosecond timestamp file doesn't correctly allow Spark Iceberg to read the file and instead results in exceptions like:

ValueError: year 53177 is out of range

I will make downcast_ns_timestamp_to_us_on_write an input argument to _check_schema_compatible, so that we can prevent nanoseconds timestamp types from being added through add_files, but can continue to support it being downcast in overwrite/append

[Fokko]

In general, I think it is okay to be able to read more broadly. We do need tests to ensure that it works correctly. Looking at Arrow, there are already some physical types that we don't support (date64, etc). In Java, we do support reading Timestamps that are encoded using INT96, we should not produce them.

[HonahX]

LGTM!