Fix getNextValue() to increase the number of milliseconds if needed

[findinpath]

Description

This is a potential correctness issue when using as input

LongTimestampWithTimeZone[epochMillis=1483228799999, picosOfMilli=999000000, timeZoneKey=0]

getNextValue() will return

Optional[LongTimestampWithTimeZone[epochMillis=1483228799998, picosOfMilli=0, timeZoneKey=0]]

even though it should have been returning

Optional[LongTimestampWithTimeZone[epochMillis=1483228800000, picosOfMilli=0, timeZoneKey=0]]

Additional context and related issues

Cherry-picked from #14648

PR contains as well a cherry-pick from #18594 for JUnit5 related changes.

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Iceberg
* Fix partition filter subsuming when dealing with `timestamp with time zone` columns. ({issue}`issuenumber`)

[alexjo2144]

Good catch

[martint]

Is there a test that reproduces the issue?

[findinpath]

Is there a test that reproduces the issue?

I stumbled over it on BaseIcebergConnectorTest#testYearTransformDate() on #14648

        assertThat(query("SELECT * FROM test_year_transform_timestamptz WHERE d BETWEEN with_timezone(DATE '2015-01-01', 'UTC') AND with_timezone(TIMESTAMP '2016-12-31 23:59:59.999999', 'UTC')"))
                .isFullyPushedDown();

I can add however a test in io.trino.type.TestLongTimestampWithTimeZoneType

[findinpath]

Can we ensure automatically that this doesn't happen?
The method is silently ignored if not annotated with @Test

[martint]

We shouldn’t be overriding tests like that. If necessary, add new tests specific for the type.

[findinpath]

I don't follow.
This is pre-existing functionality.

Please add more detail to the comment.
Simply create a new method instead of doing overrides?

[martint]

I'm just responding to your comment about "ensuring automatically it doesn't happen". Structuring tests that way is not a good practice. It makes it hard to reason about which tests apply where (in some cases, the tests in AbstractXXX apply, in others they are completely overwritten, etc. -- usual arguments about using inheritance for reusing behavior in other contexts). We should fix them separately and not do that going forward.

[hashhar]

That is how ATDQ, BCT and other similar "super" test classes are designed to work. We'll need to rethink how to structure those before those can be migrated over safely - since without that it's hard to have re-usable tests which also evolve over time (e.g. testRange is overridden today to throw SkipException but in future once the feature is there the override can be removed).

[hashhar]

cc: @ksobolew any ideas to how to get this to work or how to restructure tests to avoid subclassing when using JUnit5?

[ksobolew]

Well, I can see that the implementations in the base class assert that the relevant properties of the type are Optional.empty(). They could, instead, do some assertions if it's not empty, and use some protected methods to get the expected result. But I'm not sure if that would work with all the types. It's kind of similar to how hasBehavior is used in connector tests...

[ksobolew]

I can see there's already #18594 :)

[findepi]

That is how ATDQ, BCT and other similar "super" test classes are designed to work. We'll need to rethink how to structure those before those can be migrated over safely

I can think of many different, more "pure" approaches.
The whole idea of BCT is impure and we could remove it. Connectors should have tests for features they implement.

Of course, this is straw-man's proposal. While letting us free of test overrides, this would lead to much worse test coverage.

[alexjo2144]

For me, these all run properly as is if you change the parent class AbstractTestType to use the org.testng.annotations.Test annotation instead of the junit org.junit.jupiter.api.Test annotation.

[martint]

Indeed, but that’s not how junit works — and we’re in the process of replacing testng with junit

[hashhar]

We should invest into writing a checker for this for the transition period like we have multiple for TestNG already. It seems very easy to do a migration but not realize multiple tests aren't running anymore.

[ksobolew]

Yeah, this is a huge landmine. I already stepped on it some time ago and we briefly discussed countermeasures, but nothing came out of it. I suppose we could write an Error Prone check for this (I am surprised that there is no one already).

[findepi]

Requirement to put @Test on overridden test is just junit's "ux bug", or a quirk, and agreed that we definitely need a checker for that. Thanks @ebyhr for reporting #14027.

[findinpath]

Rebased on master to re-use the commits from #18594