Verifier: Improve determinism analysis and checksum query recording#13758
Verifier: Improve determinism analysis and checksum query recording#13758mbasmanova merged 9 commits intoprestodb:masterfrom
Conversation
5ba3880 to
e669988
Compare
presto-verifier/src/test/java/com/facebook/presto/verifier/VerifierTestUtil.java
Outdated
Show resolved
Hide resolved
presto-verifier/src/main/java/com/facebook/presto/verifier/source/VerifierDao.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
@caithagoras What is Limit here stands for? Why not just QueryDeterminismAnalysis or better yet QueryDeterminismAnalysisResult?
There was a problem hiding this comment.
We do an additional determinism check for queries with LIMIT clause on the outer layer if all previous determinism analysis runs say DETERMINISTIC, and that's the LimitQueryDeterminismAnalysis.
Select/Insert/CreateTableAsSelect queries with LIMIT clause at the same level of the select query are eligible, expect for queries with ORDER BY clause at the same level.
SELECT ... LIMIT ...SELECT ... UNION ALL SELECT ... LIMIT ...INSERT INTO ... SELECT ... LIMIT ...CREATE TABLE ... AS SELECT ... LIMIT ...
What we do:
- We run a
count(1)query with theLIMITclause removed to get thetotal row count. - If
total row count>limit count, treat as non-deterministic.
There was a problem hiding this comment.
@caithagoras Thanks for explaining? Would you add this to the docs if it is not there already?
There was a problem hiding this comment.
@caithagoras That PR seems to be stuck for a few months now. What's blocking it?
There was a problem hiding this comment.
I'll need to address the comments. There is no blocker.
mbasmanova
left a comment
There was a problem hiding this comment.
@caithagoras Export checksum query and query id even if the query fails looks good.
mbasmanova
left a comment
There was a problem hiding this comment.
@caithagoras Export determinism analysis details looks good.
mbasmanova
left a comment
There was a problem hiding this comment.
@caithagoras Make determinism analysis run count configurable looks good.
mbasmanova
left a comment
There was a problem hiding this comment.
@caithagoras Fix variable name in VerifierConfig looks good
mbasmanova
left a comment
There was a problem hiding this comment.
@caithagoras Add an option to disable teardown for determinism analysis runs looks good too.
Overall, very clean PR. Thanks for breaking things up into small commits.
Also, remove unused method in VerifierTestUtil.
Uh oh!
There was an error while loading. Please reload this page.