Simplify Hive connector scale writers test#18688
Merged
raunaqmorarka merged 1 commit intotrinodb:masterfrom Aug 17, 2023
Merged
Simplify Hive connector scale writers test#18688raunaqmorarka merged 1 commit intotrinodb:masterfrom
raunaqmorarka merged 1 commit intotrinodb:masterfrom
Conversation
gaurav8297
force-pushed
the
fix_scale_writer_flaky
branch
3 times, most recently
from
628da68 to
c2de9d5
Compare
findepi
reviewed
Aug 16, 2023
Member
There was a problem hiding this comment.
How much memory it will use and how long can it take?
Member
|
test (plugin/trino-hive) is the build failure related? |
Member
|
cc @lukasz-stec |
lukasz-stec
reviewed
Aug 16, 2023
plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java
Outdated
Show resolved
Hide resolved
gaurav8297
force-pushed
the
fix_scale_writer_flaky
branch
from
c2de9d5 to
429156b
Compare
gaurav8297
changed the title
gaurav8297
force-pushed
the
fix_scale_writer_flaky
branch
from
429156b to
60ba1bd
Compare
First, We need to use large table (sf2) to see the effect. Otherwise, a single writer will write the entire data before ScaledWriterScheduler is able to scale it to multiple machines. Second, Since we now use page size to scale up instead of physical written bytes, we don't have to test on different file formats. This reduces the number of tests and overall runtime.
gaurav8297
force-pushed
the
fix_scale_writer_flaky
branch
from
60ba1bd to
e114836
Compare
raunaqmorarka
approved these changes
Aug 17, 2023
lukasz-stec
approved these changes
Aug 17, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two changes:
First, We need to use a large table (sf2) to see the
effect. Otherwise, a single writer will write the
entire data before ScaledWriterScheduler is able to
scale it to multiple machines.
Second, since we now use page size to scale up instead
of physical written bytes, we don't have to test on different
file formats. This reduces the number of tests and overall
runtime.
fixes: #18670
Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: