feat: add table sampling support #1421
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jczhong84
commented
Mar 7, 2024
Comment on lines
+22
to
+35
| def format_query(query: str, language: Optional[str] = None): | ||
| dialect = _get_sqlglot_dialect(language) | ||
| statements = transpile( | ||
| query, | ||
| read=dialect, | ||
| write=dialect, | ||
| pretty=True, | ||
| ) | ||
|
|
||
| return _statements_to_query(statements) | ||
|
|
||
|
|
||
| def get_select_statement_limit( | ||
| statement: Union[exp.Expression, str], |
There was a problem hiding this comment.
added those functions for query limiting, but has not been used.
czgu
reviewed
Mar 16, 2024
Comment on lines
105
to
113
| const tableSamplingDOM = | ||
| !disabled && TableSamplingConfig.enabled && hasSamplingTables ? ( | ||
| <TableSamplingSelector | ||
| sampleRate={sampleRate} | ||
| setSampleRate={onSampleRateChange} | ||
| tooltipPos={runButtonTooltipPos} | ||
| /> | ||
| ) : null; | ||
|
|
There was a problem hiding this comment.
can this live outside of QueryRunButton? I dont think it should control the UI for TableSamplingSelector, that's the job of the datacell or querycomposer
There was a problem hiding this comment.
will address it in a separate PR to use backend limit logic.
| })) | ||
| ); | ||
| const DEFAULT_SAMPLE_RATE = TableSamplingConfig.default_sample_rate; | ||
| const TableSamplingSelector: React.FC<{ |
There was a problem hiding this comment.
move to another component file
There was a problem hiding this comment.
will address it in a separate PR to use backend limit logic.
czgu
previously approved these changes
Mar 19, 2024
czgu
approved these changes
Mar 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here we add two approaches of table sampling support:
Sampled Table
There will be a new separated sampled version of the original table being created.
Bucketing
There will be no new table, but the original table will be reorganized so that it can be queried with additional filters from a small portion of the dataset. This will be using the official
TABLESAMPLEclause from the language.If a table support sampling, we can add it in the column of
custom_propertiesof tabledata_table_informationIndicating if this table supports sampling or not.
Name of the sampled version of the table. This has higher priotiry. If sampled_table is not provided while sampling is true, we'll use the
TABLESAMPLEclause.If any table in the query support sampling, we'll show the
Sampleselector on the right side of the query engine selectorIf a sample rate is selected, we'll replace the table with either with sampled table or use the
TABLESAMPLEclause to transform the query before running.table_sampling.mp4