This repository was archived by the owner on May 9, 2024. It is now read-only.
Add non-lazy data import. #553
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: ienkovich <[email protected]>
alexbaden
reviewed
Jun 28, 2023
| @@ -715,7 +715,8 @@ void ArrowStorage::appendArrowTable(std::shared_ptr<arrow::Table> at, int table_ | |||
| switch (col_arr->type()->id()) { | |||
| case arrow::Type::STRING: | |||
| // if the dictionary has already been materialized, append indices | |||
| if (!config_->storage.enable_lazy_dict_materialization || | |||
| if (config_->storage.enable_non_lazy_data_import || | |||
There was a problem hiding this comment.
Is this necessary simply to force non-lazy data import in all cases when non lazy data import is enabled in config (vs just lazy dict materialization), or is there a codepath here that is important if we want non-lazy data import and lazy dict materialization is already disabled?
There was a problem hiding this comment.
I did this change assuming that lazy dictionaries are going to be enabled by default soon and I would like to disable all lazy import features with a single flag in benchmarks.
alexbaden
approved these changes
Jun 28, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new option that allows to avoid data fetch costs on execution moving it to data import. This option is for benchmarking only when we want to split data import/transformation time and execution time. This can lead to an overall workload execution time increase in the case of unused columns in imported data.