Skip to content

Conversation

@dain
Copy link
Member

@dain dain commented Mar 14, 2024

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

Copy link
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approach looks ok. I don't think it can impact perf, can it?

@sopel39
Copy link
Member

sopel39 commented Mar 14, 2024

test failures seem relevant

Copy link
Member

@sopel39 sopel39 Mar 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably should keep track of chunk first channel index, e.g: 0, 500, 1000, so that correct offsets are used when accessing: byte[] fixedChunk, byte[] variableChunk and BlockBuilder[] blockBuilders);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume there is a bug like this somewhere in here, but my memory is that all of the indexes come from KeyField and not the oridinal of the key field object in the list… If you look at the code, I don’t think we are using an indexed for loop anywhere.

@dain
Copy link
Member Author

dain commented Mar 14, 2024

I don't think it can impact perf, can it?
I doubt it. The method just call though to one or more statics. It is possible that smaller methods help as large methods cause vm optimizers to be disabled. I picked 500 arbitrarily and it is possible 100 is better for this reason

@dain dain force-pushed the wide-hash branch 3 times, most recently from 46c1603 to d6c86b5 Compare March 18, 2024 00:21
@github-actions github-actions bot added tests:hive delta-lake Delta Lake connector hive Hive connector labels Mar 18, 2024
@dain dain force-pushed the wide-hash branch 2 times, most recently from 4f9ea39 to 48c9d16 Compare March 18, 2024 01:36
@dain dain merged commit 83994ed into trinodb:master Mar 18, 2024
@dain dain deleted the wide-hash branch March 18, 2024 05:49
@github-actions github-actions bot added this to the 443 milestone Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed delta-lake Delta Lake connector hive Hive connector

Development

Successfully merging this pull request may close these issues.

2 participants