Skip to content

Conversation

@aokolnychyi
Copy link
Contributor

While working on time-related functions in the Spark function catalog, I wanted to use Transform. However, calling transforms on primitive values would require unnecessary boxing. That's why I decided to move relevant conversions to DateTimeUtil and leverage the utility directly. It is in line with what we did for bucketing in PR #5513.

@github-actions github-actions bot added the API label Nov 16, 2022
case DAYS:
return days;
default:
throw new UnsupportedOperationException("Unsupported time unit: " + granularity);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is now similar to toHumanString below.

@aokolnychyi
Copy link
Contributor Author

@kbendick @rdblue @RussellSpitzer @szehon-ho @flyrain, could you take a look at this PR? I need to use these conversions in our Spark function catalog.

return convertMicros(micros, ChronoUnit.HOURS);
}

private static int convertMicros(long micros, ChronoUnit granularity) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These conversions are covered by TestDates and TestTimestamps suites.

Copy link
Contributor

@nastra nastra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I think it would be good to also move TestDateTimeUtil to api

Copy link
Member

@szehon-ho szehon-ho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, would be good to get more eyes on it to double check, but otherwise looks like a clean refactor to me.


private static int convertMicros(long micros, ChronoUnit granularity) {
if (micros >= 0) {
long epochSecond = Math.floorDiv(micros, MICROS_PER_SECOND);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: am I missing something or can we just pull these out of individual blocks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can pull only epochSecond so I wasn't sure it was worth it. Let me do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved that var but it became somehow disconnected from the rest of the logic in each branch. I kept it separate like in the original snippet.

@aokolnychyi aokolnychyi merged commit b3eaf0c into apache:master Nov 17, 2022
@aokolnychyi
Copy link
Contributor Author

Thanks for reviewing, @nastra @szehon-ho @gaborkaszab!

sunchao pushed a commit to sunchao/iceberg that referenced this pull request May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants