Skip to content

fix decimal precision for division#2308

Merged
jycor merged 8 commits intomainfrom
james/div
Feb 2, 2024
Merged

fix decimal precision for division#2308
jycor merged 8 commits intomainfrom
james/div

Conversation

@jycor
Copy link
Contributor

@jycor jycor commented Jan 31, 2024

divScale tracks the number of division operators on the leftSubtree. It can be used to determine if the current expression is the outermost Div expression, and doubles as a way to determine how many decimal places of precision we need to add.
opScale tracks the total number of Arithmetic operators in the expression tree in total. It is used to identify the outermost Arithmetic expression.
There were a few places where we were mixing up divScale and opScale, leading to improper rounding; this PR fixes that.
As a result, we are able to properly round decimal results when the outermost operator is not a Div.

Additionally, this PR also contains a fix for functions that return float64 being incorrectly converted to decimal.

correctness: dolthub/dolt#7442
fixes dolthub/dolt#4931

Copy link
Contributor

@fulghum fulghum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome fix! No notes! 👏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

incorrect decimal scale if division is used, but it is not the last operation

2 participants