feat: Improve support for Decimal DType

[FBruzzesi]

Description

Adds precision and scale parameters to Decimal dtype.

TODO: Decide on stable API

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Closes [Enh]: Improve support for Decimal dtype #3375
• Blocking Enh: Add support for precision and scale keywords for Decimal dtype FBruzzesi/anyschema#101

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

[FBruzzesi]

Regarding the stable API, how do we feel for this feature? I would say that keeping it backward compatible for v2 might be a stretch, but for v1 it would be ok. cc @MarcoGorelli

[MarcoGorelli]

thanks for your pr!

i was under the impression that precision was going to be removed, but it looks like that didn't happen, i'll study pola-rs/polars#19784 and get back to this

[dangotbanned]

thanks for your pr!

i was under the impression that precision was going to be removed, but it looks like that didn't happen, i'll study pola-rs/polars#19784 and get back to this

To save you a long read, there was nothing conclusive in that issue.
But the PR that closed it does have a summary that's more digestible

• feat: Use fixed-scale Decimals pola-rs/polars#24542

On the Python-side, all that's changed is precision=38 gets assigned if you don't provide one to Decimal.

There are a lot more changes on the Rust-side (who would've guessed that?)
It appears that precision is here to stay, but there are a lot of (positive?) behavioral changes which we may have a tough time trying to align between backends.

I'm not pitching we do/don't do that - just trying to summarize the old-fashioned way 😉

[MarcoGorelli]

thanks @FBruzzesi !

[MarcoGorelli]

not sure what i'm missing but why not

precision: int = 38

?

[dangotbanned]

I suppose if we got an old pl.Decimal, that could have a None already?

Otherwise - yeah if we can avoid introducing the contructor with a None - I'm on board 😄

[FBruzzesi]

It's solely to mirror polars signature: polars.datatypes.Decimal and code

    def __init__(
        self,
        precision: int | None = None,
        scale: int = 0,
    ) -> None:
        if precision is None:
            precision = 38


        self.precision = precision
        self.scale = scale

[FBruzzesi]

@MarcoGorelli thanks for reviewing this. I have a couple of questions that might need to be discussed before merging:

1. Stable API - What's your take here?
2. Polars Decimal allows for any value of precision and scale at datatype initialization, however it raises an InvalidOperationError at runtime if scale is greater than precision. See example below. I would argue that to standardize the error message, we could raise such error during initialization.

Example:

import polars as pl

dtype = pl.Decimal(2, 3)  # Ok so far
pl.Series([1.2]).cast(dtype)  
> InvalidOperationError: scale must be less than or equal to precision

[dangotbanned]

@FBruzzesi

RE: (#3377 (comment)), (#3377 (comment))
I thought Decimal was marked as unstable in narwhals, but it looks like it wasn't? 🤔

@MarcoGorelli did you mean to introduce it as stable in #1571?
polars didn't stablize until a month after pola-rs/polars#24542 in pola-rs/polars#25020

[dangotbanned]

2. Polars Decimal allows for any value of precision and scale at datatype initialization, however it raises an InvalidOperationError at runtime if scale is greater than precision.
   See example below. I would argue that to standardize the error message, we could raise such error during initialization.

@FBruzzesi I agree, let's do that

Note

Sorry for blowing up CI with pandas>=3 warnings 🫣
https://github.com/narwhals-dev/narwhals/pull/3377/checks?sha=075ceb86353c6a511c068ea2bbf5986d83e3ccd9

[dangotbanned]

Regarding the stable API, how do we feel for this feature? I would say that keeping it backward compatible for v2 might be a stretch, but for v1 it would be ok

@FBruzzesi Initially I was thinking we should do the same thing for v1 and v2.
The only precedent we have is #2192, where the attributes were only added in main.

But the more I've read through your PR, I've started to think we might not need it?
AFAICT, for most backends the issue would have been that we change the native data types that we produce.
It looks like we didn't do that much:

• Raised instead of producing a native data type
  • arrow, duckdb, pandas_like, polars, spark_like
• Continues to not have native support
  • dask

Note

Only ibis could see a runtime change by using our new defaults

Have I understood all of this correctly?
If so, I would suggest adding (if needed) version-specific behavior for ibis-only, instead of an entire DType

[MarcoGorelli]

@MarcoGorelli thanks for reviewing this. I have a couple of questions that might need to be discussed before merging:

1. Stable API - What's your take here?

seems fine, it should be stable in polars now

2. Polars Decimal allows for any value of precision and scale at datatype initialization, however it raises an InvalidOperationError at runtime if scale is greater than precision. See example below. I would argue that to standardize the error message, we could raise such error during initialization.

Example:

import polars as pl

dtype = pl.Decimal(2, 3)  # Ok so far
pl.Series([1.2]).cast(dtype)  
> InvalidOperationError: scale must be less than or equal to precision

Sure sounds good (tbh i wouldn't have worried about standardising it, but no objections as you've already done it)

[FBruzzesi]

Thanks both for reviewing 🙏🏼

@dangotbanned as mentioned in discord, I did some digging in the ibis code and settled on f98decc

Let me know if you are happy with the state of this PR 😇

[dangotbanned]