-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
String to datetime conversion with custom format #17167
Comments
Thanks For the report, I'm surprised this didn't work, will check |
This might be a "bug" or design decision by chrono (the rust parser) BUT
|
To me this looks like it might be a bug in Chrono: use chrono::NaiveDateTime;
fn main() {
let result = NaiveDateTime::parse_from_str(
"2024-06-03 20:02:48.6800000",
"%Y-%m-%d %H:%M:%S%.6f0",
);
println!("{:?}", result);
let result = NaiveDateTime::parse_from_str(
"2024-06-03 20:02:48.680000",
"%Y-%m-%d %H:%M:%S%.6f",
);
println!("{:?}", result);
} prints out
|
For reference I have created a pytest file that tests several format strings on I found that polars converts the string even without the trailing 0 in the format when using either pytest file: from datetime import datetime
import pandas as pd
import polars as pl
import pytest
dt_string = "2024-06-03 20:02:48.6800000"
dt_formats = [
"%Y-%m-%d %H:%M:%S%.f",
"%Y-%m-%d %H:%M:%S%.6f",
"%Y-%m-%d %H:%M:%S.%f",
"%Y-%m-%d %H:%M:%S.%6f",
]
dt_formats = dt_formats + [f + "0" for f in dt_formats]
@pytest.fixture
def df():
return pl.DataFrame({"dt": dt_string})
@pytest.mark.parametrize("format", dt_formats)
def test_polars(df, format, capsys):
with capsys.disabled():
print(format)
print(df)
t_ref = pl.Series(name="dt", values=[datetime(2024, 6, 3, 20, 2, 48, 680000)])
df2 = df.with_columns(dt=pl.col("dt").str.to_datetime(format, time_unit="ns"))
with capsys.disabled():
print(df2)
assert (df2["dt"] == t_ref).all()
@pytest.mark.parametrize("format", dt_formats)
def test_pandas(df, format, capsys):
with capsys.disabled():
print(format)
print(df)
t_ref = pd.Series(name="dt", data=[datetime(2024, 6, 3, 20, 2, 48, 680000)])
t = pd.to_datetime(
df["dt"].to_pandas(use_pyarrow_extension_array=True), format=format
)
with capsys.disabled():
print(t)
assert (t == t_ref).all()
@pytest.mark.parametrize("format", dt_formats)
def test_datetime(df, format, capsys):
dt_ref = datetime(2024, 6, 3, 20, 2, 48, 680000)
dt = datetime.strptime(dt_string, format)
assert dt == dt_ref |
@tritemio correct, python datetime and chrono have different behaviour for some details. you can check out the documentation Watch our for this (chrono)date = "2020-01-01 10:00:00.1234"
|
Checks
Reproducible example
Log output
Issue description
Converting from string to datetime with a format string should allow to decode custom formats.
In this example the input has 7 digits for fractional seconds after the decimal dot. However the last digit is always zero and should be ignored because there is a trailing 0 in the format string.
Instead, polars gives the above error during conversion.
Stripping the extra zero from the string before attempting the conversion works correctly in polars:
Pandas accepts the original format and convert the string correctly, as does python's
datetime
Expected behavior
The column should be converted to datetime without error, as done in pandas and datetime from python standard lib.
Installed versions
The text was updated successfully, but these errors were encountered: