Skip to content

mysql: decode DATETIME/TIMESTAMP as UTC to match the UTC-based encode#31212

Merged
alii merged 1 commit into
mainfrom
ali/mysql-datetime-utc-roundtrip
May 30, 2026
Merged

mysql: decode DATETIME/TIMESTAMP as UTC to match the UTC-based encode#31212
alii merged 1 commit into
mainfrom
ali/mysql-datetime-utc-roundtrip

Conversation

@alii

@alii alii commented May 22, 2026

Copy link
Copy Markdown
Member

Fixes #29208

A JS Date bound to a MySQL DATETIME/TIMESTAMP and read back was silently shifted by the machine's UTC offset on non-UTC machines (e.g. +240 min under TZ=America/New_York). Encode and decode used different timezone conventions: from_unix_timestamp breaks the Date's UTC epoch into Y/M/D h:m:s with pure-UTC arithmetic (writing the UTC wall-clock), but to_js_timestamp decoded via the local-time conversion, interpreting the stored wall-clock as the client's local time.

Decode the stored wall-clock as UTC (gregorian_date_time_to_ms_utc) so both sides agree, giving a timezone-independent identity round-trip — consistent with how Bun's Postgres driver already handles dates.

Adds a mock-MySQL-server round-trip test run under UTC / America/New_York / Asia/Tokyo. (Not a port regression — the Zig path has the same asymmetry.)

@alii

alii commented May 22, 2026

Copy link
Copy Markdown
Member Author

@robobun adopt

@robobun

robobun commented May 22, 2026

Copy link
Copy Markdown
Collaborator
Updated 1:29 PM PT - May 30th, 2026

@robobun, your commit bfeb3cb has 3 failures in Build #59359 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 31212

That installs a local version of the PR into your bun-31212 executable, so you can run:

bun-31212 --bun

@github-actions

Copy link
Copy Markdown
Contributor

Found 1 issue this PR may fix:

  1. Bug: Bun.SQL Date values returned from DATETIME/TIMESTAMP columns deserialized as local time instead of UTC #29208 - Reports exactly this bug: Bun.SQL Date values from DATETIME/TIMESTAMP columns are deserialized as local time instead of UTC, causing shifted dates on non-UTC machines

If this is helpful, copy the block below into the PR description to auto-close this issue on merge.

Fixes #29208

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6ca92bf2-e4e6-452e-8782-f1baab1f159f

📥 Commits

Reviewing files that changed from the base of the PR and between 1c2ab23 and bfeb3cb.

📒 Files selected for processing (1)
  • docs/runtime/sql.mdx

Walkthrough

This PR fixes a bug where MySQL DATETIME and TIMESTAMP columns were deserialized using local-time instead of UTC conversion, causing epoch mismatches on non-UTC machines. The fix switches both Rust and Zig converters to UTC-aware gregorian-to-milliseconds functions, adds a timezone-roundtrip test fixture to validate correctness, and updates documentation to clarify the UTC-based decoding behavior.

Changes

DateTime UTC Round-trip Fix

Layer / File(s) Summary
DateTime UTC Conversion in Rust and Zig
src/sql_jsc/mysql/MySQLValue.rs, src/sql_jsc/mysql/MySQLValue.zig
DateTime::to_js_timestamp and DateTime.toJSTimestamp now call UTC-aware helpers (gregorian_date_time_to_ms_utc / gregorianDateTimeToMSUTC) instead of non-UTC variants, with comments clarifying that stored wall-clock values are treated as UTC for correct round-trip encoding and decoding.
MySQL DateTime Round-trip Test Fixture
test/js/sql/sql-mysql-datetime-tz-fixture.ts
New test fixture connects to MySQL, creates a temporary table, inserts fixed UTC Date instants covering DST and boundary cases, reads them back as Date objects, and asserts exact epoch-ms equality; logs timezone context and fails on any mismatch.
Documentation of UTC Decoding Behavior
docs/runtime/sql.mdx
Type mapping documentation updated to specify that DATETIME and TIMESTAMP are decoded as UTC with matching stored UTC wall-clock values, 0000-00-00 becomes Invalid Date, and contrasts this with PostgreSQL timestamp (no time zone) and timestamptz (explicit offset).
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: converting MySQL DATETIME/TIMESTAMP decoding from local-time to UTC to match the encoding behavior.
Description check ✅ Passed The PR description provides both what it does (fixes timezone deserialization asymmetry) and how it was verified (mock-MySQL round-trip test under multiple timezones), matching the template's required sections.
Linked Issues check ✅ Passed The PR addresses all core objectives from #29208: converts DATETIME/TIMESTAMP decoding to UTC, ensures timezone-independent round-trips, covers both MySQL DATETIME and TIMESTAMP, and provides comprehensive test coverage across UTC, America/New_York, and Asia/Tokyo timezones.
Out of Scope Changes check ✅ Passed All changes are directly in scope: MySQL codec updates (Rust/Zig), test fixture for UTC round-trip verification, and documentation clarification for DATETIME/TIMESTAMP UTC decoding behavior. No unrelated changes detected.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. sql(mysql): add utcDate option for UTC DATETIME/TIMESTAMP decoding #29216 - Addresses the same MySQL DATETIME/TIMESTAMP timezone bug (decoded as local time instead of UTC). Uses a different approach (opt-in utcDate option) vs this PR which changes the default to UTC.

🤖 Generated with Claude Code

@robobun

robobun commented May 22, 2026

Copy link
Copy Markdown
Collaborator

✅ Fixes naive DATETIME/TIMESTAMP decoding to be UTC across all protocols, so a Date round-trips to the same instant regardless of the machine timezone (bfeb3cb):

  • MySQL binary + text (.simple()) DATETIME/TIMESTAMP decode as UTC; zero-dates (0000-00-00) now surface as Invalid Date on both paths.
  • Postgres timestamp (WITHOUT TIME ZONE) text decodes as UTC to match its binary path; timestamptz/date unchanged (already correct).

Verified against real MySQL and Postgres servers (docker-compose in CI, local otherwise) across Etc/UTC / America/New_York / Asia/Tokyo: binary vs text agree, and the round-trip fails without the fix.

Fixes the MySQL DATETIME/TIMESTAMP (and Postgres TIMESTAMP) case in #29208.

The SQL diff is green — all ASAN test lanes (which run these tests) pass. The only red CI lanes are unrelated to this change: a macOS-x64 AsyncLocalStorage segfault and a terminal.test.ts PTY timeout (both in subsystems this SQL-only diff does not touch), plus a Docker container-name race in the MySQL harness that passed on retry. Ready for a maintainer to merge (a re-run of the macOS lane should go green).

Comment thread src/sql_jsc/mysql/MySQLValue.rs
Comment thread test/js/sql/sql-mysql-datetime-roundtrip.test.ts Outdated
Comment thread test/js/sql/sql-mysql-datetime-roundtrip.test.ts Outdated
@alii

alii commented May 27, 2026

Copy link
Copy Markdown
Member Author

Reworked to a real MySQL server — spawns a fixture under TZ=UTC/America-New_York/Asia-Tokyo and asserts the DATETIME round-trip is identity in each. @robobun adopt

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/sql/sql-mysql.test.ts`:
- Around line 67-78: Replace the parameterized test with describe.each instead
of test.each and make the assertion verify the actual timezone token;
specifically, change the table-runner from test.each to describe.each and update
the inner test that calls bunRun(path.join(import.meta.dir,
"sql-mysql-datetime-tz-fixture.ts"), {..., TZ}) so it expects the child stdout
to include the TZ marker (e.g. expect(stdout).toContain(`OK TZ=${TZ}`)) rather
than just "OK", ensuring the fixture/sql-mysql-datetime-tz-fixture.ts emits the
`OK TZ=${TZ}` token.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d689445a-16df-4814-a897-bdf7ef81882e

📥 Commits

Reviewing files that changed from the base of the PR and between a6968bf and 1c2ab23.

📒 Files selected for processing (2)
  • test/js/sql/sql-mysql-datetime-tz-fixture.ts
  • test/js/sql/sql-mysql.test.ts

Comment thread test/js/sql/sql-mysql.test.ts Outdated
Comment thread test/js/sql/sql-mysql.test.ts Outdated
@robobun robobun force-pushed the ali/mysql-datetime-utc-roundtrip branch 2 times, most recently from 3208488 to ec612d3 Compare May 27, 2026 19:26
@alii

alii commented May 28, 2026

Copy link
Copy Markdown
Member Author

@robobun use the real mysql instance in your container and also in Ci rather than a mock server.

@robobun robobun force-pushed the ali/mysql-datetime-utc-roundtrip branch from bc8c311 to ac7238f Compare May 28, 2026 13:52
Comment thread test/js/sql/sql-mysql-datetime-roundtrip.test.ts Outdated
@robobun robobun force-pushed the ali/mysql-datetime-utc-roundtrip branch from ac7238f to f97f7d8 Compare May 28, 2026 13:59
@alii

alii commented May 30, 2026

Copy link
Copy Markdown
Member Author

@robobun, #29216 implemented a flag to not break behaviour. Since 1.4 is coming up, do you think it would be acceptable to make this slightly breaking change without the flag? What else was in the previous pull request that we might want to add here? Just do research and leave a comment first, before we commit to any further implementation.

@robobun

robobun commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Research only — no code changes yet. Here's where things stand and what #29216 (closed) covered that this PR doesn't.

Is dropping the flag acceptable for 1.4?

I think yes, unconditional is the right callutcDate: false-by-default from #29216 mostly preserved a bug (it kept the decode wrong relative to the encode), and a default-off flag would mean almost nobody gets the fix. The change is "breaking" only for code that was already silently relying on the wrong value, and the direction is strictly toward correctness:

  • Encode has always been UTC. DateTime::from_js/from_unix_timestamp break a Date's epoch into Y/M/D h:m:s with pure-UTC arithmetic, so what gets written is the UTC wall-clock. This PR just makes decode agree, so a Date round-trips to the same instant regardless of TZ.
  • It matches what we already do for Postgres. The PG binary path (postgres/types/date.rs::from_binary) decodes µs-since-2000 straight to UTC epoch ms — no local-time step. So after this PR, MySQL binary decode lines up with PG binary decode.
  • Who could be affected: only apps on a non-UTC server that (a) read DATETIME/TIMESTAMP via the prepared path and (b) were compensating for the old local-shift. For a .0/major like 1.4, a one-line changelog note ("MySQL DATETIME/TIMESTAMP now decode as UTC to match the write path; set your session/connection TZ if you need local") seems proportionate.

What #29216 did that this PR does NOT (all still gaps on the live Rust path)

#29216 targeted the old Zig tree (src/sql/mysql/**) and was never ported to the live Rust tree (src/sql_jsc/**). Beyond the binary-path UTC flip (which this PR already does at MySQLValue.rs:603), it also covered:

  1. Text/simple-protocol decode. sql.simple() and sql.unsafe(str) (no params) go through the text protocol, which still decodes via parse_date → JSC Date.parse (ResultSet.rs:234-251). An unzoned "2024-06-15 12:00:00" is parsed as local time there. So after this PR, binary decodes as UTC but text still decodes as local — the two protocols would disagree for the same column on a non-UTC host. sql(mysql): add utcDate option for UTC DATETIME/TIMESTAMP decoding #29216 added a DateTime.fromText that parses YYYY-MM-DD[ HH:MM:SS[.ffffff]] and routes it through the same UTC conversion. There is no Rust equivalent today (days_in_month/is_leap_year helpers exist in MySQLValue.rs to build one). This is the bigger remaining gap and IMO should land together with the binary change so the two paths stay consistent.

  2. Zero / partial-zero dates. MySQL permits '0000-00-00' and partials like '2024-00-15' (month 0) / '2024-06-00' (day 0). The binary path copies month/day raw with no validation (MySQLValue.rs from_binary), and gregorian_date_time_to_ms* → WTF GregorianDateTime silently wraps month 0 to December of the prior year (garbage Date, not Invalid Date). Meanwhile the text path's Date.parse returns Invalid Date for "0000-00-00" — so the two paths already disagree on zero-dates regardless of TZ. sql(mysql): add utcDate option for UTC DATETIME/TIMESTAMP decoding #29216 added a guard that surfaces these as Invalid Date (NaN). Worth adding; cheap (a range check in to_js_timestamp).

  3. Docs + types. sql(mysql): add utcDate option for UTC DATETIME/TIMESTAMP decoding #29216 touched docs/runtime/sql.mdx and packages/bun-types/sql.d.ts for the utcDate option — not needed if we go flagless, but the behavior change is worth a line in the docs/changelog.

Suggested scope for the flagless version

  • Keep the binary UTC flip (done).
  • Add the text-path UTC decode (a Rust from_text) so prepared and simple queries agree — this is the one I'd consider a must-have, since shipping binary-only re-introduces a protocol-dependent inconsistency.
  • Add the zero-date → Invalid Date guard on both paths so they agree there too.
  • Changelog note about the decode change; no utcDate option.

Open question for you: do we also want to fix the Postgres text path (DataCell.rs simple-query timestamp/arrays also go through parse_date → local), or keep this PR MySQL-only and track PG separately? PG binary is already UTC-correct, so PG is only inconsistent on simple queries / array elements.

@alii

alii commented May 30, 2026

Copy link
Copy Markdown
Member Author

Let's do it. PG can also go in this PR for correctness.

@robobun robobun force-pushed the ali/mysql-datetime-utc-roundtrip branch from f97f7d8 to 36cfe27 Compare May 30, 2026 17:22
@robobun

robobun commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Done — implemented the full scope in 36cfe27 (no flag; unconditional UTC decode):

MySQL

  • Binary DATETIME/TIMESTAMP: UTC (as before).
  • Text / .simple(): new DateTime::from_text parses the components and converts as UTC, replacing the Date.parse (local-time) path so prepared and simple queries agree.
  • Zero / partial-zero dates (0000-00-00, month/day 0): now Invalid Date on both paths (binary zero-length payload + text), instead of the Unix epoch / a wrapped Date.

Postgres

  • timestamp (WITHOUT TIME ZONE) text: decode components as UTC to match its binary path.
  • timestamptz (explicit offset) and date (UTC midnight) keep decoding via Date.parse — they were already correct.

Tests (real servers — docker-compose in CI, local instance otherwise; no mock): binary-vs-text consistency and zero-dates across Etc/UTC / America/New_York / Asia/Tokyo for both MySQL and Postgres. Verified fail-before / pass-after locally against real MySQL + Postgres (5 fail → 6 pass on the non-UTC lanes once the fix is applied).

I'll add a changelog/docs note about the decode change. Left timestamptz/date untouched since they were already UTC-correct — shout if you'd rather I normalize those through the component path too.

…write path

A JS Date bound to a MySQL DATETIME/TIMESTAMP (or Postgres timestamp) and read
back was silently shifted by the machine's UTC offset on non-UTC machines.
Encode breaks the Date's UTC epoch into Y/M/D h:m:s with pure-UTC arithmetic,
but decode used the local-time conversion. Decode the stored wall-clock as UTC
so both sides agree, on every protocol:

- MySQL binary DATETIME/TIMESTAMP: gregorian_date_time_to_ms_utc.
- MySQL text (.simple()): parse components via DateTime::from_text and convert
  as UTC, instead of JS Date.parse (which read them as local time).
- MySQL zero / partial-zero dates ('0000-00-00', month/day 0) surface as
  Invalid Date on both paths instead of the Unix epoch / a wrapped Date.
- Postgres 'timestamp' (WITHOUT TIME ZONE) text: decode as UTC to match its
  binary path; 'timestamptz' (explicit offset) and 'date' keep using Date.parse.

Tested against real MySQL and Postgres servers (docker-compose in CI, or a
local instance) — binary vs text consistency and zero-dates across
UTC / America/New_York / Asia/Tokyo.
@robobun robobun force-pushed the ali/mysql-datetime-utc-roundtrip branch from 36cfe27 to bfeb3cb Compare May 30, 2026 17:24
@alii alii merged commit 61bd997 into main May 30, 2026
75 of 77 checks passed
@alii alii deleted the ali/mysql-datetime-utc-roundtrip branch May 30, 2026 23:38
alii added a commit that referenced this pull request Jun 1, 2026
…gres (#31663)

Follow-up to #31212.

## What this does

#31212 fixed naive `DATETIME`/`TIMESTAMP` text decoding to be UTC, but
introduced two near-identical hand-rolled parsers for the `YYYY-MM-DD
HH:MM:SS[.ffffff]` wall-clock format:

- `DateTime::from_text` in `src/sql_jsc/mysql/MySQLValue.rs`
- `timestamp_text_to_ms_utc` in `src/sql_jsc/postgres/types/date.rs`

Both had the same digit accumulator, the same separator validation, and
the same fractional-second padding. This PR extracts the structural
parsing into one shared module, `src/sql_jsc/shared/datetime_text.rs`,
and reduces both call sites to thin wrappers.

No existing helper could be reused instead: the workspace has no date
crates, and the only WTF date function exposed to Rust (`parse_date`) is
JS `Date.parse` — the local-time behavior #31212 exists to avoid.

## What stays driver-specific (deliberately)

- **MySQL** (`parse_mysql`): accepts the 10-byte date-only form and a
`T` separator, validates calendar dates (`days_in_month`) and h/m/s
ranges, keeps microsecond precision.
- **Postgres** (`parse_postgres_timestamp`): requires the full
`YYYY-MM-DD HH:MM:SS` shape — anything else returns `None` so the caller
falls back to `Date.parse` (`infinity`, BC dates, 5+ digit years); no
range validation (delegated to `gregorianDateTimeToMS`); millisecond
precision.

Public signatures of both wrapped functions are unchanged, so no call
sites move.

## New test coverage

The existing roundtrip tests skip their assertions when no real
MySQL/Postgres server is reachable, which left the text-protocol date
decode path untested without Docker. This PR adds a mock-MySQL-server
test (same approach as `sql-mysql-raw-length-prefix.test.ts`) that
exercises text-protocol `DATE`/`DATETIME` decode across `Etc/UTC` /
`America/New_York` / `Asia/Tokyo`, including zero dates and impossible
calendar dates.

## How we know it works

- `USE_SYSTEM_BUN=1 bun test
test/js/sql/sql-mysql-datetime-roundtrip.test.ts` → **3 fail** on Bun
1.4.0 (text dates read as local time; `2024-02-31` wraps to March 2
instead of Invalid Date)
- `bun bd test test/js/sql/sql-mysql-datetime-roundtrip.test.ts` → **6
pass**
- Full SQL suite (`sql-mysql-mediumint`, `sql-mysql-raw-length-prefix`,
`sql-mysql-bind-oob`, both datetime roundtrip files) passes with this
change.
- The CI docker lanes run the real-server roundtrip tests, which assert
binary/text agreement for both drivers across the same three timezones.
robobun added a commit that referenced this pull request Jun 1, 2026
Merging main brought #31212's decode-side calendar validation
(DateTime::from_text / to_js_timestamp), which calls days_in_month — a
helper this branch had removed together with the loop-based
gregorian_date. Restore is_leap_year/days_in_month for the decoder; the
Hinnant-based encode path still doesn't need them.

Drop the file-level process.env.TZ pin from sql-mysql.test.ts: its own
comment said it could go once the decoder moved to UTC, which #31212 did.
Main's TZ-parametrized round-trip tests (UTC / New_York / Tokyo) pass
without it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Bun.SQL Date values returned from DATETIME/TIMESTAMP columns deserialized as local time instead of UTC

3 participants