Skip to content

sql(postgres): declare text format for Bind parameters that have no binary encoder - #41912

Closed
robobun wants to merge 5 commits into
mainfrom
robobun/ef1b5e56/postgres-bind-format-code
Closed

robobun wants to merge 5 commits into
mainfrom
robobun/ef1b5e56/postgres-bind-format-code

Conversation

@robobun

@robobun robobun commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • sql({ ids: sql.array([1, 2], "INT4") }) into an int4[] column fails with 08P01 insufficient data left in message. A decimal.js style object bound to numeric fails with 08P01, to time with 22008 time out of range. Bound to real, "1234" stores 2.5931515e-09.
  • Cause: write_bind (src/sql_jsc/postgres/PostgresRequest.rs) took each parameter's format code from Tag::is_binary_format_supported(), the decode-side list. Binary encoders only exist for bool, int4, float8, timestamp(tz) and bytea. numeric, float4, time and float4[] were declared binary and written as String(value). The int4_array arm wrote one int4.
  • Result columns had the same split: FieldDescription::type_tag() truncated the OID to 16 bits for the format code, while DataCell mapped an OID above 65535 to text.

Fix

  • A ParamEncoding enum lists the encoders that exist. write_bind decides it once per parameter and writes the format code and the value bytes from it. Types without a binary encoder go out as text, as string values already do.
  • Tag::from_oid() is now the only OID-to-Tag mapping, for parameters, result format codes and DataCell.
  • Verified: test/js/sql/postgres-bind-param-format.test.ts (new, mock server plus PostgreSQL 17; 6 of 8 tests fail on bun 1.4.3). Also sql.test.ts, sql-prepare-false.test.ts and every postgres-*.test.ts.

Background

  • Parse carries a type OID per parameter (0 = server decides). Bind carries a format code per parameter (0 text, 1 binary), the value bytes, then a format code per result column. With the default prepare: true, Bun binds with the types the server reports.
  • Bun declares an OID only for JS numbers, booleans and typed arrays. Objects, Date and sql.array() values get OID 0, so the server infers numeric, real, time or int4[] from the column or cast.
  • DataCell decodes numeric, float4, time, int4[] and float4[] from binary. Nothing encodes them.
Notes
  • Behavior for values that hit the affected parameter types, against PostgreSQL 17 (before / after):
    • sql({ ia: sql.array([1,2], "INT4") }) into int4[]: 08P01 / {1,2}
    • sql({ ra: sql.array([1.5,2.5], "REAL") }) into real[]: 54000 / {1.5,2.5}
    • { toString: () => "19.99" } into numeric: 08P01 / 19.99
    • { toString: () => "1234" } into real: stores 2.5931515e-09 / 1234
    • { toString: () => "12:34:56" } into time: 22008 / 12:34:56
    • [1, 2] into int4[]: 08P01 / 22P02 malformed array literal: "1,2" (postgres: encode array parameters as text array literals #33579 makes plain JS arrays work and builds on the text format)
    • a Date into time: 22008 time out of range / 22007 invalid input syntax for type time: "Thu Jan 01 1970 ..." (the Date text form is a separate issue, see postgres: encode Date and object parameters correctly with prepare: false #39452)
  • sql.array() in a tagged template was not affected: bindParam pushes the serialized string and appends a $1::TYPE[] cast, and a string value was already sent as text. The sql(object) helper and sql.unsafe() bind the SQLArrayParameter object itself.
  • Result columns: a user-defined type whose OID aliases a binary-decoded builtin modulo 65536 (for example 65552 = 65536 + 16, bool) was requested in binary and then read as text. The mock test a result column whose OID is above 65535 is requested and decoded as text covers it.
  • The encoding is recorded while the format codes are written and reused for the value section. Before, both sections looked at the value separately, so a toString() or getter that changed another bound value in between could still produce a format/bytes mismatch. A probe with such a value now round-trips.
  • sql(postgres): send non-integer number params as text with OID 0 so NUMERIC stays exact #35508 (numeric precision for JS numbers) contains a similar format-code change as a side effect of a larger behavior change. This PR is the standalone fix and does not change how JS numbers are declared or encoded. postgres: encode array parameters as text array literals #33579, postgres: encode Date and object parameters correctly with prepare: false #39452 and postgres: send a string parameter bound to json/jsonb verbatim #40944 edit the same match; whichever lands second needs a small rebase. Rebased on top of sql(postgres): reject a non-BufferSource value bound to a bytea parameter #41889 (bytea type check), which already landed.
  • wire-frames.ts gains pgNoData() and pgDecodeBind() for the mock tests, with a self-test in wire-frames.test.ts.
  • The local sql.test.ts run has the same failures with and without this change. They come from the local server setup (SQL_ASCII encoding, no md5/scram roles, prepared transactions disabled, PG 17 pg_database shape) and debug-build timeouts, not from parameter binding or result decoding.
  • Self-reviewed: 2 concerns raised, 2 addressed (the result-column OID mapping, and deciding the encoding once per parameter).

[human-review] gate passed · iteration 0 · 7 files touched

fails on main (without fix)
ASAN without fix: 6 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/sql/postgres-bind-param-format.test.ts test/js/sql/wire-frames.test.ts
bun test v1.4.3 (f42e98025)

test/js/sql/wire-frames.test.ts:
(pass) pgDecodeBind decodes a §55.7 Bind body [21.39ms]
(pass) mysqlLenencInt encodes per page_protocol_basic_dt_integers.html [9.69ms]
(pass) pgErrorResponse encodes per §55.7 [7.88ms]
(pass) postgres: pgAuthenticationOk + pgReadyForQuery are accepted by Bun's parser [406.29ms]
(pass) postgres: COPY OUT response frames are consumed and the following result set decodes [243.90ms]
(pass) postgres: pgMinimalReadyServer satisfies connect() [43.47ms]
(pass) mysql: mysqlHandshakeV10 + mysqlOkPacket are accepted by Bun's parser [74.89ms]

test/js/sql/postgres-bind-param-format.test.ts:
90 |     });
91 |     expect(binds).toHaveLength(1);
92 |     expect({
93 |       paramFormats: binds[0].paramFormats,
94 |       params: binds[0].params.map(p => p?.toString("latin1")),
95 |     }).toEqual({
            ^
error: expect(received).toEqual(expected)

  {
    "paramFormats": [
-     0,
-     0,
-     0,
-  
... (truncated)

release without fix: 6 FAILED
bun test v1.4.3-canary.1 (f42e98025)

test/js/sql/wire-frames.test.ts:
(pass) pgDecodeBind decodes a §55.7 Bind body [0.29ms]
(pass) mysqlLenencInt encodes per page_protocol_basic_dt_integers.html [0.11ms]
(pass) pgErrorResponse encodes per §55.7 [0.10ms]
(pass) postgres: pgAuthenticationOk + pgReadyForQuery are accepted by Bun's parser [6.29ms]
(pass) postgres: COPY OUT response frames are consumed and the following result set decodes [3.60ms]
(pass) postgres: pgMinimalReadyServer satisfies connect() [0.63ms]
(pass) mysql: mysqlHandshakeV10 + mysqlOkPacket are accepted by Bun's parser [1.50ms]

test/js/sql/postgres-bind-param-format.test.ts:
90 |     });
91 |     expect(binds).toHaveLength(1);
92 |     expect({
93 |       paramFormats: binds[0].paramFormats,
94 |       params: binds[0].params.map(p => p?.toString("latin1")),
95 |     }).toEqual({
            ^
error: expect(received).toEqual(expected)

  {
    "paramFormats": [
-     0,
-     0,
-     0,
-     0,
-     0,
+     1,
+     1,
+     1,
+     1,
+     1,
    ],
    "params": [
      "19.99",
      "1234",
      "12:34:56",
-     "{"1","2"}",
+     "",
      "{1.5}",
    ],
  }

- Expected  - 6
+ R
... (truncated)
passes on PR (with fix)
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/sql/postgres-bind-param-format.test.ts test/js/sql/wire-frames.test.ts
bun test v1.4.3 (f42e98025)

test/js/sql/wire-frames.test.ts:
(pass) pgDecodeBind decodes a §55.7 Bind body [18.19ms]
(pass) mysqlLenencInt encodes per page_protocol_basic_dt_integers.html [11.13ms]
(pass) pgErrorResponse encodes per §55.7 [6.91ms]
(pass) postgres: pgAuthenticationOk + pgReadyForQuery are accepted by Bun's parser [410.90ms]
(pass) postgres: COPY OUT response frames are consumed and the following result set decodes [240.77ms]
(pass) postgres: pgMinimalReadyServer satisfies connect() [31.71ms]
(pass) mysql: mysqlHandshakeV10 + mysqlOkPacket are accepted by Bun's parser [80.37ms]

test/js/sql/postgres-bind-param-format.test.ts:
(pass) Bind format codes (mock server) > parameters typed numeric, float4, time, int4[] and float4[] are declared and sent as text [207.04ms]
(pass) Bind format codes (mock server) > parameters with a binary encoder are declared binary, a string value stays text [40.22ms]
(pass) Bind format codes (mock server) > a resul
... (truncated)

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 609ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[0/5] cargo bun_runtime → libbun_runtime.a
�[1m�[92m   Compiling�[0m bun_core v0.0.0 (/workspace/bun/src/bun_core)
�[1m�[92m   Compiling�[0m bun_errno v0.0.0 (/workspace/bun/src/errno)
�[1m�[92m   Compiling�[0m bun_ptr v0.0.0 (/workspace/bun/src/ptr)
�[1m�[92m   Compiling�[0m bun_boringssl_sys v0.0.0 (/workspace/bun/src/boringssl_sys)
�[1m�[92m   Compiling�[0m bun_safety v0.0.0 (/workspace/bun/src/safety)
�[1m�[92m   Compiling�[0m bun_base64 v0.0.0 (/workspace/bun/src/base64)
�[1m�[92m   Compiling�[0m bun_cares_sys v0.0.0 (/workspace/bun/src/cares_sys)
�[1m�[92m   Compiling�[0m bun_zlib_sys v0.0.0 (/workspace/bun/src/zlib_sys)
�[1m�[92m   Compiling�[0m bun_zstd v0.0.0 (/workspace/bun/src/zstd)
�[1m�[92m   Compiling�[0m bun_picohttp v0.0.0 (/workspace/bun/src/picohttp)
�[1m�[92m   Compiling�[0m bun_brotli v0.0.0 (/workspace/bun/src/brotli)
�[1m�[92m   Compiling�[0m bun_output v0.0.0 (/workspace/bun/src/output)
�[1m�[92m   Compiling�[0m bun_clap v0.0.0 (/workspace/bun/src/clap)
�[1m�[92m   Compiling�[0m bu
... (truncated)
diff hotspot
src/sql/postgres/protocol/FieldDescription.rs  |   6 +-
 src/sql/postgres/types/Tag.rs                  |   8 +-
 src/sql_jsc/postgres/DataCell.rs               |   8 +-
 src/sql_jsc/postgres/PostgresRequest.rs        | 171 ++++++++++----------
 test/js/sql/postgres-bind-param-format.test.ts | 214 +++++++++++++++++++++++++
 test/js/sql/wire-frames.test.ts                |  32 ++++
 test/js/sql/wire-frames.ts                     |  53 ++++++
 7 files changed, 399 insertions(+), 93 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                            reads  edits  tests
src/sql/postgres/protocol/FieldDescription.rs       1      1     28
src/sql/postgres/types/Tag.rs                       3      4     29
src/sql_jsc/postgres/DataCell.rs                    1      1     28
src/sql_jsc/postgres/PostgresRequest.rs             9     16     28
test/js/sql/postgres-bind-param-format.test.ts      0      4     24
test/js/sql/wire-frames.test.ts                     1      1      9
test/js/sql/wire-frames.ts                          2      1     29

@coderabbitai

coderabbitai Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: dc9888af-27b9-43f6-b19d-826fed927226

📥 Commits

Reviewing files that changed from the base of the PR and between b2159f6 and b42cdb3.

📒 Files selected for processing (2)
  • src/sql/postgres/types/Tag.rs
  • src/sql_jsc/postgres/PostgresRequest.rs

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


Walkthrough

PostgreSQL OID conversion now uses a shared Tag::from_oid path. Parameter binding computes encodings once for format codes and serialization. New wire-frame, mock-server, and container tests cover text and binary binding behavior.

Changes

PostgreSQL encoding flow

Layer / File(s) Summary
Shared OID-to-tag conversion
src/sql/postgres/types/Tag.rs, src/sql/postgres/protocol/FieldDescription.rs, src/sql_jsc/postgres/DataCell.rs
Tag::from_oid converts representable OIDs and falls back to Tag::text. Field descriptions and data-cell serialization use this conversion.
Effective parameter encoding
src/sql_jsc/postgres/PostgresRequest.rs
write_bind computes one ParamEncoding per parameter and reuses it for format codes and value serialization.
Wire and integration validation
test/js/sql/postgres-bind-param-format.test.ts, test/js/sql/wire-frames.ts, test/js/sql/wire-frames.test.ts
Tests and helpers validate Bind decoding, text and binary formats, OID fallback, arrays, nulls, and database round trips.

Suggested reviewers: jarred-sumner, dylan-conway

Merge Risk: ⚪ Minimal · up to 50144

PostgreSQL parameters without binary encoders now use text consistently, preventing binding failures and value corruption while preserving supported binary encodings. The affected paths are covered by wire-level and database round-trip tests, with no current merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: PostgreSQL Bind parameters without binary encoders are declared with text format.
Description check ✅ Passed The description explains the problem, root cause, fix, affected types, OID handling, and verification results. It does not use the exact template headings, but it provides the required information and…

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the claude label Sep 8, 2026
@robobun

robobun commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status

Reproduced on bun 1.4.3 and on a main debug build against PostgreSQL 17, with the default prepare: true:

import { SQL } from "bun";
const sql = new SQL({ url, max: 1 });
await sql`create temp table s (r real, n numeric, t time, ia int4[])`;
await sql`insert into s ${sql({ ia: sql.array([1, 2], "INT4") })}`;        // 08P01 insufficient data left in message
await sql`insert into s (n) values (${{ toString: () => "19.99" }})`;       // 08P01 insufficient data left in message
await sql`insert into s (r) values (${{ toString: () => "1234" }}) returning r::text`; // stores 2.5931515e-09
await sql`insert into s (t) values (${{ toString: () => "12:34:56" }})`;    // 22008 time out of range

test/js/sql/postgres-bind-param-format.test.ts covers these with mock-server tests (exact Bind bytes) and container tests: 6 of 8 fail on bun 1.4.3 and all pass with this branch.

Rebased onto main after #41889 landed in the same function. The self-review raised two points, both addressed in 97c62ca: the result-column OID mapping now goes through the same Tag::from_oid(), and each parameter's encoding is decided once and reused for the value bytes.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the format-code and value-encoding sections of the Bind message now both derive from ParamEncoding, so they structurally cannot disagree.

What was reviewed:

  • Traced each previously-working tag (bool/int4/float8/timestamp[tz]/bytea/json[b]) through for_tag/for_value — same wire bytes as before; the string→text downgrade is preserved.
  • param_tag is behaviorally identical to the old Short::MAX guard (Int4=u32, Short=u16), just without the post-guard .unwrap().
  • Checked NULL and iter.next() == None in the format-code loop — format code stays binary but the value section writes -1 with no bytes, which Postgres accepts regardless of format.
  • Test file uses describeWithContainer (local container, no external network), await using, whole-object toEqual, and matches the existing postgres-*.test.ts layout in test/js/sql/.
Extended reasoning...

Overview

This PR fixes a class of Postgres Bind-message bugs where write_bind in src/sql_jsc/postgres/PostgresRequest.rs declared parameters as binary (format code 1) using the decode-side predicate Tag::is_binary_format_supported(), but then encoded numeric/float4/time/int4[]/float4[] values as text — causing 08P01/22008/54000 errors or silent garbage. The fix introduces a ParamEncoding enum enumerating exactly the encoders that exist (Text, Json, Bool, Int4, Float8, Timestamp, Bytea) and drives both the format-code loop and the value-encoding switch from it. The dead int4_array arm (which wrote a single int4, not an array) is removed, param_tag() collapses the duplicated OID→Tag logic, and Tag.rs gets a doc comment clarifying is_binary_format_supported is decode-side only. A new container-backed test file exercises numeric/real/time columns, int4[]/real[] via sql(object) and sql.unsafe(), and round-trips the true binary-encoded types.

Security risks

None. This is client-side wire-protocol encoding for outbound Bind messages; no auth, crypto, permissions, or untrusted-input parsing is touched. The only trust-boundary value here is the server-reported OID, and the new param_tag handles out-of-u16-range OIDs by falling back to text (identical to the prior Short::MAX guard, but without the post-guard .unwrap()).

Level of scrutiny

Medium — protocol encoding is correctness-critical, but the change is narrow and mechanical. I verified behavior preservation for every previously-handled tag: json/jsonb → Json (format 0, JSON.stringify), bool/int4/float8/timestamp/timestamptz/bytea → same binary arms, and the value.is_string() → text downgrade is preserved via for_value. Int4/Short are u32/u16, so Short::try_from fails exactly when the old (Short::MAX as Int4) < parameter_field check did. The NULL path (is_empty_or_undefined_or_null → write -1) is unchanged and is format-code-agnostic per the protocol. The refactor is a direct application of REVIEW.md's "one source of truth" rule — encode/decode were sharing a predicate that only fit decode, and the two Bind sections now cannot diverge.

Other factors

The bug hunter exited dry_streak with no findings. No CODEOWNERS entry covers src/sql/ or test/js/sql/. The new test file follows the established test/js/sql/postgres-*.test.ts per-topic pattern, uses describeWithContainer from harness (local Postgres, no external network), await using for cleanup, whole-object toEqual assertions, and no sleeps or hardcoded ports. The PR description is thorough and honest about scope (interaction with #35508, remaining Date→time and plain-array cases tracked separately, unrelated pre-existing sql.test.ts failures). The timeline shows a single commit with no prior review activity.

@robobun

robobun commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:58 AM PT - Sep 8th, 2026

✅ @robobun, your commit 50144ba0b53d8bb062b24ad4243a85d0858ced4f passed in Build #112860! 🎉


🧪   To try this PR locally:

bunx bun-pr 41912

That installs a local version of the PR into your bun-41912 executable, so you can run:

bun-41912 --bun

Comment thread src/sql/postgres/types/Tag.rs Outdated
Comment thread src/sql/postgres/types/Tag.rs Outdated
Comment thread src/sql_jsc/postgres/PostgresRequest.rs Outdated
Comment thread src/sql_jsc/postgres/PostgresRequest.rs Outdated
Comment thread src/sql_jsc/postgres/PostgresRequest.rs Outdated
Comment thread src/sql_jsc/postgres/PostgresRequest.rs Outdated
…inary encoder

write_bind picked each parameter's format code from
Tag::is_binary_format_supported(), the list of types DataCell can decode
from binary. The encoder only has binary arms for bool, int4, float8,
timestamp(tz) and bytea. A parameter the server typed as numeric, real,
time or real[] was declared binary but written as String(value), and an
int4[] parameter was written as a single int4. The server then read text
bytes as the binary representation: 08P01, 22P03 or 22008 errors, or a
silently wrong real value.

One ParamEncoding enum now drives both the format-code section and the
value section of the Bind message. Types without a binary encoder are
sent as text and parsed by the server.
… decide each Bind encoding once

FieldDescription::type_tag truncated the result-column OID to 16 bits, while
DataCell mapped an out-of-range OID to text. A user-defined type whose OID
aliases a binary-decoded builtin modulo 65536 was requested in binary and
then read as text. Tag::from_oid is now the only mapping, used for Bind
parameters, result format codes and DataCell.

write_bind now records each parameter's encoding while it writes the format
codes and reuses it for the value section, so a getter or toString() that
runs between the two sections cannot make them disagree.

Adds mock-server tests that pin the Bind bytes for both cases, and a
pgDecodeBind helper in wire-frames.ts.
@robobun

robobun commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up on overlap in write_bind: #41970 changes the bool arm (a non-boolean value now throws ERR_INVALID_ARG_TYPE instead of to_boolean()), and moves the bytea arm's error from #41889 (already on main) into a shared invalid_bind_value helper. It does not touch the format-code loop. Whichever of the two lands second needs a small rebase: in the ParamEncoding shape the bool rule is ParamEncoding::Bool with !value.is_boolean() returning that error, and as_boolean() in the arm. This branch also predates #41889, so the Bytea arm needs the same treatment on rebase.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

robobun added a commit that referenced this pull request Sep 8, 2026
…ind bytes with mock-server tests

FieldDescription::type_tag() truncated a result column's OID to 16 bits
for the Bind result format code, while DataCell mapped an OID above
65535 to text. A user-defined type whose OID aliases a binary-decoded
builtin modulo 65536 was requested in binary and read as text.
Tag::from_oid() is now the one mapping, for parameters, result format
codes and DataCell.

wire-frames.ts gains pgNoData() and pgDecodeBind(). The new mock-server
tests in postgres-bind-parameter-format.test.ts check the exact format
codes and value bytes of the Bind message without a database. The
container tests from #41912 and the Date test file from #41955 are
carried over.
@robobun

robobun commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Cross-reference: #41976 fixes the same format-code / encoder mismatch with a single pass over the parameters (the branch that writes binary bytes flips its own format code), and also stops the existing binary arms from coercing a Date, array or object (true, wrapped int32, 2000-01-01). The Tag::from_oid change and the container tests from this PR are carried over there, so if #41976 merges this one can close.

@robobun

robobun commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #41976. It makes the Bind format code follow the encoder that actually runs, which is the same fix as the ParamEncoding enum here, done in one pass (FormatCodes::set_binary). It also runs a binary encoder only for the JS class it encodes (a Date bound to bool no longer stores true) and sends a Date in text format as ISO 8601 (#29010).

The parts unique to this PR were carried over there in c2ed427: Tag::from_oid() as the one OID-to-Tag mapping for parameters, result format codes and DataCell, the pgDecodeBind() / pgNoData() helpers with their self-test, and the mock-server and container tests (merged into test/js/sql/postgres-bind-parameter-format.test.ts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant