Skip to content

fix: cast ISO timestamp strings to timestamp type in tool_registry_writer SQL - #23449

Closed
dkindlund wants to merge 1 commit into
BerriAI:mainfrom
dkindlund:fix/tool-registry-datetime-type
Closed

fix: cast ISO timestamp strings to timestamp type in tool_registry_writer SQL#23449
dkindlund wants to merge 1 commit into
BerriAI:mainfrom
dkindlund:fix/tool-registry-datetime-type

Conversation

@dkindlund

Copy link
Copy Markdown
Contributor

Summary

Fixes tool_registry_writer batch_upsert_tools error: ERROR: column "created_at" is of type timestamp without time zone but expression is of type text by adding explicit ::timestamp casts to raw SQL parameter references.

Steps to Reproduce

1. Deploy LiteLLM v1.82.0-stable with a PostgreSQL database and MCP/tool-calling enabled.

2. Send any request that triggers tool discovery (e.g., a chat completion with tool use).

3. The tool_registry_writer background job fires and fails continuously:

tool_registry_writer batch_upsert_tools error: ERROR: column "created_at" is of type
timestamp without time zone but expression is of type text
HINT: You will need to rewrite or cast the expression.

This error fires every ~8 seconds for as long as the proxy is running.

Production Evidence

Observed on Google Cloud Run deployment (litellm service, trending-threats project) running v1.82.0-stable with Cloud SQL PostgreSQL. 185 errors in a 1-hour window:

2026-03-12T13:34:30  tool_registry_writer batch_upsert_tools error: ERROR: column
  "created_at" is of type timestamp without time zone but expression is of type text
  HINT: You will need to rewrite or cast the expression.

Root Cause

The batch_upsert_tools() and update_tool_policy() functions in tool_registry_writer.py use execute_raw() with ISO 8601 timestamp strings (datetime.now(timezone.utc).isoformat()):

now = datetime.now(timezone.utc).isoformat()
# Produces: "2026-03-12T13:34:30.123456+00:00"

await prisma_client.db.execute_raw(
    'INSERT INTO "LiteLLM_ToolTable" (..., created_at, updated_at) '
    "VALUES (..., $8, $8) "                    # ← $8 is text type
    "ON CONFLICT (...) DO UPDATE SET "
    "updated_at = $8",                          # ← $8 is text type
    ..., now,
)

Prisma's execute_raw() passes the .isoformat() string as a text parameter. PostgreSQL's created_at and updated_at columns are timestamp without time zone, and PostgreSQL does not implicitly cast text to timestamp in parameterized queries.

Fix

Added explicit ::timestamp casts to all timestamp parameter references in the raw SQL:

-- Before:
VALUES (..., $8, $8)
ON CONFLICT (...) DO UPDATE SET updated_at = $8

-- After:
VALUES (..., $8::timestamp, $8::timestamp)
ON CONFLICT (...) DO UPDATE SET updated_at = $8::timestamp

Applied to both batch_upsert_tools() and update_tool_policy().

Test plan

  • Verified the SQL syntax is valid with ::timestamp casts
  • Verified ISO 8601 strings with timezone offsets cast correctly to timestamp without time zone in PostgreSQL
  • Both affected functions (batch_upsert_tools, update_tool_policy) are fixed

…iter SQL

The raw SQL upserts in batch_upsert_tools and update_tool_policy pass
datetime.isoformat() strings for created_at/updated_at columns. Prisma
execute_raw treats these as text parameters, but PostgreSQL columns are
timestamp without time zone and refuse implicit text-to-timestamp casts.

Adds explicit ::timestamp casts to all timestamp parameter references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Mar 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Ready Ready Preview, Comment Mar 12, 2026 1:49pm

Request Review

@greptile-apps

greptile-apps Bot commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a real, actively-occurring production error (column "created_at" is of type timestamp without time zone but expression is of type text) by adding ::timestamp casts to all timestamp parameter references in the raw SQL statements inside batch_upsert_tools and update_tool_policy in tool_registry_writer.py.

  • Root cause addressed correctly: Prisma's execute_raw() transmits Python datetime.isoformat() strings as text to PostgreSQL, which does not implicitly cast text → timestamp. The explicit ::timestamp casts resolve this.
  • Scope is minimal and focused: Only the two affected functions are changed; no unrelated refactors.
  • Silent timezone stripping: ::timestamp discards the +00:00 suffix from the ISO string rather than converting from UTC. This is safe today because datetime.now(timezone.utc) is always UTC, but is fragile if the source ever changes.
  • No automated tests: The test plan consists entirely of manual verification checkboxes. A unit test with a mock execute_raw verifying the SQL string shape would improve long-term confidence.

Confidence Score: 4/5

  • This PR is safe to merge — it fixes a confirmed production error with a targeted, well-understood SQL cast change.
  • The fix is correct, minimal, and directly addresses the reported PostgreSQL type-mismatch error. One point is held back because ::timestamp silently strips timezone info from the ISO string (safe today but fragile), and there are no automated regression tests for these SQL code paths.
  • No files require special attention beyond the inline style note on tool_registry_writer.py.

Important Files Changed

Filename Overview
litellm/proxy/db/tool_registry_writer.py Adds ::timestamp casts to all raw SQL timestamp parameter references in batch_upsert_tools and update_tool_policy to fix a PostgreSQL type mismatch. The fix is correct and targeted; a minor consideration is that ::timestamp silently strips timezone offset info from ISO strings, which is only safe here because the source is always timezone.utc.

Sequence Diagram

sequenceDiagram
    participant TD as Tool Discovery
    participant TRW as tool_registry_writer
    participant Prisma as Prisma execute_raw
    participant PG as PostgreSQL

    TD->>TRW: batch_upsert_tools(items)
    TRW->>TRW: now = datetime.now(UTC).isoformat()
    Note over TRW: e.g. "2026-03-12T13:34:30+00:00"

    TRW->>Prisma: execute_raw(SQL, ..., now)
    Note over Prisma: Binds now as text parameter $8

    alt Before fix
        Prisma->>PG: INSERT ... VALUES ($8, $8)<br/>[text → timestamp column]
        PG-->>Prisma: ERROR: column is of type timestamp<br/>but expression is of type text
        Prisma-->>TRW: Exception
        TRW-->>TD: logs error, silently continues
    else After fix
        Prisma->>PG: INSERT ... VALUES ($8::timestamp, $8::timestamp)<br/>[explicit cast strips timezone offset]
        PG-->>Prisma: OK (1 row upserted)
        Prisma-->>TRW: success
        TRW-->>TD: logs upserted N tool(s)
    end
Loading

Comments Outside Diff (1)

  1. litellm/proxy/db/tool_registry_writer.py, line 65-69 (link)

    Silent timezone truncation with ::timestamp cast

    datetime.now(timezone.utc).isoformat() produces a timezone-aware string like "2026-03-12T13:34:30.123456+00:00". When PostgreSQL casts this via ::timestamp (without timezone), it silently strips the +00:00 offset rather than converting from UTC — the stored wall-clock time equals the UTC time only because the source is already timezone.utc. If this call-site ever changes to use a non-UTC local time, the offset would be silently discarded and wrong timestamps would be stored with no error.

    A more robust alternative is to pass a timezone-naive UTC datetime directly (skipping .isoformat()), which avoids the ambiguity entirely:

    now = datetime.now(timezone.utc).replace(tzinfo=None)   # naive UTC datetime

    Or, if Prisma's execute_raw handles Python datetime objects (which Prisma typically does), you could drop the cast and use the object directly. Either approach makes the intent explicit and avoids silent timezone truncation.

    The same consideration applies to update_tool_policy at line 143.

Last reviewed commit: 56252e2

@dkindlund

Copy link
Copy Markdown
Contributor Author

Closing this PR — the fix was already applied on main via commit 291e6e1, which reverted the raw SQL approach back to Prisma ORM upserts (passing native datetime objects instead of .isoformat() strings). My local origin/main was stale when I created this branch. Apologies for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant