Fix PostgreSQL IP address serialization error #3900

Apmats · 2025-12-23T15:45:43Z

Closes #3879

Problem:

The PostgreSQL connector fails with serialization error because asyncpg returns PostgreSQL's INET and CIDR columns as Python ipaddress module objects (IPv4Address, IPv6Address, IPv4Network, etc.) rather than strings. Our serialization layer doesn't handle these types, causing the connector to crash when attempting to convert documents to JSON.

Solution:

The implementation converts these objects to strings before calling the parent serializer, which handles standard types. Recursive serialization handles nested structures (lists/dicts containing IP addresses). Followed the same pattern from the MongoDB connector.

Only added plain tests for these changes, not ftest coverage. You, reviewer!, give me an opinion on this!

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Security-related changes (encryption, TLS, SSRF, etc)
New external service dependencies added.

Related Pull Requests

Release Note

artem-shelkovnikov

IMO it's really easy to add this to ftest and we should.

Ideally the table(s) in ftest should have all possible data types that require serialisation.

Also, I see their test suite here https://github.com/MagicStack/asyncpg/blob/9b2b027224058e4590fdb9e41b5750a64dab1fce/tests/test_codecs.py that has some more fun data types, do you wanna try them out too? Geospatial types, for example, or UUIDs?

Apmats · 2025-12-24T11:20:12Z

Okay, the only argument for not having ftest coverage is that this is somewhat duplicating stuff, and also probably not consistent with the rest of the ftests which as far as I understand don't try to be thorough.
I initially did add them but then removed though, the argument for adding them was that asyncpg might adopt their own types and then things will just break again without us noticing until it's too late.

On more types, my coding assistant did suggest adding that stuff - I decided to try to keep the change minimal but without good reason.

I'll see what I do about both, will ping for a repeat look. 👍

…p-serialization' into am/fix-postgresql-ip-serialization

Apmats · 2025-12-31T10:29:26Z

@artem-shelkovnikov whenever you have time, have another 👀 - added some basic ftest coverage and also extended support to UUID and geometric types - I assume you meant built-in postgres geometric types from the linked test codecs file and not geospatial. I had to make some decisions about how to stringify things, and I decided to make no mention of the shape itself in the generated string, as the column type is already known from the schema, so including 'POINT' or 'POLYGON' in the string would be redundant.

seanstory

LGTM!

lorenabalan

LGTM

## Closes #3879 Problem: The PostgreSQL connector fails with serialization error because asyncpg returns PostgreSQL's INET and CIDR columns as Python ipaddress module objects (IPv4Address, IPv6Address, IPv4Network, etc.) rather than strings. Our serialization layer doesn't handle these types, causing the connector to crash when attempting to convert documents to JSON. Solution: The implementation converts these objects to strings before calling the parent serializer, which handles standard types. Recursive serialization handles nested structures (lists/dicts containing IP addresses). Followed the same pattern from the MongoDB connector. Only added plain tests for these changes, not ftest coverage. You, reviewer!, give me an opinion on this!  ## Checklists  #### Pre-Review Checklist - [x] this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check `config.yml.example`) - [x] this PR has a meaningful title - [x] this PR links to all relevant github issues that it fixes or partially addresses - [ ] if there is no GH issue, please create it. Each PR should have a link to an issue - [x] this PR has a thorough description - [x] Covered the changes with automated tests - [x] Tested the changes locally - [x] Added a label for each target release version (example: `v7.13.2`, `v7.14.0`, `v8.0.0`) - [x] For bugfixes: backport safely to all minor branches still receiving patch releases - [x] Considered corresponding documentation changes - [x] Contributed any configuration settings changes to the configuration reference - [x] if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in [Kibana](https://github.com/elastic/kibana/blob/main/packages/kbn-search-connectors/types/native_connectors.ts) #### Changes Requiring Extra Attention  - [ ] Security-related changes (encryption, TLS, SSRF, etc) - [ ] New external service dependencies added. ## Related Pull Requests  ## Release Note  --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

github-actions · 2026-01-12T08:11:00Z

💔 Failed to create backport PR(s)

Status	Branch	Result
❌	9.1	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.2	Commit could not be cherrypicked due to conflicts
✅	9.3	#3903

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
backport --pr 3900 --autoMerge --autoMergeMethod squash

Backports the following commits to 9.3: - Fix PostgreSQL IP address serialization error (#3900) --------- Co-authored-by: Apostolos Matsagkas <Apmats@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

artem-shelkovnikov · 2026-01-12T10:34:50Z

Yeah, due to our defactoring you'll need to do backports manually here, @Apmats

Apmats · 2026-01-12T10:41:32Z

@artem-shelkovnikov yeap working on that now, I knew it was coming. 😁

Backport of PR #3900 to 9.2 PostgreSQL connector uses asyncpg which returns special Python objects for certain PostgreSQL data types. These need to be serialized to strings before indexing: - Network types (INET, CIDR) -> ipaddress module objects - UUID type -> uuid.UUID objects - Geometric types (POINT, LINE, POLYGON, etc.) -> asyncpg.types objects - BitString type (BIT, VARBIT) -> asyncpg.types.BitString objects

Manually backporting the work from the following PR to 9.2: - Fix PostgreSQL IP address serialization error (#3900) I think auto backport should work for these versions and so I can avoid seeking reviews for the other branches. 😁 --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Manually backporting the work from the following PR to 8.19: - Fix PostgreSQL IP address serialization error (#3900) I thought doing #3904 would backport even when merging to something that's not main, but apparently I was wrong, so handling in separate PRs.

Manually backporting the work from the following PR to 9.1: - Fix PostgreSQL IP address serialization error (#3900) I thought doing #3904 would backport even when merging to something that's not main, but apparently I was wrong, so handling in separate PRs. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Apmats requested a review from a team as a code owner December 23, 2025 15:45

github-actions bot added auto-backport v9.4.0 labels Dec 23, 2025

Apmats force-pushed the am/fix-postgresql-ip-serialization branch from 0b4a445 to 3324a00 Compare December 23, 2025 15:55

Fix PostgreSQL IP address serialization error

f810b59

Apmats force-pushed the am/fix-postgresql-ip-serialization branch from 2a92418 to f810b59 Compare December 23, 2025 16:34

elasticmachine added 2 commits December 23, 2025 16:38

Update NOTICE.txt

0c440b5

make autoformat

ff55510

artem-shelkovnikov reviewed Dec 24, 2025

View reviewed changes

artem-shelkovnikov added v9.1.0 v8.19.0 v9.2.0 v9.3.0 labels Dec 24, 2025

Apmats and others added 6 commits December 30, 2025 17:20

Expand types covered, and cover with ftests

d72c272

Merge remote-tracking branch 'refs/remotes/origin/am/fix-postgresql-i…

ba10a26

…p-serialization' into am/fix-postgresql-ip-serialization

Expand types covered, and cover with ftests

5fe14aa

make autoformat

97f194b

Some further polish

ecf605f

Clean up tests a bit more

91da20d

Apmats and others added 2 commits January 7, 2026 15:29

Merge branch 'main' into am/fix-postgresql-ip-serialization

002a7dd

Update NOTICE.txt

437afc2

seanstory approved these changes Jan 8, 2026

View reviewed changes

lorenabalan approved these changes Jan 8, 2026

View reviewed changes

Apmats merged commit bb6bbff into main Jan 12, 2026
2 checks passed

Apmats deleted the am/fix-postgresql-ip-serialization branch January 12, 2026 08:10

github-actions bot mentioned this pull request Jan 12, 2026

[9.3] Fix PostgreSQL IP address serialization error (#3900) #3903

Merged

Apmats mentioned this pull request Jan 12, 2026

[9.2] Fix PostgreSQL IP address serialization error (#3900) #3904

Merged

This was referenced Jan 12, 2026

[9.1] Fix PostgreSQL IP address serialization error (#3900) #3905

Merged

[9.0] Fix PostgreSQL IP address serialization error (#3900) #3906

Closed

[8.19] Fix PostgreSQL IP address serialization error (#3900) #3907

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix PostgreSQL IP address serialization error #3900

Fix PostgreSQL IP address serialization error #3900

Uh oh!

Apmats commented Dec 23, 2025 •

edited

Loading

Uh oh!

artem-shelkovnikov left a comment

Uh oh!

Apmats commented Dec 24, 2025

Uh oh!

Apmats commented Dec 31, 2025 •

edited

Loading

Uh oh!

seanstory left a comment

Uh oh!

lorenabalan left a comment

Uh oh!

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

artem-shelkovnikov commented Jan 12, 2026

Uh oh!

Apmats commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Fix PostgreSQL IP address serialization error #3900

Fix PostgreSQL IP address serialization error #3900

Uh oh!

Conversation

Apmats commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Closes #3879

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Related Pull Requests

Release Note

Uh oh!

artem-shelkovnikov left a comment

Choose a reason for hiding this comment

Uh oh!

Apmats commented Dec 24, 2025

Uh oh!

Apmats commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanstory left a comment

Choose a reason for hiding this comment

Uh oh!

lorenabalan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jan 12, 2026

💔 Failed to create backport PR(s)

Uh oh!

artem-shelkovnikov commented Jan 12, 2026

Uh oh!

Apmats commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Apmats commented Dec 23, 2025 •

edited

Loading

Apmats commented Dec 31, 2025 •

edited

Loading