Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for pgvector #612

Closed
liemng opened this issue Dec 4, 2023 · 14 comments
Closed

Support for pgvector #612

liemng opened this issue Dec 4, 2023 · 14 comments
Labels
type: enhancement A general enhancement
Milestone

Comments

@liemng
Copy link

liemng commented Dec 4, 2023

Feature Request

Is your feature request related to a problem? Please describe

Support for pgvector.

Describe the solution you'd like

Support for pgvector similar to Hibernate: https://docs.jboss.org/hibernate/orm/6.4/userguide/html_single/Hibernate_User_Guide.html#vector-module

@liemng liemng added the type: enhancement A general enhancement label Dec 4, 2023
@mp911de
Copy link
Collaborator

mp911de commented Dec 5, 2023

How exactly you envision support for PG Vector with a library that requires you to provide SQL?

@davecramer
Copy link
Member

@mp911de how do you support custom types today? The only "support" that is possible in pgjdbc is to allow sending and receiving the vector in binary. Is that possible ? Pretty significant performance possible with binary mode.

@mp911de
Copy link
Collaborator

mp911de commented Dec 5, 2023

We have a Codec API that can be extended by the driver or any external Codec

public Codec<?> createCodec(ByteBufAllocator byteBufAllocator, int oid) {
switch (this) {
case HSTORE:
return new HStoreCodec(byteBufAllocator, oid);
case POSTGIS_GEOMETRY:
return new PostgisGeometryCodec(oid);
default:
throw new UnsupportedOperationException(String.format("Codec %s for OID %d not supported", name(), oid));
}
}

By parsing type/OID information, one could come up with any kind of wrapper that makes sense.

@davecramer
Copy link
Member

Cool, so there are significant savings to be had by sending and receiving vectors in binary!

@liemng
Copy link
Author

liemng commented Dec 5, 2023

Yeah, exactly it, @mp911de. It would be nice to have a built-in codec for pgvector, given the popularity of LLMs nowadays.

@mp911de
Copy link
Collaborator

mp911de commented Dec 5, 2023 via email

@davecramer
Copy link
Member

FWIW, I didn't implement it in pgjdbc, rather in pg_vector java support. I don't think it matters where it's implemented.

@mp911de mp911de added this to the 1.0.3.RELEASE milestone Dec 7, 2023
mp911de added a commit that referenced this issue Dec 7, 2023
[closes #612]

Signed-off-by: Mark Paluch <[email protected]>
@mp911de mp911de closed this as completed in e7aaec4 Dec 7, 2023
@mp911de
Copy link
Collaborator

mp911de commented Dec 7, 2023

That's in place now, including support for 1-dimensional vector arrays (array of vectors).

@mp911de
Copy link
Collaborator

mp911de commented Dec 7, 2023

Please give the new Vector codec a try, it's available from 1.1.0 and 1.0.3 snapshots via oss sonatype.

@davecramer
Copy link
Member

cool!

@liemng
Copy link
Author

liemng commented Dec 11, 2023

Working as designed! @mp911de you rock! When is 1.0.3 release planned for?

@mp911de
Copy link
Collaborator

mp911de commented Dec 12, 2023

I would want to wait for today's Reactor release before shipping a new release.

mp911de added a commit that referenced this issue Dec 12, 2023
Update reflection config for Vector.

[#612]

Signed-off-by: Mark Paluch <[email protected]>
mp911de added a commit that referenced this issue Dec 12, 2023
Update reflection config for Vector.

[#612]

Signed-off-by: Mark Paluch <[email protected]>
@chrischall
Copy link
Contributor

Apologies for the comment here vs. a bug report, but PostgreSQL 8.2 and below to not have pg_catalog.pg_type.typarray:
https://www.postgresql.org/docs/8.2/catalog-pg-type.html

AWS Redshift is based on PostgreSQL 8.0.2:
https://docs.aws.amazon.com/redshift/latest/dg/r_VERSION.html

As such, this driver can no longer be used to connect to AWS Redshift.

@mp911de
Copy link
Collaborator

mp911de commented Dec 27, 2023

Please downgrade to 1.0.2 for the time being. I filed #621 to fix the regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

4 participants