-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Add Snowflake JDBC Connector #10387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
yuuteng
wants to merge
114
commits into
trinodb:master
from
yuuteng:feature/2551-add-snowflake-connector
Closed
Add Snowflake JDBC Connector #10387
Changes from all commits
Commits
Show all changes
114 commits
Select commit
Hold shift + click to select a range
1d37327
Adding Bloomberg Snowflake connector.
dprophet 3bf24da
Fix version number problem with build.
dprophet 2afade4
Add Snowflake JDBC Connector
yuuteng e68f59c
Update trino snapshot version to 372
yuuteng bd91c9d
Update format of the doc of snowflake
yuuteng 409acde
Update trino jdbc library import
yuuteng f08dd2a
Fix date formatter from yyyy to uuuu
yuuteng ee9b920
Fix date test case
yuuteng b1eecc2
Remove defunct property allow-drop-table
yuuteng b902d0d
Update trino version to 374
yuuteng e604633
Update snowflake config to adapt 374
yuuteng 61ac1f0
Update the range of the test of Date type
yuuteng 92d42a1
Update to version 375
yuuteng ad4bbc3
Fix snowflake after updating to 375
yuuteng 37322dc
Update to 381
yuuteng 68249ba
Fix mvn pom import
yuuteng 7bab04d
Format snowflake.rst
yuuteng befa51f
Reorderd Data tests in type mapping
yuuteng c7cb13b
Update function code
yuuteng 7968933
Add product test
yuuteng 4814f37
Rename product test tablename
yuuteng f4d9f15
Add Env, Suite and properties of Snowflake for production test
yuuteng 7d73f13
Add trinoCreateAndInsert()
yuuteng 9495db8
Refactor snowflake from single node to multi node
yuuteng d61b8f6
Pass product tests
yuuteng c44be40
Removed snowflake.properties in trino server dev
yuuteng 391f4c5
Resolved issues 19 05 2022 and fixed tests
yuuteng 517ded4
Remove Types.VARBINARY
yuuteng 99921d3
Add private static SliceWriteFunction charWriteFunction
yuuteng 5b83fff
Update test case
yuuteng 83bf784
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng 68c7587
Update docs/src/main/sphinx/connector/snowflake.rst
yuuteng 3d62bd5
Update plugin/trino-snowflake/pom.xml
yuuteng e9d2897
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng 49e9325
Resolved review open issues
yuuteng 14b654d
Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case
yuuteng f83dd03
Updated properties file
yuuteng 8c3c815
Updated properties files
yuuteng fb9c4a2
Renamed properties in Testing
yuuteng e2ae8d4
Revert "Renamed properties in Testing"
yuuteng bddbb92
Renamed properties and fixed tests
yuuteng 9937df0
Update the way to pass ENV values for production test
yuuteng edb1a79
Update trino version to 388
yuuteng b375123
Update Trino version 391
yuuteng 95a943c
Update trino version to 394
yuuteng 73c3152
Update to 395
yuuteng 2a9802a
Update to 411
yuuteng 9ddc0c4
Update and fix errors
yuuteng c5729e3
Build successfully with 411
yuuteng 9002e50
Merge branch 'snowflake' of https://github.com/bloomberg/trino into b…
yuuteng 632a646
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng 901aa0b
Merge pull request #2 from yuuteng/feature/2551-add-snowflake-connector
dprophet 1687bea
Adding varchar type to mapping Snowflake.
dprophet acf5182
Fixing some tests.
dprophet bc767c0
Fix TestSnowflakeConnectorTest.
yuuteng c071cd8
Fix type mapper
yuuteng 770c467
Fix testconnector
yuuteng f3236a4
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng 14e7431
Add Snowflake JDBC Connector
yuuteng 7671dfe
Update trino snapshot version to 372
yuuteng 951aa96
Update format of the doc of snowflake
yuuteng fbde6f0
Update trino jdbc library import
yuuteng d0d82d6
Fix date formatter from yyyy to uuuu
yuuteng 6cb0ca2
Fix date test case
yuuteng 187e04b
Remove defunct property allow-drop-table
yuuteng d1d6f5c
Update trino version to 374
yuuteng a10a085
Update snowflake config to adapt 374
yuuteng 38e068f
Update the range of the test of Date type
yuuteng 765dd86
Update to version 375
yuuteng 410df33
Fix snowflake after updating to 375
yuuteng c82e5fd
Update to 381
yuuteng 7498c32
Fix mvn pom import
yuuteng 883219a
Format snowflake.rst
yuuteng cc9d2b2
Reorderd Data tests in type mapping
yuuteng 3db806c
Update function code
yuuteng 47b02ef
Add product test
yuuteng e35e282
Rename product test tablename
yuuteng eae4f64
Add Env, Suite and properties of Snowflake for production test
yuuteng 1c4892f
Add trinoCreateAndInsert()
yuuteng f775981
Refactor snowflake from single node to multi node
yuuteng f107647
Pass product tests
yuuteng e4534bf
Removed snowflake.properties in trino server dev
yuuteng b670b22
Resolved issues 19 05 2022 and fixed tests
yuuteng 502425c
Remove Types.VARBINARY
yuuteng e0f3137
Add private static SliceWriteFunction charWriteFunction
yuuteng 2830740
Update test case
yuuteng a75b731
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng f2e1f9b
Update docs/src/main/sphinx/connector/snowflake.rst
yuuteng f6c6fee
Update plugin/trino-snowflake/pom.xml
yuuteng 642d61f
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng b832673
Resolved review open issues
yuuteng c50c9ff
Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case
yuuteng 0b4a89f
Updated properties file
yuuteng c6f2c53
Updated properties files
yuuteng a61977b
Renamed properties in Testing
yuuteng 515d67e
Revert "Renamed properties in Testing"
yuuteng cf9e4a4
Renamed properties and fixed tests
yuuteng c7fd30d
Update the way to pass ENV values for production test
yuuteng 327cebf
Update trino version to 388
yuuteng b469b41
Update Trino version 391
yuuteng 44b1ef8
Update trino version to 394
yuuteng ceadf76
Update to 395
yuuteng 1b4c80a
Update to 411
yuuteng 0327ab8
Update and fix errors
yuuteng baaac25
Build successfully with 411
yuuteng 1a82298
Adding Bloomberg Snowflake connector.
dprophet 11f220a
Adding varchar type to mapping Snowflake.
dprophet 1d649b7
Fixing some tests.
dprophet 22d8c8f
Fix TestSnowflakeConnectorTest.
yuuteng 4a5f3c2
Fix type mapper
yuuteng 6c95ead
Fix testconnector
yuuteng d5ac0dc
Update version to 413-SNAPSHOT
yuuteng 4618db6
Merge branch 'snowflake' into bloomberg-snowflake
yuuteng 8f79937
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| =================== | ||
| Snowflake connector | ||
| =================== | ||
|
|
||
| The Snowflake connector allows querying and creating tables in an | ||
| external Snowflake account. This can be used to join data between | ||
| different systems like Snowflake and Hive, or between two different | ||
| Snowflake accounts. | ||
|
|
||
| Configuration | ||
| ------------- | ||
|
|
||
| To configure the Snowflake connector, create a catalog properties file | ||
| in ``etc/catalog`` named, for example, ``snowflake.properties``, to | ||
| mount the Snowflake connector as the ``snowflake`` catalog. | ||
| Create the file with the following contents, replacing the | ||
| connection properties as appropriate for your setup: | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| connector.name=snowflake | ||
| connection-url=jdbc:snowflake://<account>.snowflakecomputing.com | ||
| connection-user=root | ||
| connection-password=secret | ||
| snowflake.account=account | ||
| snowflake.database=database | ||
| snowflake.role=role | ||
| snowflake.warehouse=warehouse | ||
|
|
||
|
|
||
| Multiple Snowflake databases or accounts | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| The Snowflake connector can only access a single database within | ||
| a Snowflake account. Thus, if you have multiple Snowflake databases, | ||
| or want to connect to multiple Snowflake accounts, you must configure | ||
| multiple instances of the Snowflake connector. | ||
|
|
||
| .. snowflake-type-mapping: | ||
|
|
||
| Type mapping | ||
| ------------ | ||
|
|
||
| Trino supports the following Snowflake data types: | ||
|
|
||
| ================================== =============================== | ||
| Snowflake Type Trino Type | ||
| ================================== =============================== | ||
| ``boolean`` ``boolean`` | ||
| ``tinyint`` ``bigint`` | ||
| ``smallint`` ``bigint`` | ||
| ``byteint`` ``bigint`` | ||
| ``int`` ``bigint`` | ||
| ``integer`` ``bigint`` | ||
| ``bigint`` ``bigint`` | ||
| ``float`` ``real`` | ||
| ``real`` ``real`` | ||
| ``double`` ``double`` | ||
| ``decimal`` ``decimal(P,S)`` | ||
| ``varchar(n)`` ``varchar(n)`` | ||
| ``char(n)`` ``varchar(n)`` | ||
| ``binary(n)`` ``varbinary`` | ||
| ``varbinary`` ``varbinary`` | ||
| ``date`` ``date`` | ||
| ================================== =============================== | ||
|
|
||
| Complete list of `Snowflake data types | ||
| <https://docs.snowflake.com/en/sql-reference/intro-summary-data-types.html>`_. | ||
|
|
||
| .. _snowflake-sql-support: | ||
|
|
||
| SQL support | ||
| ----------- | ||
|
|
||
| The connector provides read access and write access to data and metadata in | ||
| a Snowflake database. In addition to the :ref:`globally available | ||
| <sql-globally-available>` and :ref:`read operation <sql-read-operations>` | ||
| statements, the connector supports the following features: | ||
|
|
||
| * :doc:`/sql/insert` | ||
| * :doc:`/sql/delete` | ||
| * :doc:`/sql/truncate` | ||
| * :doc:`/sql/create-table` | ||
| * :doc:`/sql/create-table-as` | ||
| * :doc:`/sql/drop-table` | ||
| * :doc:`/sql/alter-table` | ||
| * :doc:`/sql/create-schema` | ||
| * :doc:`/sql/drop-schema` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| <?xml version="1.0"?> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <parent> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-root</artifactId> | ||
| <version>413-SNAPSHOT</version> | ||
| <relativePath>../../pom.xml</relativePath> | ||
| </parent> | ||
|
|
||
| <artifactId>trino-snowflake</artifactId> | ||
| <description>Trino - Snowflake Connector</description> | ||
| <packaging>trino-plugin</packaging> | ||
|
|
||
| <properties> | ||
| <air.main.basedir>${project.parent.basedir}</air.main.basedir> | ||
| </properties> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-base-jdbc</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-plugin-toolkit</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>configuration</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>log</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.google.guava</groupId> | ||
| <artifactId>guava</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.google.inject</groupId> | ||
| <artifactId>guice</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>javax.inject</groupId> | ||
| <artifactId>javax.inject</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>net.snowflake</groupId> | ||
| <artifactId>snowflake-jdbc</artifactId> | ||
| <version>3.13.29</version> | ||
| </dependency> | ||
|
|
||
| <!-- Trino SPI --> | ||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-spi</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>slice</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.core</groupId> | ||
| <artifactId>jackson-annotations</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.openjdk.jol</groupId> | ||
| <artifactId>jol-core</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <!-- for testing --> | ||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-base-jdbc</artifactId> | ||
| <type>test-jar</type> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-main</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-main</artifactId> | ||
| <type>test-jar</type> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-testing</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-testing-services</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino</groupId> | ||
| <artifactId>trino-tpch</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.trino.tpch</groupId> | ||
| <artifactId>tpch</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>testing</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.assertj</groupId> | ||
| <artifactId>assertj-core</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.jetbrains</groupId> | ||
| <artifactId>annotations</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.testcontainers</groupId> | ||
| <artifactId>jdbc</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.testcontainers</groupId> | ||
| <artifactId>testcontainers</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.testng</groupId> | ||
| <artifactId>testng</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-surefire-plugin</artifactId> | ||
| <configuration> | ||
| <argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.