-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat: add support for DuckDB #879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
8d6a66d
Add support for DuckDB
bpranava 0c17088
Retrigger CLA
bpranava c1e1d6c
add template param integration tests
bpranava ea8cd4e
update duckdb readme
bpranava eb4a7ed
delete db file after integration test run
bpranava ecd9cf6
rename db file in duckdb test
bpranava 7444d74
add google copyright comments
bpranava 8baccb7
add duckdb to integration cloudbuild
bpranava a54253e
address review comments
bpranava 3fe85ac
update go mod
bpranava ddf89a0
- update duckdb tool documentation
bpranava 0c660f7
fix duckdb integration tests
bpranava 18392b4
rename duckdb tool packages
bpranava 3ebb587
reuse integration test functions in duckdb test
bpranava e7f91e9
Merge branch 'main' into feat/duckdb
averikitsch 288745f
Update docs/en/resources/tools/duckdb/duckdb-sql.md
averikitsch 00ee56e
Merge branch 'main' into feat/duckdb
averikitsch 20875cb
address review comments
bpranava f67eb7e
fix broken link in doc
bpranava ac991d6
update docs
bpranava 20250a5
Update docs/en/resources/tools/duckdb/duckdb-sql.md
averikitsch 86f134a
Update docs/en/resources/tools/duckdb/duckdb-sql.md
averikitsch b28df50
Merge branch 'main' into feat/duckdb
averikitsch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| title: DuckDB | ||
| linkTitle: DuckDB | ||
| type: docs | ||
| weight: 1 | ||
| description: > | ||
| DuckDB is an in-process SQL OLAP database management system designed for analytical query processing. | ||
| --- | ||
|
|
||
| ## About | ||
|
|
||
| [DuckDB](https://duckdb.org/) is an embedded analytical database management system that runs in-process with the client application. It is optimized for analytical workloads, providing high performance for complex queries with minimal setup. | ||
|
|
||
| DuckDB has the following notable characteristics: | ||
|
|
||
| - In-process, serverless database engine | ||
| - Supports complex SQL queries for analytical processing | ||
| - Can operate on in-memory or persistent storage | ||
| - Zero-configuration - no external dependencies or server setup required | ||
| - Highly optimized for columnar data storage and query execution | ||
|
|
||
| For more details, refer to the [DuckDB Documentation](https://duckdb.org/). | ||
|
|
||
| ## Available Tools | ||
| - [`duckdb-sql`](../tools/duckdb/duckdb-sql.md) | ||
| Execute pre-defined prepared SQL queries in DuckDB. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Database File | ||
|
|
||
| To use DuckDB, you can either: | ||
|
|
||
| - Specify a file path for a persistent database stored on the filesystem | ||
| - Omit the file path to use an in-memory database | ||
|
|
||
| ## Example | ||
|
|
||
| For a persistent DuckDB database: | ||
|
|
||
| ```yaml | ||
| sources: | ||
| my-duckdb: | ||
| kind: "duckdb" | ||
| dbFilePath: "/path/to/database.db" | ||
| configuration: | ||
| memory_limit: "2GB" | ||
| threads: "4" | ||
| ``` | ||
|
|
||
| For an in-memory DuckDB database: | ||
|
|
||
| ```yaml | ||
| sources: | ||
| my-duckdb-memory: | ||
| name: "my-duckdb-memory" | ||
| kind: "duckdb" | ||
| ``` | ||
|
|
||
| ## Reference | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | **field** | **type** | **required** | **description** | | ||
| |-------------------|:-----------------:|:------------:|---------------------------------------------------------------------------------| | ||
| | kind | string | true | Must be "duckdb". | | ||
| | dbFilePath | string | false | Path to the DuckDB database file. Omit for an in-memory database. | | ||
| | configuration | map[string]string | false | Additional DuckDB configuration options (e.g., `memory_limit`, `threads`). | | ||
|
|
||
| For a complete list of available configuration options, refer to the [DuckDB Configuration Documentation](https://duckdb.org/docs/stable/configuration/overview.html#local-configuration-options). | ||
|
|
||
|
|
||
| For more details on the Go implementation, see the [go-duckdb package documentation](https://pkg.go.dev/github.com/scottlepp/go-duckdb#section-readme). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| title: "DuckDB" | ||
| type: docs | ||
| weight: 1 | ||
| description: > | ||
| Tools that work with DuckDB Sources. | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| --- | ||
| title: "duckdb-sql" | ||
| type: docs | ||
| weight: 1 | ||
| description: > | ||
| Execute SQL statements against a DuckDB database using the DuckDB SQL tools configuration. | ||
| aliases: | ||
| - /resources/tools/duckdb-sql | ||
| --- | ||
|
|
||
| ## About | ||
|
|
||
| A `duckdb-sql` tool executes a pre-defined SQL statement against a [DuckDB](https://duckdb.org/) database. It is compatible with any DuckDB source configuration as defined in the [DuckDB source documentation](../../sources/duckdb.md). | ||
|
|
||
| The specified SQL statement is executed as a prepared statement, and parameters are inserted according to their position: e.g., `$1` is the first parameter, `$2` is the second, and so on. If template parameters are included, they are resolved before execution of the prepared statement. | ||
|
|
||
| DuckDB's SQL dialect closely follows the conventions of the PostgreSQL dialect, with a few exceptions listed in the [DuckDB PostgreSQL Compatibility documentation](https://duckdb.org/docs/stable/sql/dialect/postgresql_compatibility.html). For an introduction to DuckDB's SQL dialect, refer to the [DuckDB SQL Introduction](https://duckdb.org/docs/stable/sql/introduction). | ||
|
|
||
| ### Concepts | ||
|
|
||
| DuckDB is a relational database management system (RDBMS). Data is stored in relations (tables), where each table is a named collection of rows. Each row in a table has the same set of named columns, each with a specific data type. Tables are stored within schemas, and a collection of schemas constitutes the entire database. | ||
|
|
||
| For more details, see the [DuckDB SQL Introduction](https://duckdb.org/docs/stable/sql/introduction). | ||
|
|
||
| ## Example | ||
|
|
||
| > **Note:** This tool uses parameterized queries to prevent SQL injections. Query parameters can be used as substitutes for arbitrary expressions but cannot be used for identifiers, column names, table names, or other parts of the query. | ||
|
|
||
| ```yaml | ||
| tools: | ||
| search-users: | ||
| kind: duckdb-sql | ||
| source: my-duckdb | ||
| description: Search users by name and age | ||
| statement: SELECT * FROM users WHERE name LIKE $1 AND age >= $2 | ||
| parameters: | ||
| - name: name | ||
| type: string | ||
| description: The name to search for | ||
| - name: min_age | ||
| type: integer | ||
| description: Minimum age | ||
| ``` | ||
|
|
||
| ## Example with Template Parameters | ||
|
|
||
| > **Note:** Template parameters allow direct modifications to the SQL statement, including identifiers, column names, and table names, which makes them more vulnerable to SQL injections. Using basic parameters (see above) is recommended for performance and safety. For more details, see the [templateParameters](../#template-parameters) section. | ||
|
|
||
| ```yaml | ||
| tools: | ||
| list_table: | ||
| kind: duckdb-sql | ||
| source: my-duckdb | ||
| statement: | | ||
| SELECT * FROM {{.tableName}}; | ||
| description: | | ||
| Use this tool to list all information from a specific table. | ||
| Example: | ||
| {{ | ||
| "tableName": "flights", | ||
| }} | ||
| templateParameters: | ||
| - name: tableName | ||
| type: string | ||
| description: Table to select from | ||
| ``` | ||
|
|
||
| ## Reference | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | **field** | **type** | **required** | **description** | | ||
| |--------------------|:-------------------------------:|:------------:|--------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | kind | string | true | Must be "duckdb-sql". | | ||
| | source | string | true | Name of the DuckDB source configuration (see [DuckDB source documentation](../../sources/duckdb.md)). | | ||
| | description | string | true | Description of the tool that is passed to the LLM. | | ||
| | statement | string | true | The SQL statement to execute. | | ||
| | authRequired | []string | false | List of authentication requirements for the tool (if any). | | ||
| | parameters | [parameters](../#specifying-parameters) | false | List of parameters that will be inserted into the SQL statement | | ||
| | templateParameters | [templateParameters](../#template-parameters) | false | List of template parameters that will be inserted into the SQL statement before executing the prepared statement. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.