feat(wren-launcher): introduce CLI tool for dbt integration#1827
feat(wren-launcher): introduce CLI tool for dbt integration#1827wwwy3y3 merged 14 commits intoCanner:mainfrom
Conversation
WalkthroughThis update introduces a comprehensive dbt project conversion feature to the Wren launcher, enabling users to convert dbt projects into Wren-compatible formats via both CLI and interactive flows. It adds new Go packages for parsing dbt profiles, converting data sources, and generating Wren MDL manifests, along with integration into Docker configurations for local storage support. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Launcher
participant DbtConverter
participant DockerUtils
User->>Launcher: Run launch command
Launcher->>User: Prompt for dbt project path
User->>Launcher: Provide dbt project path (or skip)
alt dbt project provided
Launcher->>User: Prompt for profile and target
User->>Launcher: Provide profile/target (or skip)
Launcher->>DbtConverter: Convert dbt project to Wren format
DbtConverter-->>Launcher: Return local storage path
else no dbt project
Launcher-->>Launcher: Use default local storage path
end
Launcher->>DockerUtils: Prepare Docker files with local storage path
DockerUtils-->>Launcher: Docker files prepared
Launcher-->>User: Launch complete
Suggested labels
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (4)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (3)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (10)
wren-launcher/config/config.go (1)
42-46: Handle Windows on ARM to future-proof the helper
GetPlatformspecial-caseswindowsbut hard-codeslinux/amd64, ignoring the possibility of Windows on ARM machines. Consider mirroring thedarwin/linuxbranches:case "windows": - return "linux/amd64" // Windows typically uses amd64 + if runtime.GOARCH == "arm64" { + return "linux/arm64" + } + return "linux/amd64" // Default to amd64This keeps the public contract unchanged for x86-64 systems while adding graceful support for ARM laptops such as Surface Pro X.
wren-launcher/main.go (1)
41-55: Consider improving help output formatting.The help function provides good information but could benefit from better formatting and consistency.
Consider this improvement for better readability:
func showHelp() { - pterm.Info.Println("Usage of Wren launcher:") - pterm.Info.Println("") - pterm.Info.Println("Commands:") - pterm.Info.Println(" (default) Launch Wren AI service") - pterm.Info.Println(" dbt-auto-convert --path --output [--profile] [--target] Auto-convert dbt project to WrenDataSource and Wren MDL") - pterm.Info.Println("") - pterm.Info.Println("Flags:") + pterm.Info.Println("Usage of Wren launcher:\n") + pterm.Info.Println("Commands:") + pterm.Info.Println(" (default) Launch Wren AI service") + pterm.Info.Println(" dbt-auto-convert Auto-convert dbt project to WrenDataSource and Wren MDL") + pterm.Info.Println(" Required: --path, --output") + pterm.Info.Println(" Optional: --profile, --target") + pterm.Info.Println("\nFlags:") flag.PrintDefaults() - pterm.Info.Println("") - pterm.Info.Println("Examples:") + pterm.Info.Println("\nExamples:") pterm.Info.Println(" wren-launcher # Launch Wren AI") pterm.Info.Println(" wren-launcher dbt-auto-convert --path /path/to/dbt --output ./output # Auto-convert dbt project") pterm.Info.Println(" wren-launcher dbt-auto-convert --path /path/to/dbt --output ./output --profile my_profile --target dev # Convert with specific profile/target") }wren-launcher/commands/dbt/profiles_analyzer.go (1)
173-176: Remove redundant conditional logic.Both Windows and non-Windows cases return the same path. The conditional check is unnecessary.
- if runtime.GOOS == "windows" { - return filepath.Join(homeDir, ".dbt", "profiles.yml") - } return filepath.Join(homeDir, ".dbt", "profiles.yml")wren-launcher/commands/dbt/README.md (2)
22-22: Fix markdown heading punctuation.Remove trailing colons from headings to follow markdown best practices.
-### Steps: +### Steps -### Example: +### Example -### Steps: +### Steps -### Example: +### ExampleAlso applies to: 51-51, 81-81, 86-86
57-67: Replace hard tabs with spaces in code blocks.Use consistent indentation with spaces instead of tabs for better rendering across different platforms.
Convert the tab characters to spaces (typically 4 spaces per tab level) in the Go code examples.
Also applies to: 92-112
wren-launcher/commands/dbt.go (1)
31-31: Extract duplicated usage message to reduce code duplication.The same usage message appears twice. Consider extracting it to a constant or helper function.
+const dbtAutoConvertUsage = "Usage: wren-launcher dbt-auto-convert --path /path/to/dbt/project --output /path/to/output" + func DbtAutoConvert() { // ... flag definitions ... // Validate required parameters if opts.ProjectPath == "" { pterm.Error.Println("Error: --path parameter is required") - pterm.Info.Println("Usage: wren-launcher dbt-auto-convert --path /path/to/dbt/project --output /path/to/output") + pterm.Info.Println(dbtAutoConvertUsage) os.Exit(1) } if opts.OutputDir == "" { pterm.Error.Println("Error: --output parameter is required") - pterm.Info.Println("Usage: wren-launcher dbt-auto-convert --path /path/to/dbt/project --output /path/to/output") + pterm.Info.Println(dbtAutoConvertUsage) os.Exit(1) }Also applies to: 37-37
wren-launcher/commands/dbt/data_source_test.go (1)
274-375: Good test coverage, but consider adding tests for missing scenarios.The existing tests are well-structured and cover the main functionality. However, consider adding tests for:
MapTypefunctionality for both PostgreSQL and LocalFile data sources- File path handling edge cases (absolute vs relative paths with/without dbt_home)
DefaultDataSourceimplementationWould you like me to generate additional test cases for these missing scenarios?
wren-launcher/commands/dbt/converter.go (3)
144-149: Consider making the container mount path configurable.The hard-coded path "/usr/src/app/data" could be made configurable for flexibility.
Consider adding a constant or configuration option:
const DefaultContainerDataPath = "/usr/src/app/data"
299-308: Consider making staging model filtering configurable.The code automatically excludes staging models (containing ".stg_" or ".staging_"), which might be too restrictive for some users who want to include these models in their Wren setup.
Consider adding an option to include/exclude staging models rather than hard-coding the exclusion.
423-427: Column sorting may not preserve dbt's intended order.The code sorts columns alphabetically, but dbt models often have a specific column order that might be meaningful (e.g., primary keys first). Consider preserving the original order or making sorting configurable.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (16)
docker/.env.example(1 hunks)docker/docker-compose.yaml(1 hunks)wren-launcher/commands/dbt.go(1 hunks)wren-launcher/commands/dbt/README.md(1 hunks)wren-launcher/commands/dbt/converter.go(1 hunks)wren-launcher/commands/dbt/data_source.go(1 hunks)wren-launcher/commands/dbt/data_source_test.go(1 hunks)wren-launcher/commands/dbt/profiles.go(1 hunks)wren-launcher/commands/dbt/profiles_analyzer.go(1 hunks)wren-launcher/commands/dbt/utils.go(1 hunks)wren-launcher/commands/dbt/wren_mdl.go(1 hunks)wren-launcher/commands/launch.go(5 hunks)wren-launcher/config/config.go(1 hunks)wren-launcher/go.mod(1 hunks)wren-launcher/main.go(1 hunks)wren-launcher/utils/docker.go(5 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: cyyeh
PR: Canner/WrenAI#1293
File: wren-ai-service/pyproject.toml:38-38
Timestamp: 2025-02-12T22:05:37.109Z
Learning: The qdrant-client package version in wren-ai-service must match the Qdrant server version (1.11.0) to maintain compatibility.
docker/docker-compose.yaml (1)
Learnt from: paopa
PR: Canner/WrenAI#976
File: wren-ai-service/tools/dev/.env:17-17
Timestamp: 2025-02-03T06:25:14.605Z
Learning: The IBIS server version sha-ce21e44 refers to a commit in the internal wren-engine repository (https://github.com/Canner/wren-engine), not the upstream ibis-project/ibis repository. This version includes fixes for server startup issues.
🧬 Code Graph Analysis (6)
wren-launcher/main.go (2)
wren-launcher/config/config.go (1)
InitFlags(17-24)wren-launcher/commands/dbt.go (1)
DbtAutoConvert(13-56)
wren-launcher/commands/dbt/profiles_analyzer.go (2)
wren-launcher/commands/dbt/profiles.go (3)
DbtProfiles(4-7)DbtProfile(10-13)DbtConnection(16-39)wren-launcher/commands/dbt/utils.go (1)
FileExists(30-36)
wren-launcher/commands/dbt.go (1)
wren-launcher/commands/dbt/converter.go (3)
ConvertOptions(15-22)ConvertDbtProjectCore(33-230)ConvertResult(25-29)
wren-launcher/commands/dbt/data_source_test.go (2)
wren-launcher/commands/dbt/profiles.go (3)
DbtProfiles(4-7)DbtProfile(10-13)DbtConnection(16-39)wren-launcher/commands/dbt/data_source.go (6)
FromDbtProfiles(18-40)WrenPostgresDataSource(166-172)WrenLocalFileDataSource(122-125)GetActiveDataSources(204-242)GetDataSourceByType(245-259)ValidateAllDataSources(262-284)
wren-launcher/commands/launch.go (1)
wren-launcher/commands/dbt.go (1)
DbtConvertProject(60-71)
wren-launcher/commands/dbt/data_source.go (1)
wren-launcher/commands/dbt/profiles.go (2)
DbtProfiles(4-7)DbtConnection(16-39)
🪛 LanguageTool
wren-launcher/commands/dbt/README.md
[grammar] ~1-~1: Use proper spacing conventions.
Context: # How to Support a New Data Source This document outlines the steps require...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~3-~3: Use proper spacing conventions.
Context: ...ata source to the dbt project converter. The target data source must be supported...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~5-~5: There might be a mistake here.
Context: ... by both dbt and the Wren engine: - [dbt supported databases](https://docs.getdbt...
(QB_NEW_EN_OTHER)
[grammar] ~6-~6: There might be a mistake here.
Context: ...supported-data-platforms) - [Wren engine supported data sources](https://docs.get...
(QB_NEW_EN_OTHER)
[grammar] ~6-~6: Use proper spacing conventions.
Context: ...s) - Wren engine supported data sources ## 1. Implement the DataSource Interface T...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~8-~8: Place a period at the end of declarative sentences.
Context: ...## 1. Implement the DataSource Interface The first step is to define a new struct...
(QB_NEW_EN_OTHER_ERROR_IDS_000178)
[grammar] ~10-~10: Use proper spacing conventions.
Context: ...interface defined indata_source.go. The DataSource` interface is as follows...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~12-~12: Use proper spacing conventions.
Context: ...he DataSource interface is as follows: go type DataSource interface { GetType() string Validate() error MapType(sourceType string) string } ### Steps: 1. Define Your Struct: Crea...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~22-~22: Use proper spacing conventions.
Context: ...rceType string) string } ``` ### Steps: 1. Define Your Struct: Create a new struc...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~24-~24: Use proper spacing conventions.
Context: ...nectionInfo) for the target data source. For example, to add support for `Snowfla...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~26-~26: Use proper spacing conventions.
Context: ..., you would define the following struct: go type WrenSnowflakeDataSource struct { Account string `json:"account"` User string `json:"user"` Password string `json:"password"` Database string `json:"database"` Warehouse string `json:"warehouse"` // ... other properties } 2. Implement GetType(): This method sho...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~39-~39: Use proper spacing conventions.
Context: ... data source type (e.g., "snowflake"). 3. Implement Validate(): This method sh...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~41-~41: Use proper spacing conventions.
Context: ...id. Return an error if validation fails. 4. Implement MapType(): This method is ...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~43-~43: Use proper spacing conventions.
Context: ....g., integer, varchar, timestamp). ## 2. Add Conversion Logic in `data_source....
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~45-~45: Use proper spacing conventions.
Context: ...stamp). ## 2. Add Conversion Logic in data_source.go` After implementing the interface, you ne...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~47-~47: Use proper spacing conventions.
Context: ...ataSourcefunction indata_source.go. Add a new caseto theswitch` stateme...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~49-~49: Use proper spacing conventions.
Context: ... struct from the dbt connection details. ### Example: ```go // in data_source.go fu...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~51-~51: Use proper spacing conventions.
Context: ...he dbt connection details. ### Example: go // in data_source.go func convertConnectionToDataSource(conn DbtConnection, dbt_home_path, profileName, outputName string) (DataSource, error) { switch strings.ToLower(conn.Type) { case "postgres", "postgresql": return convertToPostgresDataSource(conn) case "duckdb": return convertToLocalFileDataSource(conn, dbt_home_path) // Add your new case here case "snowflake": return convertToSnowflakeDataSource(conn) // Implement this function default: // ... } } // Implement the conversion function func convertToSnowflakeDataSource(conn DbtConnection) (*WrenSnowflakeDataSource, error) { // Logic to extract snowflake properties from conn // and return a new *WrenSnowflakeDataSource } ## 3. Handle the New Data Source in `Conver...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~77-~77: Use proper spacing conventions.
Context: ...`` ## 3. Handle the New Data Source in ConvertDbtProjectCore The `ConvertDbtProjectCore` function in ...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~79-~79: Use proper spacing conventions.
Context: ...on to ensure it is correctly serialized. ### Steps: 1. **Locate the switch statem...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~81-~81: Use proper spacing conventions.
Context: ... it is correctly serialized. ### Steps: 1. Locate the switch statement: Find th...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~83-~83: Use proper spacing conventions.
Context: ...)block insideConvertDbtProjectCore. 2. **Add a new case**: Add a new case` for...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~84-~84: Use proper spacing conventions.
Context: ...ith the correct type and properties. ### Example: ```go // in converter.go's Con...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~86-~86: Use proper spacing conventions.
Context: ...t type and properties. ### Example: go // in converter.go's ConvertDbtProjectCore function // ... switch typedDS := ds.(type) { case *WrenPostgresDataSource: // ... case *WrenLocalFileDataSource: // ... // Add your new case here case *WrenSnowflakeDataSource: wrenDataSource = map[string]interface{}{ "type": "snowflake", "properties": map[string]interface{}{ "account": typedDS.Account, "user": typedDS.User, "password": typedDS.Password, "database": typedDS.Database, "warehouse": typedDS.Warehouse, // ... other properties }, } default: // ... } // ... Note on File-Based Data Sources: If yo...
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
[grammar] ~116-~116: Use proper spacing conventions.
Context: ...ren engine where to find the data files.
(QB_NEW_EN_OTHER_ERROR_IDS_000007)
🪛 markdownlint-cli2 (0.17.2)
wren-launcher/commands/dbt/README.md
22-22: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
51-51: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
57-57: Hard tabs
Column: 1
(MD010, no-hard-tabs)
58-58: Hard tabs
Column: 1
(MD010, no-hard-tabs)
59-59: Hard tabs
Column: 1
(MD010, no-hard-tabs)
60-60: Hard tabs
Column: 1
(MD010, no-hard-tabs)
61-61: Hard tabs
Column: 1
(MD010, no-hard-tabs)
63-63: Hard tabs
Column: 1
(MD010, no-hard-tabs)
64-64: Hard tabs
Column: 1
(MD010, no-hard-tabs)
65-65: Hard tabs
Column: 1
(MD010, no-hard-tabs)
66-66: Hard tabs
Column: 1
(MD010, no-hard-tabs)
67-67: Hard tabs
Column: 1
(MD010, no-hard-tabs)
81-81: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
86-86: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
92-92: Hard tabs
Column: 1
(MD010, no-hard-tabs)
93-93: Hard tabs
Column: 1
(MD010, no-hard-tabs)
94-94: Hard tabs
Column: 1
(MD010, no-hard-tabs)
95-95: Hard tabs
Column: 1
(MD010, no-hard-tabs)
96-96: Hard tabs
Column: 1
(MD010, no-hard-tabs)
98-98: Hard tabs
Column: 1
(MD010, no-hard-tabs)
99-99: Hard tabs
Column: 1
(MD010, no-hard-tabs)
100-100: Hard tabs
Column: 1
(MD010, no-hard-tabs)
101-101: Hard tabs
Column: 1
(MD010, no-hard-tabs)
102-102: Hard tabs
Column: 1
(MD010, no-hard-tabs)
103-103: Hard tabs
Column: 1
(MD010, no-hard-tabs)
104-104: Hard tabs
Column: 1
(MD010, no-hard-tabs)
105-105: Hard tabs
Column: 1
(MD010, no-hard-tabs)
107-107: Hard tabs
Column: 1
(MD010, no-hard-tabs)
108-108: Hard tabs
Column: 1
(MD010, no-hard-tabs)
109-109: Hard tabs
Column: 1
(MD010, no-hard-tabs)
110-110: Hard tabs
Column: 1
(MD010, no-hard-tabs)
111-111: Hard tabs
Column: 1
(MD010, no-hard-tabs)
112-112: Hard tabs
Column: 1
(MD010, no-hard-tabs)
🔇 Additional comments (29)
wren-launcher/go.mod (1)
218-218: Consider tidyinggo.modto stabilise YAML dependencies
gopkg.in/yaml.v3 v3.0.1is now promoted to a direct dependency whilegopkg.in/yaml.v2 v2.4.0 // indirectremains earlier in the file. Having both major versions is legal, but it is worth confirming that:
- All new code really imports
gopkg.in/yaml.v3, notv2, otherwise builds may still compile against the older API via transitive deps.- Nothing accidentally still relies on
yaml.v2that could instead migrate tov3, which simplifies the tree and reduces binary size.A quick
go mod tidyafter the change will drop any truly-unused version and ensure the module graph reflects reality.If both versions are genuinely required, no action needed beyond documenting the rationale (e.g., comment on the
v2line).wren-launcher/config/config.go (1)
43-46: No-op change looks good
Only trailing-whitespace removed; functional behavior untouched.docker/.env.example (1)
49-52: LGTM! Well-documented environment variable addition.The addition of
LOCAL_STORAGEenvironment variable is properly documented and positioned in the appropriate section. The default value of "." is sensible for local development scenarios.docker/docker-compose.yaml (1)
44-45: LGTM! Proper volume mount configuration.The volume mount syntax is correct and follows Docker Compose best practices. The use of
${LOCAL_STORAGE:-.}provides a sensible default while allowing environment customization.wren-launcher/main.go (1)
15-27: Fix potential index out of bounds issue in argument handling.The current argument manipulation logic assumes
os.Argshas at least 2 elements, which could cause a panic if the slice is shorter.Apply this diff to add bounds checking:
// Check if we have subcommands if len(os.Args) > 1 { switch os.Args[1] { case "dbt-auto-convert": - // Remove the subcommand from args so flag.Parse() works correctly - os.Args = append([]string{os.Args[0]}, os.Args[2:]...) + // Remove the subcommand from args so flag.Parse() works correctly + if len(os.Args) > 2 { + os.Args = append([]string{os.Args[0]}, os.Args[2:]...) + } else { + os.Args = []string{os.Args[0]} + } commands.DbtAutoConvert() return case "help", "-h", "--help": showHelp() return } }Likely an incorrect or invalid review comment.
wren-launcher/commands/dbt/utils.go (1)
1-37: LGTM! Well-structured utility functions.The implementation is clean and follows Go best practices:
- Proper error handling with
os.IsNotExist- Clear function names and logic flow
- Appropriate validation for dbt project structure
- Good separation of concerns between the utility functions
wren-launcher/commands/dbt/wren_mdl.go (1)
1-57: LGTM! Well-designed data structures for Wren MDL.The struct definitions are well-thought-out with:
- Consistent and appropriate JSON tags
- Proper use of
omitemptyfor optional fields- Good type choices for each field
- Logical hierarchy for representing MDL manifests
- Flexible
Propertiesmaps for extensibilitywren-launcher/commands/dbt/profiles_analyzer.go (2)
90-121: Well-designed helper functions for safe type conversion.The getString, getInt, and getBool helper functions provide robust type assertion with fallback defaults, preventing runtime panics from type mismatches.
179-198: Excellent search strategy and error reporting.The function searches profiles.yml in logical priority order and provides comprehensive error messages listing all attempted paths.
wren-launcher/commands/dbt/README.md (1)
1-117: Excellent developer documentation.This README provides clear, comprehensive instructions for extending the dbt converter with new data sources. The step-by-step approach with concrete examples makes it easy for developers to follow.
wren-launcher/utils/docker.go (1)
39-39: Clean integration of LOCAL_STORAGE support.The changes follow the established pattern for environment variable replacement and integrate seamlessly with the existing Docker configuration functionality. The default value of "." for local storage is appropriate.
Also applies to: 81-86, 243-243, 283-283
wren-launcher/commands/dbt.go (1)
41-48: Well-designed separation of CLI and wrapper functionality.The distinction between
DbtAutoConvert(strict CLI requiring catalog.json) andDbtConvertProject(flexible wrapper allowing optional catalog.json) provides appropriate interfaces for different use cases.Also applies to: 61-68
wren-launcher/commands/dbt/profiles.go (1)
3-39: Excellent data structure design for dbt profiles.The struct definitions are comprehensive and well-organized:
- Clear separation of concerns between profiles, individual profile, and connection
- Proper YAML/JSON tag annotations for serialization
- Good coverage of database-specific fields (BigQuery, Snowflake, Postgres, DuckDB)
- Flexible
Additionalfield for extensibilitywren-launcher/commands/dbt/data_source_test.go (5)
7-66: LGTM! Comprehensive test for PostgreSQL data source conversion.The test thoroughly validates field mapping, type conversion, and validation for PostgreSQL connections.
116-141: LGTM! Good edge case testing for unsupported database types.The test correctly verifies that unsupported connection types are ignored without errors.
143-149: LGTM! Essential nil check test.Good defensive programming practice to test nil input handling.
151-206: LGTM! Excellent table-driven test for PostgreSQL validation.Well-structured test that covers all validation scenarios including edge cases for required fields and port range validation.
208-272: LGTM! Thorough test coverage for active data source retrieval.The test comprehensively covers default selection, explicit selection, and error cases for profile and target retrieval.
wren-launcher/commands/launch.go (4)
125-186: LGTM! Well-structured user prompt functions.The new prompt functions follow the existing patterns and provide clear instructions for users, properly handling empty inputs for optional parameters.
467-485: LGTM! Smart conditional prompting logic.Good design decision to skip target prompt when profile is empty, avoiding unnecessary user interaction.
487-521: LGTM! Well-implemented dbt project processing.The function properly handles user input, creates necessary directories, and integrates cleanly with the dbt conversion logic. Good use of
filepath.Joinfor cross-platform path handling.
299-316: LGTM! Clean integration of dbt processing into launch flow.The dbt project processing is properly integrated with appropriate error handling and the local storage path is correctly passed to Docker file preparation.
wren-launcher/commands/dbt/data_source.go (4)
9-40: LGTM! Well-designed interface and conversion logic.The DataSource interface provides a clean abstraction, and FromDbtProfiles properly handles the nested profile structure with appropriate error handling.
122-164: LGTM! Clean implementation of local file data source.The type mapping is comprehensive and validation logic is appropriate.
196-199: Verify if PostgreSQL MapType implementation is intentional.The MapType method returns the input unchanged, unlike the LocalFileDataSource implementation which performs actual type mapping. Is this intentional because PostgreSQL handles its own type mapping, or should it provide similar type normalization?
244-315: LGTM! Well-implemented utility functions and default data source.The filtering, validation aggregation, and default type mapping are all properly implemented.
wren-launcher/commands/dbt/converter.go (3)
14-46: LGTM! Well-structured conversion options and proper initialization.Good validation, directory creation, and informative logging.
47-77: LGTM! Flexible file handling with helpful user guidance.The code properly handles missing files with appropriate warnings and hints, while providing flexibility through the RequireCatalog option.
232-242: LGTM! Smart handling of localhost for container environments.The function correctly addresses the common Docker networking challenge with clear documentation.
goldmedal
force-pushed
the
feat/dbt-cli-command
branch
from
f345e06 to
f21fbc5
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (4)
wren-launcher/commands/dbt/profiles_analyzer.go (1)
145-150: The "path" field issue has been resolved.The past review comment about the missing "path" field in the knownFields map has been addressed. I can see that "path": true is correctly included in the knownFields map on line 149, ensuring that the path field extracted on line 142 is properly recognized as a known field.
wren-launcher/commands/dbt/data_source.go (3)
43-55: Parameter naming conventions have been addressed.The past review comments about fixing parameter naming from
dbt_home_pathtodbtHomePathhave been resolved. The function signature and all usages now correctly follow Go naming conventions.
204-242: Parameter naming conventions have been addressed.The past review comment about renaming
dbt_home_pathtodbtHomePathhas been resolved. The parameter is correctly named and used throughout the function.
82-107: Refactor duplicated path resolution logic.The path resolution logic is duplicated between handling
conn.Pathandconn.Additional["file"]. This creates maintenance overhead and potential for inconsistency.Extract the path resolution logic into a helper function:
+func resolvePath(path, dbtHome string) string { + if filepath.IsAbs(path) { + return filepath.Dir(path) + } + relativeDir := filepath.Dir(path) + if dbtHome != "" { + return filepath.Join(dbtHome, relativeDir) + } + return relativeDir +} + func convertToLocalFileDataSource(conn DbtConnection, dbtHome string) (*WrenLocalFileDataSource, error) { var url, format string // Try to get file path from different fields if conn.Path != "" { - if filepath.IsAbs(conn.Path) { - url = filepath.Dir(conn.Path) - } else { - relativeDir := filepath.Dir(conn.Path) - if dbtHome != "" { - url = filepath.Join(dbtHome, relativeDir) - } else { - url = relativeDir - } - } + url = resolvePath(conn.Path, dbtHome) } else if file, exists := conn.Additional["file"]; exists { if fileStr, ok := file.(string); ok { - if filepath.IsAbs(fileStr) { - url = filepath.Dir(fileStr) - } else { - relativeDir := filepath.Dir(fileStr) - if dbtHome != "" { - url = filepath.Join(dbtHome, relativeDir) - } else { - url = relativeDir - } - } + url = resolvePath(fileStr, dbtHome) } }
🧹 Nitpick comments (2)
wren-launcher/commands/dbt/README.md (1)
1-117: Address markdown formatting issues to improve readability.The static analysis tools have identified several formatting issues that should be addressed:
- Trailing punctuation in headings (lines 22, 51, 81, 86): Remove colons from heading text
- Hard tabs in code blocks: Replace tabs with spaces for consistent formatting
- Grammar and spacing issues: Minor corrections needed throughout
Apply these formatting improvements:
-### Steps: +### Steps -### Example: +### Example -### Steps: +### Steps -### Example: +### ExampleAlso, ensure consistent spacing after code blocks and replace hard tabs with spaces in the code examples.
wren-launcher/commands/dbt/profiles_analyzer.go (1)
173-177: Redundant platform-specific logic in GetDefaultProfilesPath.The Windows and non-Windows paths are identical, making the platform check unnecessary.
if runtime.GOOS == "windows" { return filepath.Join(homeDir, ".dbt", "profiles.yml") } - return filepath.Join(homeDir, ".dbt", "profiles.yml") + return filepath.Join(homeDir, ".dbt", "profiles.yml")Since both paths are the same, you can simplify to:
func GetDefaultProfilesPath() string { homeDir, err := os.UserHomeDir() if err != nil { return "" } - - if runtime.GOOS == "windows" { - return filepath.Join(homeDir, ".dbt", "profiles.yml") - } return filepath.Join(homeDir, ".dbt", "profiles.yml") }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (16)
docker/.env.example(1 hunks)docker/docker-compose.yaml(1 hunks)wren-launcher/commands/dbt.go(1 hunks)wren-launcher/commands/dbt/README.md(1 hunks)wren-launcher/commands/dbt/converter.go(1 hunks)wren-launcher/commands/dbt/data_source.go(1 hunks)wren-launcher/commands/dbt/data_source_test.go(1 hunks)wren-launcher/commands/dbt/profiles.go(1 hunks)wren-launcher/commands/dbt/profiles_analyzer.go(1 hunks)wren-launcher/commands/dbt/utils.go(1 hunks)wren-launcher/commands/dbt/wren_mdl.go(1 hunks)wren-launcher/commands/launch.go(5 hunks)wren-launcher/config/config.go(1 hunks)wren-launcher/go.mod(1 hunks)wren-launcher/main.go(1 hunks)wren-launcher/utils/docker.go(5 hunks)
✅ Files skipped from review due to trivial changes (2)
- wren-launcher/go.mod
- wren-launcher/commands/dbt/wren_mdl.go
🚧 Files skipped from review as they are similar to previous changes (11)
- wren-launcher/config/config.go
- docker/.env.example
- docker/docker-compose.yaml
- wren-launcher/main.go
- wren-launcher/commands/dbt/utils.go
- wren-launcher/commands/dbt.go
- wren-launcher/commands/dbt/profiles.go
- wren-launcher/commands/launch.go
- wren-launcher/utils/docker.go
- wren-launcher/commands/dbt/converter.go
- wren-launcher/commands/dbt/data_source_test.go
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: cyyeh
PR: Canner/WrenAI#1293
File: wren-ai-service/pyproject.toml:38-38
Timestamp: 2025-02-12T22:05:37.109Z
Learning: The qdrant-client package version in wren-ai-service must match the Qdrant server version (1.11.0) to maintain compatibility.
wren-launcher/commands/dbt/profiles_analyzer.go (1)
Learnt from: andreashimin
PR: Canner/WrenAI#1414
File: wren-ui/src/apollo/server/utils/error.ts:0-0
Timestamp: 2025-03-18T10:28:10.593Z
Learning: The typo in error code enum names ("IDENTIED_AS_GENERAL" and "IDENTIED_AS_MISLEADING_QUERY" instead of "IDENTIFIED_AS_GENERAL" and "IDENTIFIED_AS_MISLEADING_QUERY") in wren-ui/src/apollo/server/utils/error.ts was recognized but intentionally not fixed as it was considered out of scope for PR #1414.
🪛 LanguageTool
wren-launcher/commands/dbt/README.md
[grammar] ~1-~1: Use correct spacing
Context: # How to Support a New Data Source This document outlines the steps requir...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~5-~5: Use hyphens correctly
Context: ...ted by both dbt and the Wren engine: - [dbt supported databases](https://docs.getdbt...
(QB_NEW_EN_OTHER_ERROR_IDS_29)
[grammar] ~6-~6: Use hyphens correctly
Context: .../docs/supported-data-platforms) - [Wren engine supported data sources](https://docs.get...
(QB_NEW_EN_OTHER_ERROR_IDS_29)
[grammar] ~6-~6: Use correct spacing
Context: ...latforms) - Wren engine supported data sources ## 1. Implement the DataSource Interface ...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~8-~8: There might be a mistake here.
Context: ...onInfo) ## 1. Implement the DataSource Interface The first step is to define a new struc...
(QB_NEW_EN_OTHER)
[grammar] ~10-~10: Use correct spacing
Context: ...t the DataSource interface defined in data_source.go. The DataSource interface is as follow...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~12-~12: There might be a problem here.
Context: ....go. The DataSource` interface is as follows: go type DataSource interface { GetType() string Validate() error MapType(sourceType string) string } ### Steps: 1. Define Your Struct: Create a new stru...
(QB_NEW_EN_MERGED_MATCH)
[grammar] ~24-~24: Use correct spacing
Context: ...lakeConnectionInfo) for the target data source. For example, to add support for `Snowfl...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~26-~26: Use correct spacing
Context: ...wflake, you would define the following struct: ```go type WrenSnowflakeDataSource struct { Account string json:"account" User stringjson:"user" Password stringjson:"password" Database stringjson:"database" Warehouse stringjson:"warehouse" // ... other properties } ``` 2. **ImplementGetType()`**: This method sh...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~39-~39: Use correct spacing
Context: ...identifies your data source type (e.g., "snowflake"). 3. Implement Validate(): This method s...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~41-~41: Use correct spacing
Context: ...nd valid. Return an error if validation fails. 4. Implement MapType(): This method is...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~43-~43: Use correct spacing
Context: ...data types (e.g., integer, varchar, timestamp). ## 2. Add Conversion Logic in `data_source...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~45-~45: Use correct spacing
Context: ...imestamp). ## 2. Add Conversion Logic in data_source.go` After implementing the interface, you ...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~47-~47: Use correct spacing
Context: ...vertConnectionToDataSourcefunction indata_source.go. Add a new caseto theswitch` statem...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~49-~49: There might be a problem here.
Context: ...a source struct from the dbt connection details. ### Example: go // in data_source.go func convertConnectionToDataSource(conn DbtConnection, dbtHomePath, profileName, outputName string) (DataSource, error) { switch strings.ToLower(conn.Type) { case "postgres", "postgresql": return convertToPostgresDataSource(conn) case "duckdb": return convertToLocalFileDataSource(conn, dbtHomePath) // Add your new case here case "snowflake": return convertToSnowflakeDataSource(conn) // Implement this function default: // ... } } // Implement the conversion function func convertToSnowflakeDataSource(conn DbtConnection) (*WrenSnowflakeDataSource, error) { // Logic to extract snowflake properties from conn // and return a new *WrenSnowflakeDataSource } ## 3. Handle the New Data Source in `Conve...
(QB_NEW_EN_MERGED_MATCH)
[grammar] ~77-~77: Use correct spacing
Context: ...} ``` ## 3. Handle the New Data Source in ConvertDbtProjectCore The `ConvertDbtProjectCore` function i...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~79-~79: There might be a problem here.
Context: ...this function to ensure it is correctly serialized. ### Steps: 1. Locate the switch statement: Find t...
(QB_NEW_EN_MERGED_MATCH)
[grammar] ~84-~84: There might be a problem here.
Context: ...Sourcemap with the correcttypeandproperties`. ### Example: go // in converter.go's ConvertDbtProjectCore function // ... switch typedDS := ds.(type) { case *WrenPostgresDataSource: // ... case *WrenLocalFileDataSource: // ... // Add your new case here case *WrenSnowflakeDataSource: wrenDataSource = map[string]interface{}{ "type": "snowflake", "properties": map[string]interface{}{ "account": typedDS.Account, "user": typedDS.User, "password": typedDS.Password, "database": typedDS.Database, "warehouse": typedDS.Warehouse, // ... other properties }, } default: // ... } // ... Note on File-Based Data Sources: If y...
(QB_NEW_EN_MERGED_MATCH)
🪛 markdownlint-cli2 (0.17.2)
wren-launcher/commands/dbt/README.md
22-22: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
51-51: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
57-57: Hard tabs
Column: 1
(MD010, no-hard-tabs)
58-58: Hard tabs
Column: 1
(MD010, no-hard-tabs)
59-59: Hard tabs
Column: 1
(MD010, no-hard-tabs)
60-60: Hard tabs
Column: 1
(MD010, no-hard-tabs)
61-61: Hard tabs
Column: 1
(MD010, no-hard-tabs)
63-63: Hard tabs
Column: 1
(MD010, no-hard-tabs)
64-64: Hard tabs
Column: 1
(MD010, no-hard-tabs)
65-65: Hard tabs
Column: 1
(MD010, no-hard-tabs)
66-66: Hard tabs
Column: 1
(MD010, no-hard-tabs)
67-67: Hard tabs
Column: 1
(MD010, no-hard-tabs)
81-81: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
86-86: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
92-92: Hard tabs
Column: 1
(MD010, no-hard-tabs)
93-93: Hard tabs
Column: 1
(MD010, no-hard-tabs)
94-94: Hard tabs
Column: 1
(MD010, no-hard-tabs)
95-95: Hard tabs
Column: 1
(MD010, no-hard-tabs)
96-96: Hard tabs
Column: 1
(MD010, no-hard-tabs)
98-98: Hard tabs
Column: 1
(MD010, no-hard-tabs)
99-99: Hard tabs
Column: 1
(MD010, no-hard-tabs)
100-100: Hard tabs
Column: 1
(MD010, no-hard-tabs)
101-101: Hard tabs
Column: 1
(MD010, no-hard-tabs)
102-102: Hard tabs
Column: 1
(MD010, no-hard-tabs)
103-103: Hard tabs
Column: 1
(MD010, no-hard-tabs)
104-104: Hard tabs
Column: 1
(MD010, no-hard-tabs)
105-105: Hard tabs
Column: 1
(MD010, no-hard-tabs)
107-107: Hard tabs
Column: 1
(MD010, no-hard-tabs)
108-108: Hard tabs
Column: 1
(MD010, no-hard-tabs)
109-109: Hard tabs
Column: 1
(MD010, no-hard-tabs)
110-110: Hard tabs
Column: 1
(MD010, no-hard-tabs)
111-111: Hard tabs
Column: 1
(MD010, no-hard-tabs)
112-112: Hard tabs
Column: 1
(MD010, no-hard-tabs)
🔇 Additional comments (7)
wren-launcher/commands/dbt/README.md (2)
14-20: Documentation accurately reflects the DataSource interface.The interface definition and explanation correctly match the expected implementation pattern. The three-method interface (GetType, Validate, MapType) is well-documented with clear descriptions of each method's purpose.
24-37: Example struct definition follows Wren API conventions.The Snowflake example struct correctly demonstrates the expected JSON tag format and field naming conventions that align with the Wren engine API documentation.
wren-launcher/commands/dbt/profiles_analyzer.go (2)
13-51: Well-structured YAML parsing with proper error handling.The AnalyzeDbtProfiles function demonstrates good practices:
- Proper error wrapping with contextual information
- Clean separation of YAML parsing and structured conversion
- Appropriate handling of optional config section
90-122: Type-safe helper functions provide robust value extraction.The helper functions (getString, getInt, getBool) properly handle type assertions and provide sensible defaults, preventing runtime panics from type mismatches in the YAML data.
wren-launcher/commands/dbt/data_source.go (3)
9-14: Well-designed DataSource interface promotes extensibility.The DataSource interface provides a clean abstraction with appropriate methods for type identification, validation, and type mapping. This design facilitates easy addition of new data source types as documented in the README.
166-199: Robust PostgreSQL data source implementation.The WrenPostgresDataSource implementation includes:
- Proper validation of required fields and port range
- Sensible default port assignment (5432)
- Clean struct definition with appropriate JSON tags
143-164: Comprehensive type mapping for local file sources.The MapType implementation covers common database types and provides a fallback strategy, ensuring robust type conversion from source systems to Wren's type system.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
wren-launcher/commands/dbt/data_source.go (2)
80-89: Document the path resolution logic.The path resolution takes the directory of the file path rather than the full path. This behavior should be documented to clarify the intent, as it might be unexpected to users.
Add a comment explaining why
filepath.Dir()is used:resolvePath := func(path string) string { + // Extract directory from file path since URL should point to the containing directory if filepath.IsAbs(path) { return filepath.Dir(path) } relativeDir := filepath.Dir(path) if dbtHome != "" { return filepath.Join(dbtHome, relativeDir) } return relativeDir }
189-192: Placeholder MapType implementation needs completion.The PostgreSQL data source's
MapTypemethod is a placeholder that returns the input unchanged. This will likely cause issues when type mapping is needed.The comment indicates this method is not used, but it's part of the required interface. Consider implementing proper PostgreSQL type mapping:
func (ds *WrenPostgresDataSource) MapType(sourceType string) string { - // This method is not used in WrenPostgresDataSource, but required by DataSource interface - return sourceType + // Map PostgreSQL types to standard types + switch strings.ToLower(sourceType) { + case "integer", "int", "bigint", "int8", "int4", "int2": + return "integer" + case "varchar", "text", "char", "character": + return "varchar" + case "timestamp", "timestamptz", "datetime": + return "timestamp" + case "date": + return "date" + case "double precision", "float8", "float4", "real", "numeric", "decimal": + return "double" + case "boolean", "bool": + return "boolean" + default: + return "string" + } }Would you like me to generate the complete PostgreSQL type mapping implementation?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
wren-launcher/commands/dbt/data_source.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: cyyeh
PR: Canner/WrenAI#1293
File: wren-ai-service/pyproject.toml:38-38
Timestamp: 2025-02-12T22:05:37.109Z
Learning: The qdrant-client package version in wren-ai-service must match the Qdrant server version (1.11.0) to maintain compatibility.
🧬 Code Graph Analysis (1)
wren-launcher/commands/dbt/data_source.go (1)
wren-launcher/commands/dbt/profiles.go (2)
DbtProfiles(4-7)DbtConnection(16-39)
🔇 Additional comments (2)
wren-launcher/commands/dbt/data_source.go (2)
18-40: LGTM - Well-structured profile conversion.The function properly handles the conversion of all profiles and their outputs to data sources, with good error handling and nil checks.
194-235: LGTM - Robust active data source selection.The function provides good defaults for profile and target selection, with comprehensive error handling for missing profiles or targets.
|
It works ! 👍 |
3 participants
Description
This PR is the first step to integrate with dbt-core. It's used to convert a qualified DBT project to the required connection information and MDL file for Wren AI.
dbt-auto-convertcommandI implement a new command for
wren-launcher:dbt-auto-convertcan accept a path of a DBT project and generate the required files in a specific output-dir:wren-mdl.jsonwren-datasource.jsonRequirement for DBT project
To ensure the DBT project is qualified and generates the required files, we should execute the following command in the DBT project home:
They will generate
catalog.jsonandmanifest.jsonfor your DBT project, which is required for the conversion.Another required file is
profile.yml, which defines the destination of your DBT project. It will be used to generate the connection info for Wren AI.New step for the launcher if enabled dbt mode
If we start the launcher with the flag, it will show an extra step for dbt integration.
New step:
Currently, this step only generates the required files in the project directory and mounts the path if required. The user can just skip the step for now.
Supported Database
Currently, it's not mature. I only convert the common type and configs for now.
How to build and test
The following command can build the binary and run the test cases.
Changed Summary
dbt-auto-convertcommand.docker-compose.yamland.env.examplefiles to mount the local path for accessing local storage.TODO
wren-uishould accept the MDL and connection info file for onboarding.Summary by CodeRabbit
New Features
Documentation
Bug Fixes
Tests
Chores