v1.2.22

Sling v1.2.22

1. Direct Insert Support

• Added new SLING_DIRECT_INSERT environment variable to enable direct table inserts
• Implemented writeToDbDirectly() function to bypass temp table creation
• Added new test cases for direct insert functionality

2. File Path Improvements

• Added CleanTableName() function to sanitize table names in temp file paths
• Updated temp file paths across multiple database connectors to use cleaned table names
• Made temp file paths more consistent and descriptive

3. Stream Configuration Updates

• Refactored stream configuration handling to use structured configs instead of maps
• Added LoaderStreamConfig() for standardized loader configurations
• Improved compression and datetime format handling

4. Database Connector Enhancements

• Added trusted connection support for SQL Server
• Updated Prometheus client authentication
• Improved error handling and transaction management
• Enhanced column casing support for upserts

v1.2.21

Sling v1.2.21

Main Changes

1. Dependency Updates

   • Updated several Go module dependencies, including golang.org/x/crypto, golang.org/x/exp, golang.org/x/net, golang.org/x/sys, golang.org/x/term, golang.org/x/text, golang.org/x/time, golang.org/x/tools, and various Google APIs.
   • Updated gorm.io/gorm from v1.25.9 to v1.25.10.
   • Updated github.com/trinodb/trino-go-client from v0.316.0 to v0.318.0.

2. GitHub Actions Updates

   • Updated the Infisical action from v2 to v3 across multiple workflow files.
   • Modified environment variable handling in workflows.

3. Code Improvements

   • Enhanced error handling and logging in various parts of the codebase.
   • Improved configuration parsing and handling.
   • Updated CLI command processing and output formatting.

4. Documentation Updates

   • Significantly revised the README.md, particularly the "Contributing" section.
   • Added more detailed guidelines for branch naming, testing, and pull request processes.

5. Testing Enhancements

   • Added new test files and updated existing test suites.
   • Improved test configuration and execution instructions.

v1.2.20

Sling v1.2.20

Bug Fixes

• Improved error handling in MergeDataflow function
  • Now properly closes datastream on error
• Fixed potential issue in MakeShaper function
  • Now uses Names(true) to get column names, ensuring consistent behavior

v1.2.19

Sling v1.2.19 (2024-09-22T20:03:08Z)

Summary of Changes

Configuration and Templates

• Updated various database templates (e.g., Azure SQL, BigQuery, Clickhouse, DuckDB, MariaDB, MySQL, Oracle, Postgres, Redshift, Snowflake, SQLite, SQL Server, StarRocks, Trino) to include:

  • Added schema_name and is_view columns to table and view queries
  • Modified queries to support optional schema filtering
  • Standardized ordering of results

• Updated DuckDB template:

  • Added stream scanner functions for various file formats
  • Added timestamp layout variables

• Updated Motherduck template:

  • Added scan functions for Iceberg, Delta, and Parquet formats

Core Functionality

• Enhanced environment variable handling in env.go
• Improved error handling and panic recovery in config unmarshalling
• Added support for SQL-based streaming in file sources
• Enhanced incremental value handling for database sources
• Improved wildcard processing for replication configs
• Added dataset output option for file targets

Build and CI Scripts

• Added connection discovery step to build scripts
• Created new preparation scripts for Linux, macOS, and Windows

Testing

• Updated test scripts to use a centralized CLI test

Dependencies

• Updated various Go module dependencies

v1.2.18

Sling v1.2.18 (2024-08-30T08:29:24Z)

Changelog

• Fix sling conns test

v1.2.17

Sling v1.2.17 (2024-08-28T16:39:52Z)

Changelog

• Fix Setting & GetMachineID
• Add home/sling in Dockerfiles
• Update caching and logging

v1.2.16

Sling v1.2.16 (2024-08-27T21:25:58Z)

Changelog Summary

Docker and CI/CD

• Updated Dockerfiles and CI workflows
• Improved build processes for dev and production
• Added self-hosted Mac to build-dev workflow

Database Connections and Operations

• Enhanced connection handling for various databases (DuckDB, Snowflake, Oracle)
• Improved timestamp and timezone handling
• Added support for Delta and Iceberg formats
• Enhanced file system operations and caching

Testing and Error Handling

• Added and updated various tests
• Improved error handling and logging
• Added panic recovery in Parquet operations

Features and Improvements

• Added column constraints and improved column ordering
• Enhanced replication and incremental mode functionality
• Improved wildcard matching and folder glob operations
• Added support for Azure service principal keys
• Implemented connection objects and entries

Code Refactoring and Cleanup

• Moved and reorganized various functions and types
• Cleaned up and updated dependencies (go.mod)
• Improved code structure and naming conventions

Security and Configuration

• Updated SSH/SFTP key exchanges
• Improved secrets handling in CI/CD
• Updated gitignore and other configuration files

v1.2.15

Sling v1.2.15

Major Enhancements

• Performance Improvements

  • Enhanced CastVal function
  • Improved setGetMetadata for StarRocks
  • Updated fs.GetRefTs to handle timestamp values
  • Improved Replace0x00 function

• New Features

  • Added SSH tunnel support for FTP, S3 (MinIO), and SFTP
  • Implemented KEEP_TEMP_FILES option for BigQuery and StarRocks
  • Added EnvFileConnectionEntries and LoadSlingEnvFileBody
  • Introduced store.Settings
  • Implemented primary key and index table-key logic
  • Added column coercion for BigQuery, Snowflake & Redshift sources

Bug Fixes

• Resolved issues with ClickHouse tests and DDL with primary key
• Fixed blank value logic when using JSON
• Addressed ds.SetFields functionality
• Fixed MotherDuck checksum_boolean issue

Database-Specific Updates

• ClickHouse: Improved DDL handling with primary key
• BigQuery: Allow partitioning with custom SQL
• Snowflake, BigQuery, DuckDB, Redshift: Added dummy select for create_index & drop_index
• Oracle: Updated client handling
• Microsoft SQL Server: Upgraded driver

Configuration and Logging

• Enhanced logging throughout the application
• Improved DefaultStreamConfig to preserve transforms
• Updated SQL logging to retain full text

Dependency Updates

• Multiple updates to github.com/flarco/g
• Upgraded github.com/microsoft/go-mssqldb

Miscellaneous

• Expanded allowed glob pattern
• Improved sling update process
• Added warning for large single parquet file writes
• Updated README and documentation

v1.2.14

Sling v1.2.14 (2024-07-25T13:35:14Z)

General Improvements and Fixes

• Enhanced error handling and messages:
  • Improve errors (d89b764)
  • Improve fs.Delete error messages (4299414, 9d65968)
  • Improve connection error logic (4715616)
  • Fix SQLite binary location (d38ed0f)
  • Update processConns return (8deba9b)
  • Add proper os.Exit (86c541f)
  • Use legacy FILEPATH_SLUG (f007fde)
• Improved functions and logic:
  • Improve CastToStringSafeMask (c625f8c)
  • Improve limit logic & templates (c3c8e47)
  • Fix Table.Select for SQL Server offset (745f0b2)
  • Allow URI connection for wildcard (9233fe6)
  • Improve Snowflake GetFile (eb69995)
  • Clean up BulkExportFlow tables var (571a266)
  • Optimize Snowflake BulkExportFlow (5dde593)
  • Add partition functions (5765734)
  • Update BulkExportFlow params (6d82013)
  • Add IsJSONorYAML (8d70514)
  • Update store struct & logic (815628c)
  • Improve FormatTargetObjectName (6002868, ff7d4d9)
  • Improve ParseTableName (c6c3ff1)
  • Use CastToStringSafeMask for processBwRows (4732505)
• Fixed specific issues:
  • Fix schemata_test (0fec0e2)
  • Fix Snowflake StageGET (0af47e2)
  • Fix Table.Select for SQL Server offset (745f0b2)
  • Use SLING_TEMP_CLOUD_FOLDER (d76949b)

Database Enhancements

• Improved PostgreSQL and Snowflake functionalities:
  • Improve PostgreSQL schemata columns listing (materialized views) (8a2198c)
  • Fix Snowflake StageGET (0af47e2)
  • Improve Snowflake GetFile (eb69995)
  • Optimize Snowflake BulkExportFlow (5dde593)
  • Customize the batch size for proton (5cbce10)
  • Update database_proton.go (adaed27)
• Added and refined Snowflake and DuckDB features:
  • Update DuckDB version error (f68d71a)
  • Upgrade DuckDB to 1.0.0 (4da9b34)
  • Add offset option (726dd02)
  • Update processConns return (8deba9b)
  • Use SLING_TEMP_CLOUD_FOLDER (d76949b)
• Enhanced MongoDB connection handling:
  • Close cursor when done (e7a1f39)

Code and Dependency Updates

• Updated dependencies and modules:
  • Update github.com/flarco/g (2946b04, d70c8b5)
  • Upgrade Go modules to 1.22 (c195e79)
  • Update Go mod minor versions (fc70cc0)
  • Upgrade github.com/parquet-go/parquet-go v0.23.0 (b8d6fe2)

New Features and Configurations

• Added new functionalities:
  • Add IsJSONorYAML (8d70514)
  • Ignore test 22 for Oracle (129ceef)
  • Clean up BulkExportFlow tables var (571a266)
  • Add partition functions (5765734)
  • Customize the batch size for proton (5cbce10)
  • Override insert_from_table SQL template (3ec03ca)
  • Update MD5 logic, add lookupReplication (41e5ba0)
  • Use replication.OriginalCfg (a9b9a6e)
  • Add props to parquet writer (fdcd75d)
• Updated configurations and parameters:
  • Update BulkExportFlow params (6d82013)
  • Clean up store (4e43b84)
  • Update store struct & logic (815628c)
  • Improve FormatTargetObjectName (6002868, ff7d4d9)
  • Improve ParseTableName (c6c3ff1)
  • Use CastToStringSafeMask for processBwRows (4732505)

Test and Documentation Updates

• Improved tests and documentation:
  • Fix schemata_test (0fec0e2)
  • Improve tests (72d1488)
  • Update README, mention timeplus proton (ea2592d)

v1.2.13

Sling v1.2.13 (2024-06-29T16:06:47Z)

Changelog

• 8bfa6f8 update goreleaser version