Development Release - v0.4.0

Bug Fixes

• Improve performance of date parsing (#2306)
• Remove ApiEndpoint::name to be consistent with other kinds of sinks (#2316)
• Log aerospike sink error and increase batch timeout (#2336)
• Dozer-core now propagates recoverable errors instead of panicking (#2348)
• Don't bail out if log manager can't be started (#2365)
• Read full message in aerospike tcp server (#2368)
• Fix table indexes usage in aerospike connector (#2370)
• Avoid sending repeated messages when error happens in Oracle log miner (#2373)
• Fix oracle connector update parser (#2386)
• i8 -> c_char (#2410)
• Fix dummy sink performance output (#2411)
• Oracle source skips uncommitted operations that are before another commit (#2413)
• Oracle source replication dead loop (#2416)
• Oracle redo log can have NULL REDO_SQL (#2418)
• Aggregate transactions during replication to avoid losing part of long running transactions (#2423)

Features

• Enable postgres continue ingestion (#2290)
• Snowflake sink (#2266)
• Support different kinds of endpoints (#2294)
• Implement batch processing for stateless processors (#2319)
• Continue mysql ingestion (#2322)
• Webhook-connector (#2355)
• Clickhouse sink (#2353)
• Oracle connector snapshotting (#2359)
• Oracle replication (#2361)
• Aerospike ingestion (#2366)
• Use http1 server for aerospike (#2374)
• Dozer ui and dozer ui update (#2376)
• Stop ingestion after number of events in dummy sink (#2394)
• Print average latency in dummy sink (#2395)
• Allow aerospike sink to handle multiple tables (#2393)
• Support integer on oracle ingestion/sink (#2419)

Fix

• Revert oracle resumability and fix snapshot data insert (#2425)

Miscellaneous Tasks

• Move last_checkpoint from Source::start to SourceFactory::build (#2264)
• Add latency measuring code in dummy sink (#2318)
• Diable stateful output port (#2330)
• Adapt UI for multiple tables in 1 sink (#2408)
• Add dev container (#2412)
• Stop using larger runners (#2417)
• Add oracle instant client library to docker image (#2421)
• Clean up dozer_sql::builder (#2424)
• Remove clones and duplicated code in dozer_sql::builder (#2428)
• Update version number to 0.4.0 (#2432)

Performance

• Stop using record store (#2307)
• Add Operation::BatchInsert (#2308)

Refactor

• Change OpIdentifier to RestartableState (#2263)
• Asyncify Source (#2329)
• Make source state source wide instead of per table (#2344)
• Send source state and op id to sink (#2354)
• Make connector methods take unique reference (#2358)
• Remove dozer-lambda (#2380)
• Respect source transaction boundaries (#2335)
• Handle schema and types in aerospike (#2375)
• Remove dozer-api (#2388)
• Guarentee operation order inside a transaction (#2389)
• Remove dozer-cache (#2390)

Support

Contact us at https://getdozer.io

Development Release - v0.3.0

Bug Fixes

• Runtime might crash when using multiple lambdas (#2222)
• Properly report error happened in pipeline (#2227)
• Json conversion of Decimal doesn't work with arrow-json (#2226)
• serde-v8 conversion functions don't work well with serde_json::Value (#2230)
• Use default_max_num_records in list REST endpoint (#2243)
• Avoid opening secondary envs multiple times (#2246)
• Don't retry indexing if key size is too long (#2247)
• Decimal should be serialized as a JSON Number not JSON String (#2244)
• SQL layer fixes (#2235)
• Simplify format_type and resolve an obscure bug (#2259)
• Consider API security from environment variable in pgwire (#2260)

Features

• Implement dozer run lambda (#2202)
• Include lambda running in app (#2216)
• Add termination reason and timestamp (#2223)
• Support importing modules in js apps (#2224)
• Add JavaScript UDF (#2221)
• Enable SQL queries on cache endpoints (#2196)
• Add casting from timestamp to date (#2238)
• Handle breaking and non-breaking schema changes for MySQL connector (#2250)
• Support async lambda and udf (#2239)
• Implement authentication and access filtering for pgwire using JWT (#2258)
• Support stateful javascript UDF (#2253)
• Add CheckpointedLogReader (#2257)

Fix

• Make postgres connector schema and table names case-sensitive (#2245)

Miscellaneous Tasks

• Bump version to 0.3.0 (#2207)
• Move deno dependency to dozer-deno to be shared across multiple crates (#2213)
• Make Expression::evaluate accept &mut self (#2215)
• Update arrow version (#2225)

Support

Contact us at https://getdozer.io

Development Release - v0.2.1

Bug Fixes

• Bump dozer-log-js version. Remove release workflow timeout (#2204)
• Use rust-cache and fix .deb release (#2205)

Miscellaneous Tasks

• Bump version to 0.2.1 (#2206)

Support

Contact us at https://getdozer.io

Development Release - v0.2.0

Bug Fixes

• Fix dozer connectors (#2147)
• Host from postgres connection url was not parsed correctly (#2148)
• Record writer was not checkpointed (#2165)
• Handle app restart in cache builder (#2162)
• Remove redundant idx in ProcessorRecordStore (#2180)
• ProcessorRecordStore::serialize_slice was returning wrong length (#2182)
• Stop lexically compare secondary index key (#2178)
• Fix message when permission denied to use cloud service (#2194)
• Cache name is not unique when there are multiple endpoints (#2199)
• Include -p parameter in cross build to avoid building web3 (#2200)

Features

• Store SourceStates in log and cache
• Store SourceStates in log and cache (#2158)
• Split cache name and label (#2161)
• Don't create checkpoint if source table is not restartable (#2181)
• Deno connector (#1816)

Fix

• dozer connectors command (#2146)

Miscellaneous Tasks

• Add unit tests for join processor (#2154)
• Remove printout login token (#2159)
• Bump rustix in /dozer-recordstore/fuzz (#2164)
• Remove legacy build_name and descriptor_bytes from EndpointMeta (#2172)
• Enable checkpointing for dozer and snowflake connector (#2179)
• Remove outdated !Delta table in object store connector (#2193)
• Bump version to 0.2.0 (#2203)

Refactor

• Store source states and log position in one database (#2160)
• Split connectors to separate crates (#2189)
• Include id in Record response (#2156)
• Remove duplicated code in object store connector (#2197)

Support

Contact us at https://getdozer.io

Development Release - v0.1.39

Bug Fixes

• Fix replication slot check (#1031)
• Remove non-existent proto include, which made cargo always rerun (#1064)
• Record readers not inserted in release builds (#1068)

Features

• Implement start from in postgres connector (#994)
• Add RoCacheEndpoint::redirect_cache. (#1027)
• Enhance python udf (#984)

Miscellaneous Tasks

• Refactor grpc types (#1036)

Refactor

• Make tables vector required for connector start (#1058)

Testing

• Add tests for local storage (#1030)

Support

Contact us at https://getdozer.io

Development Release - v0.1.38

What's Changed

• chore: Ignore tests that are using stocks connection by @chubei in #2014
• Bump tempfile from 3.4.0 to 3.8.0 by @dependabot in #2011
• chore: Start v0.1.38 dev by @mediuminvader in #2012
• chore: read env DOZER_MASTER_SECRET for api security by @duonganhthu43 in #1991
• chore: Debug print aws sdk errors by @chubei in #2009
• fix: Join table entries was not removed even if value is empty by @chubei in #1998
• chore: Handle empty columns in ConnectorSouceFactory by @chubei in #1244
• Bump thiserror from 1.0.47 to 1.0.48 by @dependabot in #2018
• chore: clean up health check endpoint by @duonganhthu43 in #2021
• Bump reqwest from 0.11.16 to 0.11.20 by @dependabot in #2023
• refactor: Change OpIdentifier scope to table instead of connection by @chubei in #2016
• chore: remove old methods by @v3g42 in #2015
• fix: Make Ingestor async-aware by @chubei in #2022
• fix: Fix checkpoint failure when using s3 storage by @chubei in #2026
• chore: Enable tokio-console traces by @chubei in #2019
• Add default_max_records_num to dozer-config to allow configurable default limit by @aaryaattrey in #2024
• added temp_dir to LiveState by @supergi0 in #2025
• perf: Use serde-bytes for Vec<u8> serialization by @chubei in #2027
• feat: Make DozerConnector restartable by @chubei in #2017
• Shrink ProcessorRecord by @Jesse-Bakker in #2028
• chore: add created_at, updated_at in deployment info by @duonganhthu43 in #2029
• fix: make snowflake connector async by @abcpro1 in #2032
• fix: retry on network errors in the snowflake connector by @abcpro1 in #2033
• fix: handle a postgres timeout error by @abcpro1 in #2034
• feat: Add instance type to deployment command by @karolisg in #2030
• fix override logic for live by @aaryaattrey in #2035
• fix: resume a non-SELECT query in snowflake connector by @abcpro1 in #2036
• feat: Add subscribe interface by @chubei in #2037
• chore: Update tonic crates by @chubei in #2039
• fix: snowflake bug fixes by @abcpro1 in #2040
• fix: mysql use a random server_id if none was specified by @abcpro1 in #2041
• Bump rust_decimal from 1.28.1 to 1.32.0 by @dependabot in #2042
• Bump sqllogictest from 0.15.3 to 0.16.0 by @dependabot in #2043
• Bump hex-literal from 0.3.4 to 0.4.1 by @dependabot in #2044

Full Changelog: v0.1.37...v0.1.38

Development Release - v0.1.37

What's Changed

• chore: Start v0.1.37 dev by @mediuminvader in #1968
• Bump futures-util from 0.3.27 to 0.3.28 by @dependabot in #1966
• Bump url from 2.4.0 to 2.4.1 by @dependabot in #1970
• check for empty endpoint array by @aaryaattrey in #1975
• Add path to endpoint schema by @Jesse-Bakker in #1978
• new readme by @snork-alt in #1976
• chore: Remove duplicated code in JoinOperator by @chubei in #1971
• chore: Remove source state merging from ReceiverLoop to EpochManager by @chubei in #1982
• chore: Make clippy happy by @Jesse-Bakker in #1983
• fix: Use span for orchestrator error by @karolisg in #1973
• added step by step section by @snork-alt in #1987
• Bump rustls from 0.21.6 to 0.21.7 by @dependabot in #1985
• Clippy all targets in lint workflow by @Jesse-Bakker in #1988
• Add Display impl to SecondaryIndex by @Jesse-Bakker in #1993
• feat: Restore record store, processors and logs from checkpoints by @chubei in #1984
• Bump rustls-native-certs from 0.6.2 to 0.6.3 by @dependabot in #1995
• feat: Add endpoint and timeout_in_seconds to XRayConfig by @chubei in #2000
• chore: Remove unused code by @chubei in #2001
• chore: Add checkpoint config options and disable checkpoint in live by @chubei in #2003
• Add allow-incompatible flag to cloud deploy by @Jesse-Bakker in #2006
• fix: mysql select query offset syntax is wrong by @abcpro1 in #2007
• change on_event to OnEvent by @aaryaattrey in #2005
• Bump bson from 2.6.1 to 2.7.0 by @dependabot in #2008
• fix: Don't call Storage::delete_objects with empty objects array by @chubei in #2010
• chore: configure non reserved ports for by @v3g42 in #2002

Full Changelog: v0.1.36...v0.1.37

Development Release - v0.1.36

What's Changed

• Make file watcher non-blocking by @Jesse-Bakker in #1918
• feat: Use custom CountingBloomFilter that implements serde by @chubei in #1917
• Bump ordered-float from 3.4.0 to 3.9.1 by @dependabot in #1919
• chore: setup MySQL and MariaDB services in the CI by @abcpro1 in #1920
• test: unittests for the MySQL connector by @abcpro1 in #1867
• fix: Remove duplicated UI graph nodes by @chubei in #1921
• Bump serde_yaml from 0.9.19 to 0.9.25 by @dependabot in #1927
• chore: derive Hash for S3Storage by @chubei in #1924
• Tempfix: added MongoDB, MySQL and S3 templates to dozer init by @supergi0 in #1928
• Remove some unnecessary tokio tasks in mongodb connector by @Jesse-Bakker in #1923
• chore: Remove unused generic type parameter in dozer-core by @chubei in #1929
• feat: on_event can subscribe to multiple endpoints at once by @chubei in #1881
• Change command to run_all to dozer run by @Jesse-Bakker in #1935
• fix: Include connection type in GenerateDot. Fix AggregationProcessorFactory::type_name by @chubei in #1934
• chore: Increase e2e test timeout to 90 minutes by @chubei in #1936
• fix: Don't show Table processor in dot graph by @chubei in #1937
• Add version to config and contract by @Jesse-Bakker in #1938
• Bump clap from 4.3.11 to 4.4.1 by @dependabot in #1941
• feat: Label metrics from live run with "live_run_id" by @chubei in #1943
• Bump rustyline-derive from 0.8.0 to 0.9.0 by @dependabot in #1946
• chore: Create unit tests workflow by @mediuminvader in #1910
• fix: Unit tests concurrency group by @mediuminvader in #1949
• Write dozer.lock by @Jesse-Bakker in #1945
• feat: Add sink_operation metric by @chubei in #1948
• chore: Serve live frontend at 62999 by @chubei in #1950
• chore: Remove unnecessary error log by @chubei in #1952
• feat: Add --locked flag to build and run by @Jesse-Bakker in #1951
• fix: Check if all output tables are used in endpoints by @chubei in #1953
• handle-invalid-request by @aaryaattrey in #1885
• Bump mongodb from 2.6.0 to 2.6.1 by @dependabot in #1955
• fix: Fix duplicate INTO check by @chubei in #1957
• chore: Refactor QueryExpression extractor implementation by @chubei in #1960
• fix: Allow to use an optional app_id argument in delete command by @mediuminvader in #1939
• chore: Prepare v0.1.36 by @mediuminvader in #1961
• fix: proto name error by @aaryaattrey in #1962
• feat: onnx runtime integration by @chloeminkyung in #1838
• feat: Implement get_protos by @v3g42 in #1963
• Upload lockfile when deploying by @Jesse-Bakker in #1965
• fix: resume connectors on network errors by @abcpro1 in #1956

Full Changelog: v0.1.35...v0.1.36

Development Release - v0.1.35

What's Changed

• chore: Start v0.1.35 dev by @mediuminvader in #1870
• Wait for connectors to stop on shutdown by @Jesse-Bakker in #1865
• fix: start dozer live ui server by @duonganhthu43 in #1874
• chore: Remove short form of enable_progress because it's conflicting with dozer cloud by @chubei in #1876
• chore: Enable push_events by default by @chubei in #1877
• chore: Add Access deserialization test for All variant by @chubei in #1878
• test: provide a MySQL service for CI coverage tests by @abcpro1 in #1879
• chore: Add flag to disable live ui for local development by @supergi0 in #1880
• feat: Write record store checkpoints by @chubei in #1854
• ci: introduce labeler workflow by @tungbq in #1884
• chore: Run e2e tests nightly by @mediuminvader in #1886
• feat: Dozer connector by @Jesse-Bakker in #1883
• docs: advertise support for the MySQL connector by @abcpro1 in #1872
• bump rustls-webpki from 0.100.1 to 0.100.2 by @dependabot in #1889
• chore: Split ContractService from CodeService by @chubei in #1890
• fix: CommonGrpcService::getEndpoints should preserve order by @chubei in #1893
• Clean up postgres snapshotter by @Jesse-Bakker in #1895
• feat: add endpoint for latest deployment status and quick health check by @duonganhthu43 in #1894
• fix: do not require primary key for CdcType::FullChanges by @abcpro1 in #1896
• Add merge_group trigger for e2e-test GHA workflow by @Jesse-Bakker in #1897
• Skip e2e workflow on pull requests by @Jesse-Bakker in #1900
• chore: Move ContractService implementation to Contract by @chubei in #1899
• fix: introduce build states and fix inconsistencies by @v3g42 in #1902
• chore: Hint user to run force build when contract format changes by @chubei in #1903
• add mongodb feature to dozer-cli package by @aaryaattrey in #1905
• chore: Remove LogOperation::Terminate by @chubei in #1907
• chore: Add PipelineContract type alias by @chubei in #1904
• fix: Put live cache in temporary directory. by @chubei in #1906
• Run sql-tests with custom libtest-mimic harness by @Jesse-Bakker in #1901
• Abort starting log reader when shutting down by @Jesse-Bakker in #1909
• chore: Update for Rust 1.72.0 by @chubei in #1911
• Bump thiserror from 1.0.39 to 1.0.47 by @dependabot in #1914
• feat: make probabilistic optimizations optional and tunable in the config by @abcpro1 in #1912

New Contributors

• @tungbq made their first contribution in #1884

Full Changelog: v0.1.34...v0.1.35

Development Release - v0.1.34

What's Changed

• chore: Start v0.1.34 dev by @mediuminvader in #1845
• feat: Publish Source contracts as JSON file by @mediuminvader in #1824
• load config from stdin by @aaryaattrey in #1839
• fix: Support uploading empty multipart file to s3 by @chubei in #1846
• Add Mongodb connector MVP by @Jesse-Bakker in #1810
• Add mongodb connector e2e test by @Jesse-Bakker in #1849
• feat: dowload and use dozer-live ui by @supergi0 in #1848
• fix: Mark log creation to avoid writing to old log version by @chubei in #1850
• chore: Remove dead code by @chubei in #1853
• chore: Test dozer CI with workflow_dispatch by @mediuminvader in #1859
• error for SELECT statements with no INTO clause by @aaryaattrey in #1851
• fix: Don't start api until internal server is up by @chubei in #1861
• feat: replace jaeger with xray by using ADOT by @duonganhthu43 in #1862
• feat: implement graph on live ui by @v3g42 in #1847
• feat: Introduce MySQL Connector by @abcpro1 in #1829
• fix: Should not panic if source sender errors out because of channel error by @chubei in #1864

New Contributors

• @aaryaattrey made their first contribution in #1839
• @Jesse-Bakker made their first contribution in #1810
• @supergi0 made their first contribution in #1848

Full Changelog: v0.1.33...v0.1.34