v4.45.1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Fixed

• Empty files read by input aws_s3 no longer cause spurious errors. (@rockwotj)
• Fixes a SIGSEGV in postgres_cdc when using TOAST values with tables that don't have FULL replica identity. (@rockwotj)

The full change log can be found here.

v4.45.0

For installation instructions check out the getting started guide.

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@mihaitodor)
• The postgres_cdc now correctly handles null values. (@rockwotj)
• The redpanda_migrator output no longer rejects messages if it can't perform schema ID translation. (@mihaitodor)
• The redpanda_migrator input no longer converts the kafka key to string. (@mihaitodor)

Added

• aws_sqs input now has a max_outstanding field to prevent unbounded memory usage. (@rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
• Field content_type added to the amqp_1 output. (@timo102)
• Field fetch_max_wait added to the kafka_franz, ockam_kafka, redpanda, redpanda_common and redpanda_migrator inputs. (@birdayz)
• snowpipe_streaming output now supports interpolating table names. (@rockwotj)
• snowpipe_streaming output now supports interpolating channel names. (@rockwotj)
• snowpipe_streaming output now supports exactly once delivery using offset_token. (@rockwotj)
• ollama_chat processor now supports tool calling. (@rockwotj)
• New ollama_moderation processor which allows using LlamaGuard or ShieldGemma to check if LLM responses are safe. (@rockwotj)
• Field queries added to sql_raw processor and output to support rummong multiple SQL statements transactionally. (@rockwotj)
• New redpanda_migrator_offsets input. (@mihaitodor)
• Fields offset_topic, offset_group, offset_partition, offset_commit_timestamp and offset_metadata added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field topic_lag_refresh_period added to the redpanda and redpanda_common inputs. (@mihaitodor)
• Metric redpanda_lag now emitted by the redpanda and redpanda_common inputs. (@mihaitodor)
• Metadata kafka_lag now emitted by the redpanda and redpanda_common inputs. (@mihaitodor)
• The redpanda_migrator_bundle input and output now set labels for their subcomponents. (@mihaitodor)
• (Benthos) Field label added to the template tests definitions. (@mihaitodor)
• (Benthos) Metadata field label can now be utilized within a template's mapping field to access the label that is associated with the template instantiation in a config. (@mihaitodor)
• (Benthos) bloblang scalar type added to template fields. (@mihaitodor)
• (Benthos) Go API: Method SetOutputBrokerPattern added to the StreamBuilder type. (@mihaitodor)
• (Benthos) New error_source_name, error_source_label and error_source_path bloblang functions. (@mihaitodor)
• (Benthos) Flag --verbose added to the benthos lint and benthos template lint commands. (@mihaitodor)

Changed

• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)
• Fix an issue with postgres_cdc with TOAST values not being propagated with REPLICA IDENTITY FULL. (@rockwotj)
• Fix a initial snapshot streaming consistency issue with postgres_cdc. (@rockwotj)
• Fix bug in sftp input where the last file was not deleted when watcher and delete_on_finish were enabled. (@ooesili)
• Fields batch_size, multi_header, replication_factor, replication_factor_override and output_resource for the redpanda_migrator input are now deprecated. (@mihaitodor)
• Fields kafka_key and max_in_flight for the redpanda_migrator_offsets output are now deprecated. (@mihaitodor)
• Field batching for the redpanda_migrator output is now deprecated. (@mihaitodor)
• The redpanda_migrator input no longer emits tombstone messages. (@mihaitodor)
• (Benthos) The branch processor no longer emits an entry in the log at error level when the child processors throw errors. (@mihaitodor)
• (Benthos) Streams and the StreamBuilder API now use reject by default when no output is specified in the config and stdout isn't registered (for example when the io components are not imported). (@mihaitodor)

The full change log can be found here.

v4.45.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• aws_sqs now has a max_outstanding field to prevent unbounded memory usage. (@rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
• Field content_type added to the amqp_1 output. (@timo102)
• kafka_franz, ockam_kafka, redpanda, redpanda_common, redpanda_migrator now support fetch_max_wait configuration field.
• snowpipe_streaming now supports interpolating table names (@rockwotj)
• snowpipe_streaming now supports interpolating channel names (@rockwotj)
• snowpipe_streaming now supports exactly once delivery using offset_token (@rockwotj)

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@mihaitodor)
• The postgres_cdc now correctly handles null values. (@rockwotj)
• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)

The full change log can be found here.

v4.44.0

For installation instructions check out the getting started guide.

Added

• Go API: New public/license package added to allow custom programmatic instantiations of Redpanda Connect to run enterprise license components. (@Jeffail)

Fixed

• gcp_bigquery output with parquet format no longer returns errors incorrectly. (@rockwotj)
• postgres_cdc input now allows quoted identifiers for the table names. (@mihaitodor, @rockwotj)

The full change log can be found here.

v4.43.1

For installation instructions check out the getting started guide.

Fixed

• Trial Redpanda Enterprise licenses are now considered valid. (@Jeffail)
• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured. (@mihaitodor)

The full change log can be found here.

v4.43.0

For installation instructions check out the getting started guide.

Changed

• The pg_stream input has been renamed to postgres_cdc. The old name will continue to function as an alias. (@rockwotj)
• The postgres_cdc input no longer emits mode metadata and instead snapshot reads set operation metadata to be read instead of insert. (@rockwotj)

Fixed

• The redpanda_migrator_bundle output no longer attempts to translate schema IDs when a schema registry is not configured. (@mihaitodor)

The full change log can be found here.

v4.42.0

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.42.0-rc1

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.41.0

For installation instructions check out the getting started guide.

Added

• Field max_records_per_request added to the aws_sqs output. (@Jeffail)

Fixed

• (Benthos) Fixed an issue where running a CLI with a custom environment would cause imported templates to be rejected. (@Jeffail)

Changed

• The -cgo suffixed docker images are no longer built and pushed along with the regular images. This decision was made due to low demand, and the unacceptable cadence with which the image base (Debian) receives security updates. It is still possible to create your own CGO builds with the command CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. (@Jeffail)

The full change log can be found here.

v4.40.0

For installation instructions check out the getting started guide.

Added

• New pg_stream input supporting change data capture (CDC) from PostgreSQL. (@le-vlad)
• Field metadata_max_age added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator outputs. (@mihaitodor)
• (Benthos) New Bloblang method timestamp. (@mihaitodor)
• (Benthos) New benchmark processor. (@ooesili)

Fixed

• Addresses an issue where snowflake_streaming could create more channels than configured. (@rockwotj)

Changed

• The snowflake_streaming output with schema_evolution.enabled set to true can now autocreate tables. (@rockwotj)
• Fields translate_schema_ids and schema_registry_output_resource added to the redpanda_migrator output. (@mihaitodor)
• Fields backfill_dependencies and input_resource added to the schema_registry output. (@mihaitodor)
• The schema_registry input and output and the schema_registry_encode and schema_registry_decode processors now use the github.com/twmb/franz-go/pkg/sr SchemaRegistry client. (@mihaitodor)
• Metadata field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs now contains a unix timestamp with millisecond precision. (@mihaitodor)
• Metadata field kafka_timestamp removed from the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs. (@mihaitodor)

The full change log can be found here.