Package pump
provides a minimalist framework for composing data processing pipelines.
The pipelines are type-safe, impose little overhead, and can be composed either statically,
or dynamically (for example, as a function of configuration). A running pipeline stops on and
returns the first error encountered.
The package defines two generic types:
- Data generator
G[T]
: a callback-based ("push") iterator that supplies a stream of data of any typeT
, and - Pipeline stage
S[T,U]
: a function that invokes input generatorG[T]
, does whatever processing it is programmed to do, and feeds the supplied callback with data items of typeU
.
The package also provides a basic set of functions for composing pipeline stages and binding stages to generators, as well as a stage that runs its generator in a separate goroutine, and parallel execution of stages.
For more details see documentation.
Tested on several Linux Mint versions from 21.x range. Requires Go version 1.21 or higher.