PETL: A Pipeline ETL process and receive data from pipe in Golang

Installation and Usage

Install

go get github.com/kkdai/petl

Usage

func main() {
	ret := PipeProcess(Extract("a  s", " smm", "sss"), TransformRemoveSpace, TransformRemoveSpace)
	log.Println("ret:", <-ret) //"as"
	log.Println("ret2:", <-ret) //"smm"
}

Install Console App

go get github.com/kkdai/petl/petl_cli

Usage

Use pipeline for pipe string

//Remove all space in output
cat somefile.txt | petl_cli pipeline="r"

//Make upper case
cat somefile.txt | petl_cli pipeline="u"

//Make lower case 
cat somefile.txt | petl_cli pipeline="l"

//Pipeline it
//Will make lower case first then upper case, final remove all space.
cat somefile.txt | petl_cli pipeline="lur"

Inspired By

• Experimenting with Go pipelines
• Go Blog: Go Concurrency Patterns: Pipelines and cancellation
• Comparing Golang, Scala, Elixir, Ruby, and now Python3 for ETL: Part 2
• stackoverflow: Can Functions be passed as parameters in Go?

Project52

It is one of my project 52.

License

etcd is under the Apache 2.0 license. See the LICENSE file for details.