Merge listener and replay functionality

[buger]

I like architecture of http://fluentd.org/, it have multiple input and output sources, and fluentd instances can be combined like LEGO bricks.

I want same for Gor, and think It should be logical move, and it will improve overall flexibility.

Examples of current functionality:

Listen data raw socket and replay it different Gor instance

gor --input-raw 9000 --output-tcp replay.server:27017
gor --input-tcp 27017 --output-http staging.dev

File replay:

gor --input-raw 9000 --output-file requests.gor # capture
gor --input-file requests.gor --output-http staging.org # replay

Replay to multiple hosts:

gor --input-raw 9000 --output-tcp replay.server:27017
gor --input-tcp 27017 --output-http staging.org --output-http "dev.org|10"

Examples of functionality that will be possible after change:

Listen and replay with one Gor instance:

gor --input-raw 9000 --output-http "staging.org"

HTTP Input #45:

gor --input-http 8080 --output-tcp "replay.org:27017"

And much more.

What would you think @slawosz @dcarley ?

[slawosz]

I like the idea and new, clean flag names.

[buger]

This is also allow to bring much more Go-like architecture with simple modules and interfaces.

[buger]

Here is basic architecture diagram:

[dcarley]

👍 from me.

[buger]

@slawosz @dcarley pls check this branch, concept is working, and it's more awesome than i thought :)

https://github.com/buger/gor/tree/modules_refactoring

You can start by playing like this:

gor --input-dummie 0 --output-dummie 0

And continue with smth like:

gor --input-raw :27017 --output-http localhost:27018

[slawosz]

So, after 2 minutes of looking (no more time now :() here is core of gor now?: https://github.com/buger/gor/blob/modules_refactoring/emitter.go#L20

Each type of i/o implements io.{Read,Write} and uses bytes?

[dcarley]

Wow, I like the look of it. Will have a play around.

[buger]

@slawosz basically yes. Input and Output just satisfy standard io.Reader and io.Writer interfaces, which gives you ability to use any standard library (bufio ?).

[slawosz]

It is beautiful piece of code 👍

[buger]

Converted issue to pull-request

[slawosz]

Why we have copy here?

[buger]

Maybe it's not needed here. Just want to make sure that data will be immutable.

[buger]

Oh, btw, i remembered. It needed :)

Thing is that io.Reader interface says that Read function should received slice as argument and we should write to it.

If i just do data = buf it will rewrite original data pointer, and original data still be empty. Thats why we use copy.

[buger]

This pull-request is long-going, and require adding lot of tests. Also it not support rate-limiting.

[slawosz]

Oh, testing right now will be much easier :)

[buger]

@slawosz @dcarley I added tests and missing plugins. Now this branch should support all features of previous versions (except some minor stuff like custom headers and elastic search stats, but this will be added soon). If you have time pls test it. Thanks!

[buger]

Added custom headers and ElasticSearch plugin. Also added option to round-robin traffic if have multiple outputs:

--split-output=true

[buger]

MERGED!

[dcarley]

Awesome work 🍻