Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One-shot operations #448

Open
hvisage opened this issue Apr 29, 2015 · 2 comments
Open

One-shot operations #448

hvisage opened this issue Apr 29, 2015 · 2 comments

Comments

@hvisage
Copy link

hvisage commented Apr 29, 2015

Goodday,

I'd like to have a "one shot" type setting/option.

Reason: I'm importing mass numbers of files from an operation that mv/move/rename the files into the directory I need to process, and then the files stays static, ie. nothing added to them again, but the number grows.

I want to tell logstash-forwarder, to read the files (or even better from stdin), and once the EOF have been reached, to close the file(s), and exit when all of them have been processed. I can then mark/move/etc. the files out of the way, and wait for the next batch and restart/run logstash-forwarder on that set of files.
Currently, logstash-forwarder "resets" the stdin input (even though nothing is sending there anymore/closed it down) and then I'll have to kill it myself. This can't be automated reliably to know when logstash-forwarder have sent all the entries reliably to the remote/central logstash instance ;(

@hvisage hvisage changed the title One-shop operations One-shot operations Apr 29, 2015
@hvisage hvisage mentioned this issue Apr 29, 2015
@hvisage
Copy link
Author

hvisage commented May 8, 2015

I'm looking at (learning GO and) to implement this "option".

To do this, I want to do something like this:

1a) Harvester threads to check for EOF on it's inputs and quit when:
i) it's harvesting "-"
ii) it's harvesting a file AND the --oneshot flags have been set.

1b) the last harvester thread to close the output channel when it exits.

  1. The various channel "listeners" will be checking for close (ie. false status) on their channels (Those are all single instances). Once their channel is closed, they will close their respective output/sending channels and "exit"/quit their thread. This should have a cascading effect starting from the spooler, to the publisher to the registrar.
    Once the registrar finds a closed channel, it would then quit and the application quit.

Any comments??

@driskell
Copy link
Contributor

driskell commented May 8, 2015

@hvisage You're welcome to look at Log Courier's source for inspiration. It's forked from LSF and already has this feature via a command line flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants