Rename default pipeline or ask users to set it

[suyograo]

The default pipeline in LS is named as main. This was an ok choice back then, given the understanding we had then about multi-pipeline and other future features. With the pipeline viewer project and multipipelines landing in 6.x, we think it makes sense to rethink our naming of the default pipeline.

The idea is to force the user to change the pipeline name to something more descriptive of what it does, like apache, cloudwatch_logs or so, but it is really hard to choose a name without knowing the pipeline's context. The worry here is that every pipeline would be named main, which makes troubleshooting in the UI really hard. There are a couple of options:

1. Force user to set a pipeline name, without which LS wouldn't start. This change is pretty disruptive and many many folks have been doing -f and -e. Does it mean we have to add an option to set the pipeline name in the config? Or as a --pipeline.name CLI option?
2. Rename main to _default_pipeline_name_ so it becomes really obvious this needs to be changed. This is kinda analogous to how ES names its cluster name as my-application. It also makes it glaringly obvious in the UI (pipeline listing) if the name is _default_pipeline_name_.

Breaking change

The rename (regardless if we do it now, or if a user chooses to do it at any point) is not simple since we have the pipeline name as a namespace for PQ and DLQ. For example, it is now path.data/main/queue. Changing this will mean users would have to drain their queue and then rename it. This point (thanks for raising it @robbavey) makes option #1 compelling.

Thoughts?

[jakelandis]

My $.02

• Rename main -> default
• Prevent LS from starting if 2 pipelines have the same name.
• Prevent LS from starting if there is PQ or DLQ directory that is un-accounted for and not empty.
• Ensure failure to start error messages are word such that they are actionable.

This prevents confusion by ensuring uniquely named queues. Forces the user to willfully delete data if not fully drained while attempting to change names. Keeps the out of box experience cleaner.

Main vs Default - IMO Calling one "main" presumes more importance over others, where "default" means I the one I didn't bother to change.

[guyboertje]

Caveats to Jake's third bullet point.

• We would need to instruct a user to copy the PQ/DLQ folder to another disk if they wanted to keep the contents for e.g. diagnostics.

• Although the requests for near-realtime have not materialised as yet, in the early days of PQ blue-sky usage discussions. It was suggested that people who want the newest data first AND had a queue full of events on start would want to background process the queued events while keeping new events flowing. With multiple pipelines this is now possible but they would have to rename the old queue folder and the pipeline to another name. This suggests a drain-and-stop feature at a pipeline level.

[jsvd]

What if, by default, we name the pipeline the same as the hash? it is the same practice we have for plugin ids.

having a rather cryptic name will create the pressure to provide a better name

Prevent LS from starting if 2 pipelines have the same name.

This is already checked 👍

Prevent LS from starting if there is PQ or DLQ directory that is un-accounted for and not empty.

This is has some interesting implications..for example, namely if you are reloading pipelines, using configuration management, would mean deleting a pipeline during the reload could either cause logstash to stop, or be in a state that it would never restart until deletes are performed

[jakelandis]

would mean deleting a pipeline during the reload could either cause logstash to stop, or be in a state that it would never restart until deletes are performed

I think this is OK. If a user cares enough about their data to use PQ or DLQ, they would probably be thankful that we don't allow orphaning of data. This does raise the issue of having a reliable health check that CM systems could hook into.

we name the pipeline the same as the hash?

What would be the input to the hash ? If it is the configuration, then when the config changes for any reason, then the hash would change? Also, we need to be careful with assumptions of deterministic hashes across restarts.

[jsvd]

What would be the input to the hash ? If it is the configuration, then when the config changes for any reason, then the hash would change? Also, we need to be careful with assumptions of deterministic hashes across restarts.

We already compute hashes for pipelines based on the contents of the config files. Now the LIR code brings a more advanced way of computing hashes with benefits like being blind to whitespace.

[jakelandis]

@jsvd - sorry, I wasn't clear. I was implying that if using a hash of the config as the name of pipeline, then any changes to the config would result in re-naming the pipeline.

[jsvd]

@jakelandis indeed, users will see little value in this default value, also it has a negative impact as lifecycle becomes more complex, which are two characteristics we want, that will nudge users towards naming their pipelines

[colinsurprenant]

• Not sure I understand the original concern of having multiple pipelines named "main" if per @jsvd we already "prevent LS from starting if 2 pipelines have the same name"?

• If we feel we want to go away from "main", I am +1 on using "default".

• The only relation between a PQ and a pipeline is the PQ dir name. This already provides some failsafe but it won't prevent a user from changing a pipeline config which becomes invalid with a PQ content for the same name. I am not sure we want/need to add more failsafes for this specifically. I personally prefer the fluidity of keeping it simple: dir name -> pipeline name.

Prevent LS from starting if there is PQ or DLQ directory that is un-accounted for and not empty.

I am not in favor of this. I think users should be able to copy around PQ content for backup or other reasons. A PQ dir which is not referenced by a pipeline is ok I believe.

I would rather suggest that we instead look at the situation of starting LS when a PQ dir does not exist.

Generally speaking I am not too keen on preventing LS to start for non critical conditions, which could be very problematic for prod environments.

[jsvd]

Not sure I understand the original concern of having multiple pipelines named "main" if per @jsvd we already "prevent LS from starting if 2 pipelines have the same name"?

if you are collecting monitoring data from multiple hosts running logstash, by default all of them will publish the main pipeline id, making vizualisations trickier

[colinsurprenant]

Ah ok, the case of multiple hosts.
In this case I believe that using a generated unique ID will probably make things more complicated.
Why not just prepend the host name, or give the option to prepend the host name if it is confusing?
Or wait to get feedback from users and see how they react to this problem if is ever becomes a problem?

[edited] -> A user seeing multiple "main" or "default" pipeline names, if not differentiated by a host name, will probably decide to go ahead a rename these pipelines to a more descriptive name, no?

[talevy]

in general, I agree: default is probably better than main.

I also agree with Joao's point on just using the id-hash as the name if we expect to have
many Logstash installations with multiple pipelines... then the hash signifies just another pipeline... happens to be the first and only (in the beginning).

So, if we expect and encourage people to run multiple pipelines concurrently... then I'd go the hash-name route. No reason to bias towards naming and nurturing the first default one.

[colinsurprenant]

@jsvd @talevy 2 questions so I understand, maybe I am missing something:

• how is it better to have multiple pipelines with hash as a name instead of "default" for differentiating these pipelines in the viewer?
• if the pipelines name hash is generated from the config content, what happen with the PQ dir when the config changes?