-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Description
When users delete a pipeline and recreate the same pipeline with the same pipeline ID and config string, Logstash is unable to pick up again because the hash value is the same and the hash value does not consider metadata.
{
"pipeline": "input { generator { count => 100000 }} output { stdout { codec => \"rubydebug\"}} ",
"last_modified": "2020-10-01T15:42:30.229Z",
"pipeline_metadata": { "version": "1"},
"username": "log.stash",
}
If users delete the above pipeline and recreate it with the same config string with a different metadata {"version": "2"}, Logstash won't recreate the pipeline because the hash function only considers "input { generator { count => 100 }} output { stdout { codec => \"rubydebug\"}} " which is the same as before.
The proposal is to change the hash function to take the metadata into account. Logstash will be able to recompile the pipeline.
Version: 8.0.0-SNAPSHOT
Operating System: Mac
Config File (if you have sensitive info, please remove it): xpack.management.pipeline.id: ["pipeline_id"]
Steps to Reproduce:
-
enable xpack
-
config xpack.management.pipeline.id with a valid pipeline id
-
create a pipeline in elasticsearch version 7.10+
curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
"pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
"last_modified": "2020-10-01T15:42:30.229Z",
"pipeline_metadata": { "version": "1", "type": "logstash_pipeline"},
"username": "log.stash",
"pipeline_settings": {"pipeline.batch.delay": "50"}
}' -
delete the same pipeline in elasticsearch
curl -X DELETE "localhost:9200/_logstash/pipeline/heartbeat" -
recreate the pipeline with the same content but "version" update to "2"
curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
"pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
"last_modified": "2020-10-01T15:42:30.229Z",
"pipeline_metadata": { "version": "2", "type": "logstash_pipeline"},
"username": "log.stash",
"pipeline_settings": {"pipeline.batch.delay": "50"}
}' -
Logstash unable to output count
Action item:
- Logstash hash function include metadata fix delete and recreate pipeline with same pipeline ID and config string #12389
- Kibana update version number on save (upstream issue: pipeline management hard code version in metadata kibana#82045) (PR: Fix pipeline management metadata versioning kibana#82255)
- delete pipeline in registry delete pipeline in registry #12414