Skip to content

[Meta] support xpack delete and recreate pipeline with same pipeline ID and config string #12387

@kaisecheng

Description

@kaisecheng

When users delete a pipeline and recreate the same pipeline with the same pipeline ID and config string, Logstash is unable to pick up again because the hash value is the same and the hash value does not consider metadata.

{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => \"rubydebug\"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1"},
    "username": "log.stash",
}

If users delete the above pipeline and recreate it with the same config string with a different metadata {"version": "2"}, Logstash won't recreate the pipeline because the hash function only considers "input { generator { count => 100 }} output { stdout { codec => \"rubydebug\"}} " which is the same as before.
The proposal is to change the hash function to take the metadata into account. Logstash will be able to recompile the pipeline.

Version: 8.0.0-SNAPSHOT
Operating System: Mac
Config File (if you have sensitive info, please remove it): xpack.management.pipeline.id: ["pipeline_id"]
Steps to Reproduce:

  1. enable xpack

  2. config xpack.management.pipeline.id with a valid pipeline id

  3. create a pipeline in elasticsearch version 7.10+
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  4. delete the same pipeline in elasticsearch
    curl -X DELETE "localhost:9200/_logstash/pipeline/heartbeat"

  5. recreate the pipeline with the same content but "version" update to "2"
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "2", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  6. Logstash unable to output count

Action item:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions