Reject message on failure

[Recodify]

If a message received by the plugin fails at the output stage (for example, due to an indexing issue in ES), is it possible to have the message rejected so that it is dead-lettered in RabbitMq?

What I'm currently seeing is that a message that fails to be indexed to ES is still ACKed and therefore is lost forever.

• Version: logstash 6.8.2
• Operating System: ubuntu 18.04
• Config File:

input {
    rabbitmq {
      host => "blah.eu-west-2.compute.amazonaws.com"
      queue => "reading-elastic"
      durable => true
      arguments => {
          "x-dead-letter-exchange" => "reading-retry"
          "x-dead-letter-routing-key" => "elastic"
      }
   }
}
filter{
    mutate {
        add_field => {
            "deviceId" => "%{[payload][deviceId]}"
            "readingDate" => "%{[payload][readingDate]}"
            "value" => "%{[payload][value]}"
            "readingTypeId" => "%{[payload][readingTypeId]}"
            "collectionDate" => "%{[payload][collectionDate]}"
            "propertyReference" => "%{[payload][propertyReference]}"
            "propertyId" => "%{[payload][propertyId]}"
            "deviceSerialNumber" => "%{[payload][deviceSerialNumber]}"
            "manufacturerReference" => "%{[payload][manufacturerReference]}"
            "modelReference" => "%{[payload][modelReference]}"
        }

        remove_field => [ "[payload]", "[queuedAt]" ]
    }
    mutate {
        convert => { "value" => "float" }
    }
    date {
        match => [ "readingDate", "UNIX_MS" ]
    }
    date {
        match => [ "collectionDate", "UNIX_MS" ]
        target => "collectionDate"
    }
    date {
        match => [ "readingDate", "UNIX_MS" ]
        target => "readingDate"
    }
}
output {
  elasticsearch { hosts => ["https://blah.eu-west-2.es.amazonaws.com:443"]
                  index => "readings-%{+YYYY.MM.dd}"
  }
}

[ThiagoBarradas]

I have the same problem as you. I tried with many ways, but I have not success.

Today I opened a topic in elastic discuss, waiting for some help.
https://discuss.elastic.co/t/logstash-rabbitmq-input-send-ack-after-event-processed/203709

If it will not possible, I will try implement it and send a pull request, although I not a expert with ruby.

Thx.

[yaauie]

There have been efforts in the past to add end-to-end ACKs to Logstash (which the above request would require; elastic/logstash#8514), but these efforts have largely stalled out due to complexity and resulting poor performance.

Without end-to-end acknowledgement in the Logstash pipeline, the Elasticsearch Output Plugin supports the Dead Letter Queue, which persists failed events to disk along-side failure metadata, and the dead_letter_queue Input Plugin can be used in a separate pipeline to hand these events back over to another system or queue for later processing.