Dead lettering a message can crash when upgrading from 3.13.x to 4.0.x #12933
Comments
mergify bot
pushed a commit
that referenced
this issue
Dec 13, 2024
Fixes #12933 The assumption that `x-last-death-*` annotations must have been set whenever the `deaths` annotation is set was wrong. Reproducation steps, Option 1: 1. In v3.13.7, dead letter a message from Q1 to Q2 (both can be classic queues). 2. Re-publish the message including its x-death header from Q2 back to Q1. (RabbitMQ 3.13.7 will interpret this x-death header and set the deaths annotation.) 3. Upgrade to v4.0.4 4. Dead letter the message from Q1 to Q2 will cause the following crash: ``` crasher: initial call: rabbit_amqqueue_process:init/1 pid: <0.577.0> registered_name: [] exception exit: {{badkey,<<"x-last-death-exchange">>}, [{mc,record_death,4,[{file,"mc.erl"},{line,410}]}, {rabbit_dead_letter,publish,5, [{file,"rabbit_dead_letter.erl"},{line,38}]}, {rabbit_amqqueue_process,'-dead_letter_msgs/4-fun-0-', 7, [{file,"rabbit_amqqueue_process.erl"},{line,1060}]}, {rabbit_variable_queue,'-ackfold/4-fun-0-',3, [{file,"rabbit_variable_queue.erl"},{line,655}]}, {lists,foldl,3,[{file,"lists.erl"},{line,2146}]}, {rabbit_variable_queue,ackfold,4, [{file,"rabbit_variable_queue.erl"},{line,652}]}, {rabbit_priority_queue,ackfold,4, [{file,"rabbit_priority_queue.erl"},{line,309}]}, {rabbit_amqqueue_process, '-dead_letter_rejected_msgs/3-fun-0-',5, [{file,"rabbit_amqqueue_process.erl"}, {line,1038}]}]} ``` Reproduction steps, Option 2: 1. Run a 4.0.4 / 3.13.7 mixed version cluster where both queues Q1 and Q2 are hosted on the 4.0.4 node. 2. Send a message to Q1 which dead letters to Q2. 3. Re-publish a message with the x-death AMQP 0.9.1 header from Q2 to Q1. However, this time make sure to publish to the 3.13.7 node which forwards this message to Q1 on the 4.0.4 node. 4. Subsequently dead lettering this message from Q1 to Q2 (happening on the 4.0.4 node) will also cause the crash. The modified test case in this commit was able to repro this crash via Option 2 in the mixed version cluster tests on the `v4.0.x` branch. (cherry picked from commit b6027ec)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue for the discussion in #12932.
The text was updated successfully, but these errors were encountered: