You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description:
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct PeerState is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the consensus logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPC dump_consensus_state.
Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one PeerState struct. If dump_consensus_state is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different PeerState structs) and the chain will progress again. Then, the same process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first PeerState struct will deadlock, and not the others (RPC dump_consensus_state accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).
This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC dump_consensus_state), do not expose dump_consensus_state RPC endpoint to the public internet (e.g., via rules in one's nginx setup).
Cross references:
No existing reports found with this module or alias.
See doc/triage.md for instructions on how to triage this report.
modules:
- module: github.com/cometbft/cometbft
vulnerable_at: 0.37.2
packages:
- package: cometbft
description: |-
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state
transition machine and replicates it on many machines. An internal modification
made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized
to JSON introduced a deadlock when new function MarshallJSON is called. This
function can be called from two places. The first is via logs, setting the
`consensus` logging module to "debug" level (should not happen in production),
and setting the log output format to JSON. The second is via RPC
`dump_consensus_state`.
Case 1, which should not be hit in production, will eventually hit the deadlock
in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be
deadlocked, together with the thread(s) dealing with the RPC request(s). This
means that only one of the channels of communication to the node's peers will be
blocked. Eventually the peer will timeout and excluded from the list (typically
after 2 minutes). The goroutines involved in the deadlock will not be garbage
collected, but they will not interfere with the system after the peer is
excluded.
The theoretical worst case for case 2, is a network with only two validator
nodes. In this case, each of the nodes only has one `PeerState` struct. If
`dump_consensus_state` is called in either node (or both), the chain will halt
until the peer connections time out, after which the nodes will reconnect (with
different `PeerState` structs) and the chain will progress again. Then, the same
process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer
struct each node maintains, the possibility of reproducing the perturbation
visible with two nodes decreases. Only the first `PeerState` struct will
deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for
loop, so the deadlock at the first iteration causes the rest of the iterations
of that "for" loop to never be reached).
This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are
available. For case 1 (hitting the deadlock via logs), either don't set the log
output to "json", leave at "plain", or don't set the consensus logging module to
"debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC
`dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to
the public internet (e.g., via rules in one's nginx setup).
cves:
- CVE-2023-34450
references:
- advisory: https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr
- fix: https://github.com/cometbft/cometbft/pull/524
- fix: https://github.com/cometbft/cometbft/pull/863
- fix: https://github.com/cometbft/cometbft/pull/865
The text was updated successfully, but these errors were encountered:
CVE-2023-34450 references github.com/cometbft/cometbft, which may be a Go module.
Description:
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct
PeerState
is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting theconsensus
logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPCdump_consensus_state
.Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one
PeerState
struct. Ifdump_consensus_state
is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with differentPeerState
structs) and the chain will progress again. Then, the same process can be repeated.As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first
PeerState
struct will deadlock, and not the others (RPCdump_consensus_state
accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC
dump_consensus_state
), do not exposedump_consensus_state
RPC endpoint to the public internet (e.g., via rules in one's nginx setup).References:
Cross references:
No existing reports found with this module or alias.
See doc/triage.md for instructions on how to triage this report.
The text was updated successfully, but these errors were encountered: