In the Quorum Besu Plugins repository, we have a well-defined policy regarding bug categorization.
Our policy is inspired by the Besu Defect Prioritization Policy & Bug Triage Process :
When deciding how to respond to a bug, consider the two variables of probability and severity, and apply them to a risk matrix to determine the priority and corresponding action.
How often actual customers are likely to encounter the issue during their use of the product?
A feature rarely used, with a high likelihood of failure, may have a lower overall probability than a popular feature with a low likelihood of failure (due to a high number of customers affected).
|
Frequent |
Will occur several times in the lifetime of the product. |
|---|---|
|
Probable |
Likely to occur often in the lifetime of the product. |
|
Occasional |
Likely to occur some time in the lifetime of the product. |
|
Remote |
Unlikely but possible to occur in the lifetime of the product. |
|
Improbable |
So unlikely, it can be assumed occurrence may not be experienced. |
How bad is the problem when it is encountered?
An issue that causes data loss/corruption is automatically classed as Catastrophic.
|
Catastrophic |
Detrimental risk for the product as a whole. |
|---|---|
|
Critical |
Jeopardises feature(s) of the product, not ruining the product entirely. |
|
Moderate |
Jeopardises aspects of a feature, not ruining the feature entirely. |
|
Marginal |
Causes an inconvenience when using a feature of the product. |
|
Insignificant |
No significant threat posed and can be left unmediated. |
| Catastrophic | Critical | Moderate | Marginal | Insignificant | |
|---|---|---|---|---|---|
| Frequent | Very High | High | High | Medium | Very Low |
| Probable | High | High | Medium | Medium | Very Low |
| Occasional | High | Medium | Medium | Low | Very Low |
| Remote | Medium | Medium | Low | Low | Very Low |
| Improbable | Medium | Low | Low | Very Low | Very Low |
In a nutshell, our bugs are categorized into 5 priorities:
| Priority | Examples |
|---|---|
| Very High (P1) |
|
| High (P2) |
|
| Medium (P3) |
|
| Low (P4) |
|
| Very Low (P5) |
|
All software has bugs. And sometimes we don't find them. This is something that we need to live with. However, when we do find bugs we need to do our best to identify the potential impact of that bug in our system.
A proper triaged bug helps up to prioritize and act on bugs in a timely manner. Also, by analysing the severity and impact of our bugs, we can create/update processes to ensure we are better prepared for them in the future.
All of our bugs are reported on GitHub issues. They are identified by the BUG label. We can make use of GitHub labels to categorize our bugs. We have 5 labels, P1, P2, ..., P5 to categorize a bug according to our policy shown above.
- Identify that a bug doesn't have a priority (no Px label).
- Use the bug categorization policy to determine the final priority of the bug.
- Based on your assessment, add the corresponding label to the bug.
- (optional) Add a comment to the issue explaining the rationale (justifying the severity and probability analysis).
We try to be deterministic on our assessments, and most of the time it will involve some discussion. However, if you feel like the assessment on a bug priority is wrong, you can always add a comment on the issue and start a discussion.
You can. However, if another person was responsible for the initial assessment it is expected that you will contact that person to share your thoughts before changing it.
It is a combination of being polite and reasonable. :)
That's ok! Deeper understanding of the impact of a bug will come from the people that work closer to the affected area of the code. Leave it to them!