RCB: Update PR comment with fail reason in different RCB behaviors

[Adal3n3]

Problem to solve

Description

This task is a part of the work from RCB v2 #1020
It is unclear to customers why the check fails, leading to much confusion. Therefore, on the PR comment, we want to show the transparency:
“Your status [fail] because of [reason]” for both patch and project coverage users.

Steps to Reproduce / Current UX

1. Need a table to list all of the CI check failure scenarios for 1/ reasons based on the RCB behaviors 2/ without RCB which patch coverage only users.
2. Update the PR comment design with the new conditional check status messages.

Solution

1. Here's the aligned PR design for displaying all the failed reasons with action for the user.
   

2. Here's the list of CI failure scenarios and the UI messages. (Reviewed with Gio)

3. Here is the list of unexpected changes, we can't detect today:

   • Your project check has failed because the coverage report failed to upload.
   • Your project check has failed because you have a time-sensitive tests.
   • Your project check has failed because you are missing coverage reports.
   • Your project check has failed because you have encrypted variables may prevent some execution paths.
   • Your project check has failed because you have a different error handling paths.

4. Update the unexpected changes to the public document, if needed here

Additional Information

General Scenarios

Beta Give feedback

1. Failed check status 1 - No coverage information found on head #1595
   +0
   
2. Failed check status 2 - head_coverage is below your target threshold #1605
   +0
3. Failed check status 3 - no report for the base #1606
   +0
   
4. Failed check status 20 - [Patch coverage] No report found to compare against. #1625
   +0
   
5. Failed check status 21 - [Patch coverage] patch coverage hit target within configured threshold #1626
   +0
6. Failed check status 22 - Found unexpected coverage changes not visible in diff #1627
   +0

Removed Code Behavior - Adjust Base (Default Config)

Beta Give feedback

1. Failed check status 4 - [RCB adjust_base] project coverage in head is below acceptable threshold when compared to base #1607
   +0

Removed Code Behavior - Removal Only

Beta Give feedback

1. Failed check status 5 - [RCB removal_only] adding or removing tests #1608
   +0
2. Failed check status 6 - [RCB removal_only] have a different number of reports between HEAD and BASE #1609
   +0
   
3. Failed check status 7 - [RCB removal_only] you have a dependencies changed #1610
   +0
4. Failed check status 8 - [RCB removal_only] changes to a PR, missing report on the base or head commits #1611
   +0
5. Failed check status 9 - [RCB removal_only] not running all tests on every commit #1612
   +0
6. Failed check status 10 - [RCB removal_only] the BASE coverage is higher than the HEAD #1615
   +0
7. Failed check status 11 - [RCB removal_only] Lines were added in the diff #1616
   +0
8. Failed check status 12 - [RCB removal_only] diff is empty #1617
   +0

Removed Code Behavior - Fully Covered Patch

Beta Give feedback

1. Failed check status 13 - [RCB fully_covered_patch] the BASE coverage is higher than the HEAD #1618
   +0
2. Failed check status 14 - [RCB fully_covered_patch] adding or removing tests #1619
   +0
3. Failed check status 15 - [RCB fully_covered_patch] a different number of reports between HEAD and BASE #1620
   +0
4. Failed check status 16 - [RCB fully_covered_patch] dependencies changed resulting in a different execute plan #1621
   +0
5. Failed check status 17 - [RCB fully_covered_patch] changes to a PR, missing report on the base or head commits #1622
   +0
   
6. Failed check status 18 - [RCB fully_covered_patch] not running all tests on every commit #1623
   +0
7. Failed check status 19 - [RCB fully_covered_patch] patch coverage is less than 100% #1624
   +0

[giovanni-guidini]

Flowchart with the current possible statuses outcomes and reasons for them.
Most likely we will want to change some of those, particular on RCB to make it more clear why the RCB still failed the check.
( I really hope you can download the file and zoom in 😅 )

Based on this we can derive a table of messages we want to add to the PR comment and/or make changes to the current messages in the statuses.

[Adal3n3]

Another example: https://twitter.com/edthrn/status/1780476782090604625

[Adal3n3]

@trent-codecov @aj-codecov Can we prioritize the failed check messaging caused by upload because upload is a part of the onboarding/signup flow.

These issues are upload related:

1. Failed check status 20 - [Patch coverage] No report found to compare against. #1625 (assume we have more patch coverage users)
2. Failed check status 1 - No coverage information found on head #1595
3. Failed check status 3 - no report for the base #1606
4. Failed check status 6 - [RCB removal_only] have a different number of reports between HEAD and BASE #1609
5. Failed check status 8 - [RCB removal_only] changes to a PR, missing report on the base or head commits #1611
6. Failed check status 17 - [RCB fully_covered_patch] changes to a PR, missing report on the base or head commits #1622

[thomasrockhu-codecov]

Noting that the Q2 items have been completed here: #2094, removing from the Q2 milestone until work is scoped and prioritized

[Adal3n3]

Making a note here: as discussed during the meeting on 8/12, the next step is for the PM team group and prioritize for us. @rohan-at-sentry

[rohan-at-sentry]

Grouped the items by config type.

My guidance is to tackle General Scenarios first (as it has the broadest surface) and follow with adjust base (as it's the new default).

We can be opportunistic about adding support for other scenarios after that