1st time ux PR comment and UI user awareness (missing base report)

[codecovdesign]

Problem to solve

Context: when users sign up and integrate their repo to Codecov the UX can make the user feel there are errors even when they did everything right and the system is working as intended.

• The first PR comment makes it seem like there is a user error and links to an page in documentation titled “Error“
• The in app experience also displays something familiar to an error citing a “missing base report“ and no data
• The repo page doesn’t default to a branch selection and has copy that says “there was a problem getting repo contents from your provider“

The above UX/UI communicates errors to the user, when the truth is actually come the next PR the integration will be working properly. Alas, this experience can leave users:

• Frustrated at their first impression and abandon the tool
• There is no help for learnability of how the tool works
• It may appears either the tool and/or configuration is broken

current ux

Solution ideation

• Optimize for the workflow targeting the main branch in configuration (PR vs commit)
• Welcome the user to Codecov in the messaging (confirming/reinforcing that the configuration was successful)
• Educate user about how the data is surfaced generally (comparison of base to head concept)

ux: welcoming user in PR comment + in-app

code291.mov

Tasks

Beta Give feedback

1. Api returns whether a PR is the 1st pull for a repo #388
   +0
   
2. Worker checks whether a PR is the 1st pull for a repo #389
   +0
   
3. Add first commit commit message #403
   +0
   
4. Defaulting to file explorer tab #404
   +0
   
5. Add 1st PR type to comparison union #535
   +0
   

[codecovdesign]

from jira issue:

feedback from user related to this issue: Feedback about the new pull request comment · Issue codecov/engineering-team#215 · codecov/feedback

#15549 (https://codecov.freshdesk.com/a/tickets/15549) - Unrelated files in impacted files list

related write up: https://codecovio.atlassian.net/browse/CODE-2883

[codecovdesign]

• scenario on repo page when branch is not selected
  • why might this happen? the user needs to select a branch, but codecov only knows branch codecov is integrate with. ideation: maybe we could default to latest one for initial selection.
• re: the "how codecov works"
  • this is good, but for those not familiar with tests and running coverage they still may be confused. An example would be helpful of pytest with a coverage file (commands we might be able to show)
  • we could detect the language and coordinate example to present 1) run with test coverage, 2) download codecov CLI to upload (could happen live in front)
  • data of language usage across repos:

count	language
72,363
43,753	python
21,823	javascript
19,475	go
14,293	java
9,305	php
6,562	r
3,390	ruby
2,929	kotlin
2,897	dart
2,114	swift
1,764	c
1,700	scala
512	shell
295	perl
268	clojure
241	fortran
230	objective-c
172	d
99	groovy
3	xtend

• how do we identify the first time commit/PR on the pr comment and app
  • detect if it's the 1) 1st pull in our database + 2) missing base report
  • regarding the copy: the base report (the git graphic might help understand the comparison). the reason we could do the upload ahead, leveraging the CLI.
  • if it's the first pull let's show head report (then short copy about the missing base report )
  • should be possible to have different copy on 1st pull vs general error of missing base report

[katia-sentry]

Investigation for surfacing first commit: #38

[codecovdesign]

how to identify the first commit/PR to display banner in app?

• check if it's only one PR and if it is open/merged. if merged, remove banner...

how to identify the first PR

• if it's the first pull in our DB, we show welcome message
• empty state, could leverage graphql resolver which indicates if it's the first pull
• show file explorer only

design ToDo

• empty states for files changed
• follow up steps for discovery / potential later iterations:

we could detect the language and coordinate example to present 1) run with test coverage, 2) download codecov CLI to upload (could happen live in front)

depending on the language, or selectable set of tabs, we could show how the CLI could upload the coverage to Codecov. Outline that Codecov doesn't run test for you, but how to run coverage reports.