WIP: audit and testing of bundle analysis product v0-1.5

[codecovdesign]

Problem to solve

As we release the MVP of bundle analysis, we'd like to better understand the usability of configuration, user perception, and identify areas of improvements.

Configuration Experience:

• How intuitive was the configuration process?

  • Split into two parts 1) Codecov UI, 2) integration into the org workflow (identify edge cases / external challenges)
  • SDK portion related to configuring repo bundle(s) - would be helpful to get feedback about that ()

• What challenges, if any, did user face during the configuration?

Data Interpretation:

• How clear and actionable did user find the data in commit/PR comments?
• What additional information would user find helpful in PR comments?

Repo-Level Insights:

• Is user interested in broader repo-level data? Why or why not?
• What specific repo-level insights would be most valuable?
• What do they want to see?
  • Specific around controls: gates, thresholds, checks - what controls are desired

Participant Criteria

• Target Audience: Web Developers with experience and familiarity with JavaScript bundling tools (e.g., Webpack, Rollup).

test scipt

User Test Script for Bundle Configuration Tool

Introduction

Greet the Participant:

• "Hello [Participant's Name], thank you for joining us today. We're excited to get your input on our new bundle analysis tool."

Consent to Record:

• "Before we start, we'd like to record this session for analysis and improvement purposes. Do you consent to us recording this session?"

Explain the Test:

• "We'll be asking you to complete a task using the tool, and we'd like you to share your screen and walk us through your process. Your insights are invaluable to us."

Intro Questions:

1. "Before we begin, could you tell us a bit about how you currently manage your bundle configurations?"
2. "What challenges have you faced in this area?"
3. "Is there anything in particular you wish you had to make this easier?"

Task Instructions

Task Introduction:

• "Imagine you've just heard that Sentry/Codecov has launched a new tool for bundle analysis. You're interested in configuring it for your organization's repository to see how it can improve your workflow."

Specific Task:

• "Your organization is 'turing-corp', and you need to configure the tool for your repository. Let's pretend the repository name is '[Insert Repo Name]'. How would you go about setting this up?"

During the Task:

• Encourage the participant to think aloud as they navigate the configuration process.
• Take notes on any difficulties they encounter, any feedback they offer, and their overall user experience.

Wrap-Up:

• Thank them for their participation and valuable feedback.
• Ask if they have any final thoughts or questions about the tool.
• Inform them about the next steps and how their feedback will be used to improve the tool.

Next steps

• Create feedback issue specific to bundle comment
• User testing with 5 users to go through workflow
  • Create testing script
  • Recruit 5 users (volunteers: bart v, vu l, billy v, yagiz n, michael h, evan p if after mar 1)
    • P1: Vu 3/12 (vuluongj20)
    • P2: Billy V 3/12 (billyvg)
    • P3: Yagiz 3/13 (anonrig)
    • P4: Michael h 3/12 (``)
    • P5: Evan p 3/14 (evanpurkhiser)
    • P6: Bart V 3/12
  • Setup interviews
  • Identify key insights
  • Post finding and design updates to epic

Insights

Beta Give feedback

1. BUG: error shown upon bundle page reporting #1409
   bug +1
   
2. BUG: bundle dropdown shows it being empty #1430
   bug +1
   
3. BUG(?): org selector didn't appear upon business selection #1408
   bug +1
   
4. BUG: common unknown "there's been an error" #1410
   bug +1
   
5. BUG: after personal selection, org selection appears momentarily #1385
   Polish bug +2
   
6. ISSUE: user was configuring coverage, then bundle #1392
   +0
7. ISSUE: confusion about name input #1411
   in discovery +1
8. ISSUE: user expecting HTML index to appear (ordering confusion) #1421
   in discovery +1
9. ISSUE: installed organization not appearing immediately after app installation #1393
   Dev-Ready Polish +2
   
10. ISSUE: user expectation in 1st PR comment #1412
    P1: will do in discovery +2
11. ISSUE: GitHub app installation 1) banner update, and 2) CTA location update #1400
    Dev-Ready Polish +2
    
12. ISSUE: user wants to see change trends and delta #1413
    24 of 24
    in discovery +1
13. ISSUE: user received node version error #1424
    in discovery +1
14. INSIGHT: users navigated to docs upon prompt #1386
    in discovery +1
    
15. INSIGHT: most users used search to find org > repo (common first interaction) #1425
    in discovery +1
16. INSIGHT: interest in graphical representations (e.g., diff view graphs) of bundle changes, segments, and bloat #1423
    in discovery +1
17. DISCOVERY: sorting / tree of assets and filtering toward actionable tasks #1422
    in discovery +1
18. DISCOVERY: Identifying and Managing Unnecessary or Duplicated Dependencies #1420
    in discovery +1
19. BUG: bundle shown even though configuration was removed #1454
    bug +1
    
20. Show % change in bundle size on PR comment #1476
    +0
21. DISCOVERY: file filtering for more accurate analysis #1562
    9 of 9
    in discovery +1

[codecovdesign]

sync with @nicholas-codecov

• creating an example repo we'll use in tests (in-progress: https://github.com/Turing-Corp/bundle-example)
  • already configured with Vite, user will be tasked with sign up flow, finding bundles, and configuring per guide
  • should be 5-min config (assuming they found the guide and understood it), but 10+minutes at least on sign up etc
• we will be identifying JS/TS orgs to display bundle section
  • can we leverage this list for potential customers to introduce BA
• when will this be in production so we can start testing/onboarding
  • will be finished this sprint, will confirm end of week

[codecovdesign]

test script template

User Test Script for Bundle Configuration Tool

Introduction

Greet the Participant:

• "Hello [Participant's Name], thank you for joining us today. We're excited to get your input on our new bundle analysis tool."

Consent to Record:

• "Before we start, we'd like to record this session for analysis and improvement purposes. Do you consent to us recording this session?"

Explain the Test:

• "We'll be asking you to complete a task using the tool, and we'd like you to share your screen and walk us through your process. Your insights are invaluable to us."

Intro Questions:

1. "Before we begin, could you tell us a bit about how you currently manage your bundle configurations?"
2. "What challenges have you faced in this area?"
3. "Is there anything in particular you wish you had to make this easier?"

Task Instructions

Task Introduction:

• "Imagine you've just heard that Sentry/Codecov has launched a new tool for bundle analysis. You're interested in configuring it for your organization's repository to see how it can improve your workflow."

Specific Task:

• "Your organization is 'turing-corp', and you need to configure the tool for your repository. Let's pretend the repository name is '[Insert Repo Name]'. How would you go about setting this up?"

During the Task:

• Encourage the participant to think aloud as they navigate the configuration process.
• Take notes on any difficulties they encounter, any feedback they offer, and their overall user experience.

Wrap-Up:

• Thank them for their participation and valuable feedback.
• Ask if they have any final thoughts or questions about the tool.
• Inform them about the next steps and how their feedback will be used to improve the tool.

[codecovdesign]

Participant 0

TLDR; The participant's attempt to configure the bundle analysis tool was unsuccessful due to reliance on micro bundling, which wasn't compatible. Additionally, they encountered delays and confusion in the GitHub app recognizing the installation, impacting their ability to view their organization and repository promptly.

Intro Questions:

• How do you currently manage your bundle configurations?
  Participant mentions using plain JS and micro bundling for Project Wallace, with limited use of Rollup. They express uncertainty about whether to bundle modules and the effectiveness of their CSS tree implementation.

• What challenges have you faced in this area?
  The participant discusses inconsistencies between tests run on source files and the bundled JS file, leading to "green" tests that fail in the bundled state. They also mention not using Codecov for test validation.

• Is there anything in particular you wish you had to make this easier?
  The participant did not specify any tools or solutions.

Onboarding:

• Started with Sentry documentation related to bundles, tried installing the app but encountered issues with not seeing their organization in the dropdown after installation.
• Email confirmation from GitHub was received, but the organization wasn't immediately visible, leading to confusion.
• After reloading the page, they found the configuration button but mistakenly started setting up coverage thinking it would address bundle analysis as well.

Configuration Attempt:

• Uncertain whether micro bundling would work with the tool and stopped after considering the setup for rollup.
• Mentioned potentially abandoning the attempt but open to revisiting during future reconfigurations.

User Retro:

• Issues Highlighted:
  • Organization not appearing immediately after app installation.
  • Confusion between setting up coverage and bundle analysis.
  • Navigational challenges in the UI, especially with refreshing and finding the correct tab.
• Technical Environment:
  • Browser: Firefox 123
  • OS: macOS 14.3
  • Notable: Ad blocker active, LaunchDarkly running.

[codecovdesign]

Participant 4

TLDR: The developer successfully configured the bundle analysis tool but encountered slow loading times for the PR comment report. They appreciated the in-app summary data but expressed a desire for visibility into changes across commits.

Onboarding

• INSIGHT: user navigated to documentation via search for JS bundle analysis and started configuration using GitHub instructions outside of the UI, encountering some confusion around bundle name input.
  • Logged into Codecov, selected their organization through search, and configured using a token from repo settings (not from the bundles tab).
• ISSUE TO INVESTIGATE: encountered an error upon expecting a report in the bundles UI.
  • Assumed commit should progress in a pull request but struggled with no output.
• ISSUE: user expected Codecov to upload the bundle report automatically; realized processing takes time and comparisons require two commits.
  • Suggested a dedicated action for frontend bundle reports.
• ISSUE: PR didn’t display the expected "welcome to Codecov/bundle" confirmation; user had to push the PR to see coverage and bundle reports, leading to confusion.
• ISSUE: Noted absence of bundle selection; suggests auto-selecting if only one option is available.
  • Was looking forward to seeing a bundle report but was left hoping for more clarity.

Wrap-Up Questions and Feedback

• Thanked the participant for their feedback.
• Feedback:
  • Initial Observation: no immediate delta between branches visible in the app.
  • Desired Features: interest in a pie chart displaying percentages, preferably interactive, and enjoyed the concept of comparisons.
  • PR Comment: rxpressed a wish to see a table of assets for more detailed analysis.

[codecovdesign]

WIP: Participant 2

TLDR: The user successfully configured the repository, beginning with the documentation before moving to the app for the repo token. Their focus was on repo-level data, contributing many valuable suggestions for utilizing this data (see insights below).

Onboarding

• INSIGHT: Navigated to Codecov docs, then went to the app for the upload token. Wished the upload token was in the docs.
• Landed on the repo guide and identified the bundles tab, finding in-app guidance helpful.
• ISSUE: Encountered an error, realizing from the docs that a commit push was missing.
• User Summary: No branch context selected, suggesting the app could better resolve errors and that in-app docs should highlight the need for a git push commit.

USER RETRO of Repo > Bundle Summary

• Perception: Lists filenames and packages composing the bundle, ordered by size.
• Question Response: Confused by the ordering.
• INSIGHTS:
  • Adding a new package prompts looking for a smaller library alternative.
  • Identifying large components for dynamic loading (code splitting for lazy loading).
  • Interest in tracking hash sizes for optimizing CDN caching.
• ISSUE: Initial confusion selecting the bundle in the selector.

PR Comment

• The user did not create a PR, so they didn't see the report.

[codecovdesign]

WIP: Participant 1

TLDR The user successfully configured the bundle analysis tool, navigating through documentation and the app to obtain the necessary upload token. They experienced some confusion and errors within the UI but ultimately accessed the report. Insights on desired features and improvements are provided below.

Intro Questions (unrelated to BA feature, rather current experience)

• Bundle Configuration Management:
  Uses a TS config parser option. Managing the config file is challenging due to poor segmentation and large size, making it difficult to locate specific variables.

• Challenges:
  The main issues are the config file's complexity and finding the right variable due to its size and unstructured additions by developers.

• Desired Improvements:
  Interested in continuous integration for easier management of releases and PRs, with an emphasis on the importance of bundle size analysis.

Onboarding

• INSIGHT: the user aimed to select both options available but faced issues with the organization selector. (T and C page)
• ISSUE: encountered an unauthorized page error, causing confusion and the need to navigate to the activation page unexpectedly.
• Experienced confusion assuming that coverage needed to be configured, then bundle could be
• Experienced confusion over bundle project names and the necessity of adding a token, highlighting a need for better UI guidance on these steps.
• USER SUMMARY: Attempted local integration by cloning the repo and adding the plugin but missed adding the token, which was critical for successful upload.

User Insights & Issues

• CONFIGURATION & REPORTING:
  The configuration process was marked by UI confusion and errors, notably an "unknown error" banner that persisted across sessions.
• REPO LEVEL REPORTING:
  Expressed a desire for HTML file inclusion in bundle definitions and improved asset-level reporting.
• PR COMMENT/FEEDBACK:
  Noted the absence of PR comment reports due to merging to main and discussed the potential for enhanced reporting in PRs and releases.

Other suggestions for improvement

• UI CLARITY: highlighted the need for clearer instructions on bundle naming and merging steps 2 and 5 for token addition.
• ERROR HANDLING: suggested improvements for handling and displaying errors more informatively.
• REPORTING FEATURE: expressed interest in seeing comparison changes over time, managing package sizes, and understanding dependencies.

[codecovdesign]

Participant 3

TLDR: The participant discussed their current bundle configuration process, highlighting dependency and optimization challenges. They encountered issues and confusion during onboarding and configuration, particularly with the sequence of steps and understanding bundle names. Despite this they successfully configured the featured. Key insights include the desire for a tool that estimates optimization impacts and offers deeper dependency analysis.

Intro Questions

• Bundle Configuration Management:
  Uses Next.js bundle analyzer for personal projects, rspack.dev for Sentry, noting inefficiencies in frontend code delivery and compilation by Babel.

• Challenges:
  INSIGHT: encountered issues with duplicate packages and difficulty determining unnecessary plugin removal.

• Desired Improvements:
  INSIGHT: expressed a need for tools to estimate the impact of optimizations on dependencies and insights on removal impacts.

Onboarding

• INSIGHT: Directly referred to documentation.
• BUG: Encountered slow display issues with copy/paste functionality.
• Configuration Confusion: found step 3 ("commit your changes") confusing; suggested reordering steps for clarity.
• ISSUE: noted that step 4 could logically precede step 3.
• Moved to the app for the token, initially visiting the coverage page before switching to the bundle page.
• ISSUE: experienced confusion over the "bundle name," which appeared optional but was required in the terminal.

Configuration and Feedback

• ISSUE: encountered errors upon accessing the repo page and did not see HTML and SVG in the bundle report.
• INSIGHT: expressed interest in a diff view graph and a detailed breakdown of dependencies, including their impact on bundle size.
• INSIGHT: Highlighted a use case related to security, suggesting a comparison between local machine bundles and CI-uploaded bundles to identify optimization opportunities for assets not processed through CI.

User Insights

• INSIGHT: suggested improvements for identifying and managing unnecessary or duplicated dependencies.
• INSIGHT: noted the importance of displaying unprocessed files over processed ones for better clarity.
• INSIGHT: interested in a feature to show assets that could be optimized, especially those that bypass CI processing.

[codecovdesign]

Participant 5

TLDR: This participant successfully configured the bundle analysis tool, with some points highlighting a need for clearer documentation and improved UI clarity, especially regarding bundle names and step sequences. They expressed a desire for advanced features such as detailed dependency analysis and change tracking.

Participant Notes

Intro Questions

• Bundle Configuration Management:
  "We use webpack with an integrated configuration that heavily utilizes code splitting to identify how specific elements are utilized across various parts of the project."

• Challenges:
  INSIGHT: Noted challenges in tracking the usage of certain elements in multiple places.

• Desired Improvements:
  The lack of detail on individual chunks in previous tools was limiting.
  INSIGHTS:

  • Desire for detailed file change tracking and visual comparisons (mapping/graph/tree map).
  • Interest in insights on components' bundle placement and identification of duplications for performance optimization.
  • A tool that alerts when and where bundles increase in size.
  • Consideration for a rule system to manage dependency allowances.

Task Introduction

• The goal was to configure the new bundle analysis tool for Sentry's repository to enhance workflow efficiency.

Onboarding

• Configuration started locally after consulting quick start documentation.
• ISSUE: preference expressed for earlier step introduction.
• ISSUE: uncertainty about "bundle name" meaning.
• Node version issues highlighted a dependency check mechanism.
• INSIGHT: used search to find the repository, a common initial step.
• Challenges finding the organization in-app led to GitHub navigation for app re-authorization.
• Noted the repo guide's effectiveness over general documentation.

Configuration and Feedback

• Successful bundle configuration led to exploring the repo bundle's main page.
• INSIGHT: immediate comparison features were highly valued post-configuration.
• Exploration of versioning and bundle comparison through commit landing pages.
• INSIGHT: interest in a detailed reason for asset placement within dependency trees.
• ISSUE: confusion regarding HTML content sorting and categorization, especially for CSS files.

This participant's feedback illuminates the critical areas for tool enhancement, from onboarding to advanced analytical features, driving home the need for a user-centric approach in tool development and documentation.

[codecovdesign]

Report of Bundle Analysis Audit and Testing

TL;DR for All Participant Tests/Audits

📽️ See condensed video related to GitHub issues below (full)

• Configuration Success: 5 out of 6 participants successfully configured the bundle analysis tool. p0 was unable to proceed due to incompatibility with their repository bundle, excluding participant: 5 of 5 were able to user the feature correctly in-app 🎉
  • There was limited-to-no interaction with the the PR comment report as it did not appear in the audits (due to loading and/or known-issue WIP). Therefore feedback was more geared toward the config and repo display.
• Process Initiation: all participants began their configuration journey by googling to find documentation, reviewing it for guidance, and proceeding to the app to obtain the necessary token.
• Participants:
  - P0
  - P1
  - P2
  - P3
  - P4
  - P5

Onboarding Challenges:

• Participants encountered confusion regarding the order of operation steps and the purpose of the "bundle name" in the configuration input.
• UI elements, notably the persistent "unknown error" banner, and the default file/structure sorting mechanism presented usability issues, with a noted lack of options to sort/filter effectively.

Desired Features and Improvements:

• There is a strong interest in features that enable users to see changes and differences among components/assets, not just in PR comments but also from one release to another and trends over time.
• Participants expressed a desire for a more detailed breakdown of dependencies and their impact on bundle size. This reflects a broader theme of seeking ways the tool could facilitate actionable insights and optimizations; examples i) duplicate dependencies, ii) impact of removal, iii) proportion of assets to bundle, iv) system suggestions to improve.

Next Steps and Related Action Items:

Review incoming issues, issues from our v2 discovery, and the ones listed below from the audit. Then, identify which ones we'd like to groom and explore in upcoming design/engineering sprints.

• BUG: error shown upon bundle page reporting #1409
• BUG: bundle dropdown shows it being empty #1430
• BUG(?): org selector didn't appear upon business selection #1408
• BUG: common unknown "there's been an error" #1410
• BUG: after personal selection, org selection appears momentarily #1385
• ISSUE: user was configuring coverage, then bundle #1392
• ISSUE: confusion about name input #1411
• ISSUE: user expecting HTML index to appear (ordering confusion) #1421
• ISSUE: installed organization not appearing immediately after app installation #1393
• ISSUE: user expectation in 1st PR comment #1412
• ISSUE: GitHub app installation 1) banner update, and 2) CTA location update #1400
• ISSUE: user wants to see change trends and delta #1413
• ISSUE: user received node version error #1424
• INSIGHT: users navigated to docs upon prompt #1386
• INSIGHT: most users used search to find org > repo (common first interaction) #1425
• INSIGHT: interest in graphical representations (e.g., diff view graphs) of bundle changes, segments, and bloat #1423
• DISCOVERY: sorting / tree of assets and filtering toward actionable tasks #1422
• DISCOVERY: Identifying and Managing Unnecessary or Duplicated Dependencies #1420