Wallet E2E Test API

[ivelin]

The Web3 Frontend Brittleness Problem

Many of today's web3 apps feel brittle, partially because there is no robust e2e test framework API officially supported by MetaMask.

There are frequent ongoing UI flow breaks and misplacement or misrepresentation of user's crypto funds due to subpar UX. This is true even for top web3 apps like Uniswap, Axie and Sandbox. As users navigate between different web3 apps and interact with multiple blockchain networks and wallets, these web3 apps often get out of sync with the state of the current wallet and as a result, the user ends up looking at unexpected content.

Given web3's promise to better protect user's identity, privacy and provide more control over user's own assets, it is imperative to offer superior, trustworthy UX. That in turn requires improving on test automation practices and infrastructure.

Current Workarounds

Test frameworks like dappeteer and synpress are trying to bridge the gap, but they feel brittle as they are constantly chasing MetaMask's changing front end code. This test framework brittleness frustrates web3 developers who are used to sophisticated web2 tooling. Writing web3 e2e tests that frequently break due to underlying base layer problems, results in postponing these test automation initiatives, which then results in brittle web3 frontend apps and subpar web3 UX.

The following snippet is an example of web3 e2e test frameworks "screen scraping" metamask code.

const firstTimeFlowFormPage = '.first-time-flow__form';
const secretWordsInput = `${firstTimeFlowFormPage} .first-time-flow__seedphrase input`;
const passwordInput = `${firstTimeFlowFormPage} #password`;
const confirmPasswordInput = `${firstTimeFlowFormPage} #confirm-password`;
const termsCheckbox = `${firstTimeFlowFormPage} .first-time-flow__terms`;
const importButton = `${firstTimeFlowFormPage} .first-time-flow__button`;
const newPasswordInput = `${firstTimeFlowFormPage} #create-password`;
const newSignupCheckbox = `${firstTimeFlowFormPage} .first-time-flow__checkbox`;

and another example:

  const addressInput = await page.waitForSelector('#custom-address');
  addressInput.type(tokenAddress);

  await page.waitForTimeout(4000);

  const nextButton = await page.waitForSelector(`button[data-testid='page-container-footer-next']:not([disabled])`);
  await nextButton.click();

  const footerButtons = await page.$$('footer > button');

While web3 app test developers are looking for reliable API that looks like this:

  const browser = await dappeteer.launch(puppeteer, { metamaskVersion: 'v10.1.1' });
  const metamask = await dappeteer.setupMetamask(browser);

  // you can change the network if you want
  await metamask.switchNetwork('ropsten');

  // you can import a token
  await metamask.addToken('0x4f96fe3b7a6cf9725f59d353f723c1bdb64ca6aa');

  // go to a dapp and do something that prompts MetaMask to confirm a transaction
  const page = await browser.newPage();
  await page.goto('http://my-dapp.com');
  const payButton = await page.$('#pay-with-eth');
  await payButton.click();

  // 🏌
  await metamask.confirmTransaction();

Proposed Solution

MetaMask is the industry leading web3 wallet. It is in a position to establish a standardized API for e2e web3 app testings that improves web3 app UX and sets a high bar for the growing number of new L2&L1 specialized wallets.

This MetaMask test-dapp repo could be the starting point for standardizing web3 e2e testing by documenting and providing test-spec examples for the interaction between its page components and the metamask wallet in various browsers through a cross browser e2e framework such as playwright.

Thank you and Happy Holidays! 🎄 🎉

[danfinlay]

Hi Ivelin, thanks for the detailed write up. We’ll be giving more bandwidth to improving developer experience this year, and this issue could make a very nice target. Even if we just start to integrate some testable hooks into our own tests so we can guarantee they won’t break, I bet that would help a lot.

[ivelin]

Hi Ivelin, thanks for the detailed write up. We’ll be giving more bandwidth to improving developer experience this year, and this issue could make a very nice target. Even if we just start to integrate some testable hooks into our own tests so we can guarantee they won’t break, I bet that would help a lot.

Happy New Year, @danfinlay !

Great to know that you've been already thinking about this.

Even if we just start to integrate some testable hooks into our own tests so we can guarantee they won’t break, I bet that would help a lot.

Yes, an e2e test suite included in the MetaMask/test-dapp repo would be a great reference point for apps.
Maybe pull in dappetteer or synpress in the CI flow and help keep it stable. Ideally the e2e test API would support a framework like playwright to allow testing across all major browsers, not just thechromium based ones.

BTW, I got some positive response from WalletConnect (here) and Brave (here). Hopefully we work together towards smoother web3 app UX in 2022!

[agcty]

+1 this is the missing feature that would make metamask op

[tmm]

We would use this at mirror.xyz

[kdenhartog]

Anyone thought about setting up a gitcoin grant or gitcoin bounty for this? Seems like we could get quite a few wallets to help fund this work given how many would be impacted by this

[dayana0425]

I agree, most dapps are subpar, and, naturally, users don't trust the information they see on the frontend. They're forced to dig into blockchain transactions to ensure everything checks out. This is a key reason why people have reservations about dapps and sometimes label them as scams or too hard to use.

On that note, a robust end-to-end (e2e) testing framework could significantly alleviate these concerns. By catching errors and inconsistencies before they reach the end user, we could improve the reliability and overall user experience of these dapps. It's not just about improving the software; it's about regaining user trust in this promising technology. I believe comprehensive e2e testing is a crucial step towards it.