[FEAT] E2E testing for contributors

[tianhaoz95]

⚠️ So far the conclusion is that documenting the more complicated process for end-to-end testing is of higher priority given the complexity of implementing an automated deploy system.

Objective

To build an infra to do E2E testing which is easier for contributors to use.

Background

E2E testing requires creating a dummy GitHub app and configuring the local environment which take a non-trivial amount of work. It will be great to simplify this process for contributors.

The challenges are:

• Deploying the staging app requires the webhook secret and client authentication token which is sensitive information that should not be accessed in forks.
  • Use labels or comments that the owner can control to deploy PR to the staging app
• As there is only one staging app and one testing repository, there can only be one tester at a certain period of time.
  • Deploy one app per PR
  • Use a calendar to coordinate a time for testing
  • Use a queue to coordinate a time for testing

Design

TODO(@tianhaoz95): add some practical design when there is more time to come back to this topic.

Roadmap

• Document the complicated way to do testing: create a dummy app and dummy repository