IA1: Testing & iteration

[nadonomy]

We'd like to test & research the Phase I IA experiments in chorus, to learn, iterate and gain confidence in changes to ship to production. We'll use the following methods:

1. In-app feedback (@kittykat )

• Monitor in-app feedback
• Triage, looking for common themes
• Outreach to set up interviews with users who consented
• In-app feedback project

2. Moderated tests, internal (@niquewoodhouse )

• Put out a call to arms to folk on the core team to help with moderated user tests
• Do interviews, for initial findings and to help refine research methods
• Application form
• Responses

3. Community tests (@niquewoodhouse with support from @kittykat )

• Arrange community testing sessions to gather wider feedback
• 15th Dec results are here
  We should translate findings into separate issues, or consider tracking small snags in the final build issue.

4. Organic feedback via community (@kittykat )

• Once we have confidence in the above, broadcast to wider organic channels like Matrix rooms, TWIM, Twitter, etc

5. Unmoderated tests, external (@niquewoodhouse )

• Generate ad hoc builds with all experiments enabled by default
• Engage in paid user testing with relevant scripts

6. R&D PostHog (@kittykat )

• Once PostHog is merged, R&D how we can use it to learn more about IA efforts
• Access which metrics we want to collect for web and iOS
• Instrument metrics for web for the IA project #20613

Note: While this issue is public, most of the links in this issue are private and will not be made public, to preserve folks privacy.

---

Small iterations, based on testing

Some of the above testing has led us to identify some iterations to the experiments that will improve them for further testing.

• Amend + context menu in space header #20601
• Amend drop down menu in Space header #20606
• Amend appearance icon in space preferences #20608
• Reset highlighted result to top result when backspacing through search #20612
• Amend sidebar icon in settings #20615

[nadonomy]

Note to self: Grab conclusions from https://docs.google.com/document/d/1xGfZ0z6yXKAO-1vQTk8TxItZlCKhRvRlhsIc-a5KHyA/edit Done.

[kittykat]

Community testing artefacts are here - permission to access is needed on a case by case basis

[nadonomy]

@kittykat thx. I just updated the issue with a summary of work and included this in there.

On (1) In-app feedback, we're currently experimenting with using a GitHub project board so we can maintain context directly in GitHub. If you have any bright ideas around automation/adding labels based on inputs (e.g. if a user consents to be contacted) they would be greatly received!

[kittykat]

We are done with testing and processing results for IA1, further testing will be part of IA2.

Metrics have been split out into their own epic outside of IA, so closing this issue.