test: add development runtime tests

[DSchau]

In order to test PRs like #10455 more effectively, it makes a fair amount of sense to also test the development runtime, e.g. gatsby develop.

What I want to test (and haven't been able to get quite working yet) are things like:

• Hot reloading
  • Class components
  • Function components
  • Pages
  • Template pages
  • File content, e.g. a Markdown file
  • (Hooks, when it's done)
• Linking between pages
• Creating a new page, and navigating to it
• 404 page functionality

And other core functionalities of develop that we want to ensure remain working. I'll keep iterating on this PR next week, but wanted to get it up for some 👀 sooner rather than later.

It could make sense to test this stuff as part of the production runtime, and just create separate test scripts, but I thought this was a little cleaner. Opinions more than welcome!

[DSchau]

This actually mostly works really well, which is pretty cool.

I've been able to validate the following functionality:

• Hooks work in development mode
• Hooks work in hot reload mode
  • This is actually kinda challenging to test; updating useState (e.g. useState(0) -> useState(100)) doesn't actually update, which is what we want, e.g. to keep state persistent
  • The approach I'm following (e.g. updating count + 1 to count + 100) works outside of Cypress but not in Cypress
• onRouteUpdate is called appropriately in development mode
• class components can be updated, state can change, etc. in hot reload mode
• New JS page component can be created and hot reloads
• New file (Markdown) can be created and hot reloads
  • Seems like this is failing for some reason...
• Page component can be hot reloaded
• Template component can be hot reloaded
• Linking between pages works
• 404 functionality works

[DSchau]

@pieh or @m-allanson want to take a look at this tomorrow? Feeling like it's pretty done/ready! 😍

[m-allanson]

Added some initial comments, but haven't had a chance to dig into this fully. Looking really nice though!

[m-allanson]

Running this locally it consistently fails on hot-reloading/arrow-functions.js:

 Running: hot-reloading/arrow-functions.js...                                            (5 of 13)

Warning: Cypress can only record videos when using the built in 'electron' browser.

You have set the browser to: 'chrome'

A video will not be recorded when using this browser.


  hot-reloading anonymous arrow functions
    ✓ displays placeholders on launch (395ms)
    1) upates on change


  1 passing (3s)
  1 failing

  1) hot-reloading anonymous arrow functions upates on change:
     CypressError: cy.exec('npm run update -- --file src/components/title.js --replacements "TITLE:The title"') failed because the command exited with a non-zero code.

Pass {failOnNonZeroExit: false} to ignore exit code failures.

Information about the failure:
Code: 127

Stderr:
/bin/bash: npm: command not found
      at Object.cypressErr (http://localhost:8000/__cypress/runner/cypress_runner.js:63065:11)
      at Object.throwErr (http://localhost:8000/__cypress/runner/cypress_runner.js:63030:18)
      at Object.throwErrByPath (http://localhost:8000/__cypress/runner/cypress_runner.js:63057:17)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:52545:23
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:124314:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:122332:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:122389:18)
      at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:122434:10)
      at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:122513:18)
      at Async._drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:119238:16)
      at Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:119248:10)
      at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:119122:14)




  (Results)

  ┌────────────────────────────────────────────────┐
  │ Tests:        2                                │
  │ Passing:      1                                │
  │ Failing:      1                                │
  │ Pending:      0                                │
  │ Skipped:      0                                │
  │ Screenshots:  1                                │
  │ Video:        false                            │
  │ Duration:     2 seconds                        │
  │ Spec Ran:     hot-reloading/arrow-functions.js │
  └────────────────────────────────────────────────┘


  (Screenshots)

  - /Users/mike/dev/gatsby/e2e-tests/development-runtime/cypress/screenshots/hot-reloading/arrow-functions.js/hot-reloading anonymous arrow functions -- upates on change (failed).png (1200x1054)

This is a weird failure as npm run update works fine in all the other tests 🤷‍♂️

[pieh]

I couldn't replicate that error @m-allanson when I was testing locally - so this is super weird to see

[pieh]

🌮