monorepo: overhaul lerna/yarn/repo tools

[joshgummersall]

• Clean up lerna.json
• Move to yarn workspaces
• Move to yarn for CI
• Add botbuilder-repo-utils with update versions script
• Introduce preview and deprecated package.json metadata for more consistent and simple build & release

The intent of this set of changes is to simplify as much as possible.

The updateVersions.ts script is the brains of the release versioning step. This script now encodes and formalizes a conventional use of two separate Azure DevOps variables and a pretty gnarly sed script.

Yarn is a lot more performant than npm install - CI time is cut almost in half by this change alone. Yarn also produces a single lock file (yarn.lock) that ensures that all developers have a consistent set of installed packages. It also serves as a wonderful input to a cache key calculation.

The CI steps are extracted into shared templates and used more consistently in the various CI jobs.

Note: I'd like to drop the tools sub-tree entirely but it's currently used for a single botframework-connector test.

[joshgummersall]

I plan to handle moving the functional/streaming test scenarios to yarn as well. Eventually 😄

[joshgummersall]

Still need to add/update docs that reference npm instead of yarn.

[stevengum]

@johnataylor, @munozemilio, @EricDahlvang, @Zerryth, @mdrichardson FYI

[mdrichardson]

@joshgummersall Regarding:

Note: I'd like to drop the tools sub-tree entirely but it's currently used for a single botframework-connector test.

Is that the test using nock-helper? If so, I've got a related issue for that. Sounds like you probably have some ideas how to clean that up. I'd be happy to take that on if we can nail down some design guidelines for it.

[joshgummersall]

@mdrichardson let's have that discussion over here: microsoft/botframework-sdk#5977

[joshgummersall]

Fixes #1383

[stevengum]

Approving latest changes! (9de05af)

[Zerryth]

I plan to handle moving the functional/streaming test scenarios to yarn as well. Eventually 😄

Sounds good to me! @joshgummersall

---

Also regarding the E2E streaming tests, to keep in mind with your yarn overhaul:

Background

The React app's base was created from from create-react-app command, which defaults to yarn, which is what was originally what we had in the az. pipeline, however--and this was just lack of deeper knowledge about yarn--we switched it to npm when we were working on the patch release that included Content Security Policy updates, because we knew how to coax the dependency tree into picking up the right DLJS, and in turn the correct bf-streaming version.

React App Dependencies

• react-app
  • bf-webchat
    • pinned version of dljs
      • pinned version of bf-streaming
  • custom dljs
    • nightly version bf-streaming

I couldn't figure out how to tell yarn to not use the pinned version of webchat and to instead use the custom dljs created by the pipeline--it kept on picking up the pinned version, which didn't have the latest streaming bits
And given the time crunch and the fact that the rest of the bb-js repo used npm, we switched react app to npm, since we knew how to mess with the dependency tree there for the tests.

---

That being said, I have full confidence that you thankfully know yarn a lot better than I do, and can switch everything over 🎊 🙌

[joshgummersall]

@stevengum and @munozemilio I just pushed one last commit that supports more complicated versioning scenarios. It's worth taking a quick look at the repo utils work and the daily build YAML to make sure I'm not doing anything crazy.

[stevengum]

Still approving 👍

I had some questions regarding status code and another for import style and fs/promises

[stevengum]

Approving latest commit (3ba7142) and responses to my last review.