Smoke Tests should onboard samples as opt-out tests

[danieljurek]

Description of architecture: https://github.com/Azure/azure-sdk-for-js/tree/master/common/smoke-test

See implementation: https://github.com/Azure/azure-sdk-for-js/blob/master/common/smoke-test/Initialize-SmokeTests.ps1

Preparer

• Deploy resources from sdk/**/test-resources.json (all test-resources.json files that can be found)
• If deployment succeeds and there are samples which are not excluded*:
  • Onboard samples code (may require code editing or code generation)
  • Add to run manifest (may require code generation or, in the case of dynamic languages, adding references to a file for a runner to execute)
  • Add required dependencies to runner (i.e. add package dependency for latest dev feed of relevant packages)
• If deployment fails:
  • Note the failure of the deployment using DevOps logging commands (link)

*It may be necessary to exclude some samples that won't run correctly under this configuration. Sample exclusion should be configured in the repo in a way that, if the samples were published to documentation, the exclusion information would not be visible in that documentation. For JS this is accomplished by adding a field to the package.json file of the package itself that is under test (not the samples). The preparer skips adding samples to the run manifest which are excluded.

Runner application

• Runs different tests depending on results from the preparer (may require editing or generating code)
• Executes samples which are onboarded to run in the runner
• Uses dependencies from the latest dev feed packages
• Reports failures using DevOps logging commands (link)

[kurtzeborn]

Bumping this back to "client" label. This is work that should be done by language devs. As noted in the description, the prototype is done for JS and the smoke tests are now based on samples rather than purpose built smoke tests. Java devs are working on this now for Java smoke tests. This issue it to track the conversion for .NET samples into smoke tests.

[jsquire]

Given the age of this issue and the approach of live test snippets as samples, closing this as stale. Please feel free to reopen if this is still needed.