[Azure Stream Analytics] Add Power BI and Azure Data Lake Store output support

[atpham256]

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.

PR information

• The title of the PR is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
• Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
• If applicable, the PR references the bug/issue that it fixes.
• Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

• I have read the contribution guidelines.
• My spec meets the review criteria:
  • The spec conforms to the Swagger 2.0 specification.
  • The spec follows the guidelines described in the Swagger checklist document.
  • Validation tools were run on swagger spec(s) and have all been fixed in this PR.

[msftclas]

@atpham256,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

[atpham256]

Ran validation tools and there were no new errors compared to the previous time I ran it for the last PR.

The OBJECT_ADDITIONAL_PROPERTIES errors for oav validate-example seems to be a bug on the tool where it cannot handle response bodies with polymorphic response object models which was also present in my last PR. The examples are definitely valid in this case.

[azuresdkciprbot]

Hi There,

I am the AutoRest Linter Azure bot. I am here to help. My task is to analyze the situation from the AutoRest linter perspective. Please review the below analysis result:

File: specification/streamanalytics/resource-manager/readme.md
Before the PR: Warning(s): 4 Error(s): 5
After the PR: Warning(s): 4 Error(s): 5

AutoRest Linter Guidelines | AutoRest Linter Issues

Send feedback and make AutoRest Linter Azure Bot smarter day by day!

Thanks for your co-operation.

[olydis]

@ravbhatnagar Breaking change in existing API version (addition of new type in polymorphic hierarchy that can be returned from server => deserialization can fail in previously generated artifacts)

[olydis]

@ravbhatnagar ping

[ravbhatnagar]

@atpham256 - The service seems to be supporting a new model type (PowerBIOutput). Please update the api-version for this change since this is a breaking change.

[atpham256]

@ravbhatnagar Hi Gaurav, this is an additive change that we do not consider a breaking change for our customers. From our talks with other teams such as ADF, they seem to be following a similar model of not considering additive changes as breaking changes. We have been using this model for a while and have not had any complaints from our customers about additive changes. Also, this has also been exposed in our service for a while (if you look at Portal, you can create a Power BI or ADLS output for ASA jobs).

Having to add a new API version for each small change we do does not seem scalable since we will end up with tons of API versions we will need to keep track of.

[venkatcmsft]

Hi Gaurav - this is how we have shipped SDK for last 3 yrs. We internally talked about this and potentially changing this. We talked to few other teams and decided to continue this model as others are doing the same and also we have not heard any feedback so far on this model. Given the cost of changing API version for every change, we are only changing API version for any breaking changes for existing behaviors/APIs. all additive changes are treated as non-breaking currently.

[ravbhatnagar]

@venkatcmsft @atpham256 - I think we discussed this in our call at length and we had agreement that this was the right thing to do for the customers. Otherwise, how will be customers know when you start supporting new capabilities in your service (ex - by adding new properties etc.). This is an Azure wide standard we want to follow and push teams towards. The fact that some other team is also not following the recommendation should not be the benchmark here.
Lets make sure we do the right thing for the customers. If you feel, versioning APIs in such cases is not the right thing to do, then lets have a call to discuss this further.

[venkatcmsft]

Gaurav - Yes, we had a meeting about it. We then talked internally about how to execute on it. We concluded that incrementing API version for non-breaking additive changes is not required. reasons: Unless users do strict schema validation of the response, they would not fail on deserialization and that is not common (we have 15k unique customers and no one ever complained about this as this is how APIs have been developed for the last 3 yrs in this service team). That is the reason we went around talking to various teams about how they are managing this kind of changes and found that it has been a problem for these other services we talked to as well. so, they decided to do what is working for their customers.
I agree with coming up with standard guidelines to enforce so customers have consistent experience. However, the model you are proposing is too strict and results in ugly code to have to manage lots of API version based checks for supporting features in code.

Let us take it offline and close on this as it is hard to get to closure on this thread.

[johanste]

Please note that if you are using AutoRest to generate clients, strong client side validation will indeed be happening (at least in Python and Node/javascript). The client will fail the deserialization when discriminator values that it does not know about are encountered.

[atpham256]

@ravbhatnagar Hi Gaurav, as discussed on Friday, can we approve this PR as an exception to unblock work for a dependent team? These 2 new polymorphic types have also already been exposed in this API version for a long time so we would like them to be in the Swagger spec to accurately document the API version.

[ravbhatnagar]

@olydis - It was agreed over skype call that we will let this through since this change has been in on the service side for a long time now. This has been the model Stream Analytics has been following when adding new properties to existing APIs etc. And so, although not ideal, we are approving this PR. Stream Analytics team will take a workitem on their side to start support api-versioning for their service for such scenarios.

[salameer]

@olydis, I think the ball is in your court now :)

[atpham256]

@olydis Ping! :)

[olydis]

reviewing 😉

[olydis]

LGTM

[AutorestCI]

No modification for AutorestCI/azure-sdk-for-node

[AutorestCI]

Azure/azure-sdk-for-ruby#935

[AutorestCI]

No modification for AutorestCI/azure-sdk-for-python