Skip to content

Conversation

@avneeshrai
Copy link

@avneeshrai avneeshrai commented May 9, 2017

Adding swagger spec for RecoveryServices.SiteRecovery service (2016-08-10).

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.

PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
  • If applicable, the PR references the bug/issue that it fixes.
  • Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger


This change is Reviewable

@msftclas
Copy link

msftclas commented May 9, 2017

@avneeshrai,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Copy link
Contributor

@dsgouda dsgouda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this is not being added to the directory here instead

@avneeshrai
Copy link
Author

Note for the reviewers:
This has already been reviewed in PR #1016
I feel @veronicagg would be the best to review this PR as she already has context on it.

  1. Validator tool shows following category of issues:
    "code": "OperationsAPIImplementationValidation",
    "code": "ProvidersPathValidation",
    "code": "PutRequestResponseValidation",
    "code": "TrackedResourceListByImmediateParent",
    "code": "XmsExamplesProvidedValidation",
    "code": "DeleteOperationNameValidation",
    "code": "DefinitionsPropertiesNamesCamelCase",
    "code": "PatchBodyParametersSchemaValidation",
    "code": "BodyTopLevelProperties",
    "code": "AvoidNestedProperties",
    "code": "BooleanPropertyNotRecommended"

Except on "code": "OperationsAPIImplementationValidation" we had closed on all other issues in the current thread. About the implementation of operations API, we expect operations API for our provider Microsoft.RecoveryServices to be present in https://github.com/Azure/azure-rest-api-specs/blob/08f0f317ccb89b86f552232bd8fce02f98889463/arm-recoveryservices/2016-06-01/swagger/vaultusages.json spec (which is already there). Let me know if we have a reason to duplicate it here.
2. Examples for 102 out of 109 APIs have been included in the PR. We couldn’t include examples for remaining 7 because of the issues mentioned below with the validator tool:

  • a)All Job related APIs
    "ReplicationJobs_Get",
    "ReplicationJobs_Resume",
    "ReplicationJobs_GetTrackAsyncOperation",
    "ReplicationJobs_Restart",
    "ReplicationJobs_Cancel",
    "ReplicationJobs_Export",
    "ReplicationJobs_List"
    Reason: Recursive call in ASRTask model

Error seen in validator tool:
error: Found errors in validating the response with statusCode "200" for x-ms-example "Gets the job details." in operation "ReplicationJobs_Get".:

{ code: 'RESPONSE_VALIDATION_ERROR',
message: 'Found errors in validating the response with statusCode "200" for x-ms-example "Gets the job details." in operation "ReplicationJobs_Get".',
innerErrors:
[ { code: 'INVALID_RESPONSE_BODY',
errors:
[ { code: undefined,
message: 'Maximum call stack size exceeded',
path: undefined } ],
message: 'Invalid body: Maximum call stack size exceeded',
path: [] } ] }

  • b)Many APIs
    E.g.: ReplicationFabrics_Get

Fields having “null” value are not allowed.

Error seen in validator tool:
error: Found errors in validating the response with statusCode "200" for x-ms-example "Gets the details of a Site." in operation "ReplicationFabrics_Get".:

{ code: 'RESPONSE_VALIDATION_ERROR',
message: 'Found errors in validating the response with statusCode "200" for x-ms-example "Gets the details of a Site." in operation "ReplicationFabrics_Get".',
innerErrors:
[ { code: 'INVALID_RESPONSE_BODY',
errors:
[ { code: 'INVALID_TYPE',
params: [ 'string', 'null' ],
message: 'Expected type string but found type null',
path:
[ 'properties',
'rolloverEncryptionDetails',
'kekCertThumbprint' ],
description: 'Gets or sets the key encryption key certificate thumbprint.' },
{ code: 'INVALID_TYPE',
params: [ 'string', 'null' ],
message: 'Expected type string but found type null',
path: [ 'properties', 'encryptionDetails', 'kekCertThumbprint' ],
description: 'Gets or sets the key encryption key certificate thumbprint.' } ],
message: 'Invalid body: Value failed JSON Schema validation',
path: [] } ] }

  • c)Many APIs
    Eg: ReplicationPolicies_Get
    Polymorphic objects are not allowed, only discriminator field is getting accepted by validator.

Error in validator tool
error: Found errors in validating the response with statusCode "200" for x-ms-example "Gets the requested policy." in operation "ReplicationPolicies_Get".:

{ code: 'RESPONSE_VALIDATION_ERROR',
message: 'Found errors in validating the response with statusCode "200" for x-ms-example "Gets the requested policy." in operation "ReplicationPolicies_Get".',
innerErrors:
[ { code: 'INVALID_RESPONSE_BODY',
errors:
[ { code: 'OBJECT_ADDITIONAL_PROPERTIES',
params:
[ [ 'activeStorageAccountId',
'replicationInterval',
'applicationConsistentSnapshotFrequencyInHours',
'recoveryPointHistoryDurationInHours' ] ],
message: 'Additional properties not allowed: activeStorageAccountId,replicationInterval,applicationConsistentSnapshotFrequencyInHours,recoveryPointHistoryDurationInHours',
path: [ 'properties', 'providerSpecificDetails' ],
description: 'Hyper V Replica Azure provider specific settings.\r\n Replicated item properties specific to the HyperVReplicaAzure replication provider.' } ],
message: 'Invalid body: Additional properties not allowed: activeStorageAccountId,replicationInterval,applicationConsistentSnapshotFrequencyInHours,recoveryPointHistoryDurationInHours',
path: [] } ] }

I have sent a mail to @veronicagg , @salameer with details on closures.

@avneeshrai
Copy link
Author

@dsgouda These projects are managed by separate teams just that the provider is same. For example you can see
https://github.com/Azure/azure-rest-api-specs/tree/master/arm-recoveryservices
and
https://github.com/Azure/azure-rest-api-specs/tree/master/arm-recoveryservicesbackup

are in different directories.

@avneeshrai
Copy link
Author

@dsgouda have added you to the mail thread.

@veronicagg veronicagg assigned veronicagg and unassigned dsgouda May 9, 2017
@veronicagg
Copy link
Contributor

@avneeshrai

  1. Regarding OperationsAPIImplementation, adding Gaurav. @ravbhatnagar should this spec contain a duplicate of the operations API (the same provider has an operation here https://github.com/Azure/azure-rest-api-specs/blob/08f0f317ccb89b86f552232bd8fce02f98889463/arm-recoveryservices/2016-06-01/swagger/vaultusages.json) ?
  2. Examples should be included even though there are failures reported for some of them (unless it completely blocks the use of the tool for the rest - like a critical exceptions). For failures that are known or new tool bugs, we would ignore the failure for now and should go away once the bug gets fixed.
    a) Regarding "Maximum call stack size exceeded" issue, could you please open an issue for it at https://github.com/Azure/oav/issues (I don't think it's a known failure)
    b) You should leave the examples in, we'll ignore the failures for now
    c) are you seeing the polymorphic problem in the latest version of the tool? We fixed polymorphic types quite a bit, so I'd be curious to see the specific problem if you're using the latest. If so, please log an issue at https://github.com/Azure/oav/issues for us to take a look.
    The examples are not only used for validation but also for documentation purposes.

@veronicagg
Copy link
Contributor

Is this version of the service already deployed to ARM?

Regarding linter validation errors, our tool has been updated since the time of our last conversation, please review the following:

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be boolean? a property with the same name and description is boolean at https://github.com/Azure/azure-rest-api-specs/pull/1209/files#diff-f47a33852eff8e5696392b354d7a3636R11347

Copy link
Author

@avneeshrai avneeshrai May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. We have exception from your side in email thread.

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be boolean? a property with the same name and description is boolean at https://github.com/Azure/azure-rest-api-specs/pull/1209/files#diff-f47a33852eff8e5696392b354d7a3636R11347

Copy link
Author

@avneeshrai avneeshrai May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we had discussed in our e-mail discussions, it was not possible for us to take any change at the service end. We had decided take up any such change whenever we come up with new version in the service. We had exception from your side on this.

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same. We had decided not to go for flattening.

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same. We had decided not to go for flattening.

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had decided not to go for flattening and closure is in our mail thread.

"$ref": "#/parameters/ResourceName"
},
{
"$ref": "#/para
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need this model to be different in the body compared to the response model - see https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1340

Copy link
Author

@avneeshrai avneeshrai May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already closed on this in our mail thread with exception from your side. For context, we had approval from ARM board which was shared on the email.
Our mail thread can be referred for details.

@avneeshrai
Copy link
Author

@veronicagg

About examples:

  1. We have added the support of examples in 7 missing APIs as you suggested (which are failing in validator with Max Call Stack Size Exceeded message as explained before)
  2. We are still hitting all the three issues mentioned above in examples even after updating validator tool (oav):
    a) Max Call Stack Size Exceeded for responses with recursive references.
    b) Null values for string fields are not supported. Fails with message message: 'Expected type string but found type null'
    c) Polymorphic fields are not supported and validator fails with message: "Additional properties not allowed:"

I would raise the issues as you had suggested for these scenarios.

@avneeshrai
Copy link
Author

@veronicagg Please find inline responses in bolditalics for your queries.

Is this version of the service already deployed to ARM?
[AR] Yes, service version 2016-08-10 is already deployed to ARM.

Regarding linter validation errors, our tool has been updated since the time of our last conversation, please review the following:
[AR] We already have closure with exception from your end on all of the following issues. Please refer the email thread for details.

Consider flattening: https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L2491
(and other locations for warning AvoidNestedProperties)
The following are currently detected as RPC violations - @ravbhatnagar for review as well.
[AR] Why is this reported as RPCVoilation? It looks to be related to nested properties that we had decided to keep as it is.

Error at https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1975
Error https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1963
https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1952
https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1939
https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1927
https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1915
... please look at the other camel casing violations as well.
PutRequestResponseValidation https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1340
PutRequestResponseValidation https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1328
... and others on this category.

@avneeshrai
Copy link
Author

Raised issues for problems with examples stated above:
Azure/oav#115
Azure/oav#116
Azure/oav#117

@veronicagg
Copy link
Contributor

@avneeshrai Just to clarify, 1) the tools have been updated, so issues that may have been false positives before may not be now, so it's good to re-evaluate. 2) it's good to have this documented in this PR.
The flattening warning is not an RPC violations (I was referring to other ones above), and if you decided not to flatten those intentionally, it's ok. Because there are RPC violations reported, I've pulled in @ravbhatnagar in the PR review.

}
}
},
"/Subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.RecoveryServices/vaults/{resourceName}/replicationFabrics/{fabricName}/replicationvCenters/{vCenterName}/operationresults/{jobName}": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason for this API to be exposed? How are you expecting customers to use it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not really sure on inclusion of these APIs (operationresults) in spec. Only rationale i had behind including them in swagger spec was that customers who are looking at swagger spec as documentation of our APIs should get to know what all exists in our entire API surface and what API he can call if he wants to track some operation's result.
However, i don't see much utility for consumers of generated sdk. What is the recommendation here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless there's a specific reason to include them, we're generally not including them. For long running operations, the sdk generated code will provide methods that can poll and obtain the results as appropriate.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have removed all operationresult apis in the latest commit.

}
}
},
"/Subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.RecoveryServices/vaults/{resourceName}/replicationFabrics/{fabricName}/replicationStorageClassifications/{storageClassificationName}/replicationStorageClassificationMappings/{storageClassificationMappingName}/operationresults/{jobName}": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason for this API to be exposed? How are you expecting customers to use it?
similar question for other "operationresults" apis

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as before, I was not really sure on inclusion of these APIs (operationresults) in spec. Only rationale i had behind including them in swagger spec was that customers who are looking at swagger spec as documentation of our APIs should get to know what all exists in our entire API surface and what API he can call if he wants to track some operation's result.
However, i don't see much utility for consumers of generated sdk. What is the recommendation here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above, I would remove them, if you don't have a specific reason to include them.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@avneeshrai
Copy link
Author

@veronicagg I get the reasons now. Thanks for clarifying :)

"$ref": "#/parameters/SubscriptionId"
},
{
"name": "fabricName",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could define this parameter in the global parameters section and have a reference here, if you're using it multiple times. And similarly for any other parameters that you're planning to reuse.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the kind of usage we have for parameters like fabricName, containerName they change in general for every second operation our customers can call, to minimize confusion we have kept all parameter which would remain constant for a instance of initialized client in global list and others as local. This is also consistent with the current usage.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, up to you, my suggestion was mostly for your convenience, so you didn't need to repeat and modify everywhere you needed an update in the swagger.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@veronicagg We are generating the spec automatically. So that is not an issue for us :)

}
}
},
"deprecated": false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to include deprecated: false, as it's the default value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, removing this from all operations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good.

…n ids and 2) Taking PR comments: removing redundant deprecated field from operations.
@avneeshrai
Copy link
Author

avneeshrai commented May 11, 2017

@veronicagg Addressed the comments with new commits.

@ravbhatnagar
Copy link
Contributor

@avneeshrai - Please fix the following issues too -

  1. https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1977
    All the extra properties should go inside the properties bag.
  2. https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1207
    Add the /operations API to the swagger.
  3. https://travis-ci.org/Azure/azure-rest-api-specs/jobs/230461195#L1963
    Patch should not have a required property. Consider fixing.
  4. When you plan on fixing the camel casing violations?

1) Introducing operations API with corresponding example.
2) Fixing required field from patch input (UpdateRecoveryPlanInput).
@avneeshrai
Copy link
Author

@ravbhatnagar Taken all comments that we could. Please find the responses:

  1. Moving extra properties inside the properties bag of 'Job' model, could not be taken because of the following reasons:
    a) We had to introduce ['status,error,startTime,endTime'] properties at the top level of 'Job' because they were needed by Azure template tracking. While developing our APIs and using them through templates, templates looked for status field and other fields at the top level (absence let to failure).
    This link explains the same: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-async-operations
    b) Moreover, even if you suggest alternatives at this point for azure template tracking to work even after pushing these properties inside property bag we can take that only in our next service version (it being a breaking change).
    c) Additional information, 'Job' is not an azure tracked resource.
  2. Introduced operation API and corresponding example as suggested.
  3. Fixed the issue with patch input as per suggestion.
  4. We would be fixing all the camel casing violations in our next service version.

…d corresponding examples.

2) Converting some fields from boolean to string and making corresponding changes in examples.
@avneeshrai
Copy link
Author

@veronicagg @ravbhatnagar I have responded to all the comments (fixing issues wherever possible).
Please let me know if there is any action item left on me.

Copy link
Contributor

@veronicagg veronicagg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple more questions, inline, rest looks good to me.

"networkType": "VmNetworkAsInput",
"networkId": "/subscriptions/c183865e-6077-46f2-a3b1-deb0f4f4650a/resourceGroups/siterecoveryProd1/providers/Microsoft.Network/virtualNetworks/vnetavrai",
"skipTestFailoverCleanup": false,
"skipTestFailoverCleanup": "false",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is really a boolean value, should this property be represented as boolean on the swagger? then you can use boolean true/false.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@veronicagg Service is expecting this as string. Its an intentional change. We realized this while trying to validate a variance of testfailover.

},
"skipTestFailoverCleanup": {
"description": "A value indicating whether the test failover cleanup is to be skipped.",
"type": "boolean"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this change intentional? is the value not a boolean?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Service is expecting this as string. Its an intentional change.

@avneeshrai
Copy link
Author

@veronicagg Responded.

Copy link
Contributor

@veronicagg veronicagg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes! LGTM I'm approving from my side, we just need approval from @ravbhatnagar on the ARM side.

@ravbhatnagar
Copy link
Contributor

@avneeshrai - The following comment from you is not correct -
a) We had to introduce ['status,error,startTime,endTime'] properties at the top level of 'Job' because they were needed by Azure template tracking. While developing our APIs and using them through templates, templates looked for status field and other fields at the top level (absence let to failure).
This link explains the same: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-async-operations

Firstly, POSTs cant be invoked from the template barring a few which are of the pattern list*. Secondly, if you are using the 202 pattern for async operations, the API should return a location header which is used to track the status of the operation. When invoked from template, the above location header is what will be used to poll for completion.
The above behavior is explained in the RPC. You do not need the above properties at the top level for the template to track the provisioning status.
This is a RPC violation. It does not matter if its a proxy resource. Please fix whenever you can. Till then you will be reported for this violation.

@avneeshrai
Copy link
Author

@ravbhatnagar Thanks for the response and explanation Gaurav. That was a gap in our understanding then. Would try taking this up whenever we come up with new service version for breaking changes.

@veronicagg As now we don't have any open actionable item left, can we have this PR merged please :)

@veronicagg veronicagg merged commit e382d03 into Azure:master May 13, 2017
@AutorestCI
Copy link

No modification for Ruby

@AutorestCI
Copy link

No modification for NodeJS

@AutorestCI
Copy link

No modification for Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants