Semantic config by shmed · Pull Request #16257 · Azure/azure-rest-api-specs

shmed · 2021-10-01T00:48:35Z

Early draft describing the new semantic configuration

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Changelog

Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?
- new service onboarding
- new API version
- update existing version for new feature
- update existing version to fix swagger quality issue in s360
- Other, please clarify
When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific langauge SDKs and CLIs that must be refreshed after the swagger is published.
- SDK of .NET (need service team to ensure code readiness)
- SDK of Python
- SDK of Java
- SDK of Js
- SDK of Go
- PowerShell
- CLI
- Terraform
- No refresh required for updates in this PR

Contribution checklist:

I commit to follow the Breaking Change Policy of "no breaking changes"
I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.

ARM API Review Checklist

Applicability: ⚠️

If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs

Adding new properties

All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

Check this box if any of the following apply to the PR so that label “WaitForARMFeedback” will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.
- Adding a new service
- Adding new API(s)
- Adding a new API version
  -[ ] To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits.
Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.
If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist

If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

Removing API(s) in a stable version
Removing properties in a stable version
Removing API version(s) in a stable version
Updating API in a stable or public preview version with Breaking Change Validation errors
Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.

Please follow the link to find more details on PR review process.

openapi-workflow-bot · 2021-10-01T00:48:37Z

Hi, @shmed Thanks for your PR. I am workflow bot for review process. Here are some small tips.

Please ensure to do self-check against checklists in first PR comment.

PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.

For specs comparison cross API versions, Use API Specs Comparison Report Generator

If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?

Any feedback about review process or workflow bot, pls contact swagger and tools team. vsswagger@microsoft.com

openapi-workflow-bot · 2021-10-01T00:48:38Z

[Call for Action] To better understand Azure service dev/test scenario, and support Azure service developer better on Swagger and REST API related tests in early phase, please help to fill in with this survey https://aka.ms/SurveyForEarlyPhase. It will take 5 to 10 minutes. If you already complete survey, please neglect this comment. Thanks.

openapi-pipeline-app · 2021-10-01T00:48:39Z

Swagger Validation Report

️❌BreakingChange: 1 Errors, 0 Warnings failed [Detail]

Compared Swaggers (Based on Oad v0.9.1)
- current:Azure.Search/preview/2021-04-30-Preview compared with base:Azure.Search/preview/2021-04-30-Preview

Rule	Message
`❌` Runtime Exception	"new":"https://github.com/Azure/azure-rest-api-specs/blob/14c5f3290c84f894e52f258d82912202bba1b159/specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview", "old":"https://github.com/Azure/azure-rest-api-specs/blob/main/specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview", "details":"Command failed: node /home/vsts/work/1/a/unified-pipeline-runtime/common/temp/node_modules/.pnpm/@Azure+oad@0.9.1/node_modules/autorest/dist/app.js /tmp/readme.oad.old.md --tag=oad-default-tag --output-artifact=swagger-document.json --output-artifact=swagger-document.map --output-file=old --output-folder=/tmp\nFATAL: swagger-document/compose - FAILED\nFATAL: Error: '$["x-ms-parameterized-host"].hostTemplate' has incompatible values (---\n'endpoint/indexes(''indexName'')'\n, ---\n'endpoint'\n).\n Error: '$["x-ms-parameterized-host"].hostTemplate' has incompatible values (---\n'endpoint/indexes(''indexName'')'\n, ---\n'endpoint'\n).\n"

️⚠️LintDiff: 1 Warnings warning [Detail]

Linted configuring files (Based on source branch, openapi-validator v1.10.1 , classic-openapi-validator v1.1.10 )
- data-plane/Azure.Search/readme.md tag:package-2021-04-searchservice-preview
- data-plane/Azure.Search/readme.md tag:package-2021-04-searchindex-preview
Linted configuring files (Based on target branch, openapi-validator v1.10.1 , classic-openapi-validator v1.1.10 )
- data-plane/Azure.Search/readme.md tag:package-2021-04-searchservice-preview
- data-plane/Azure.Search/readme.md tag:package-2021-04-searchindex-preview

The following errors/warnings are introduced by current PR:

Rule	Message
⚠️ R4020 - DescriptiveDescriptionRequired	The value provided for description is not descriptive enough. Accurate and descriptive description is essential for maintaining reference documentation. Location: Azure.Search/preview/2021-04-30-Preview/searchservice.json#L5857

The following errors/warnings exist before current PR submission:

Rule	Message
⚠️ R2007 - LongRunningOperationsWithLongRunningExtension	The operation 'Indexers_Run' returns 202 status code, which indicates a long running operation, please enable 'x-ms-long-running-operation. Location: Azure.Search/preview/2021-04-30-Preview/searchservice.json#L413
⚠️ R4000 - ParameterDescriptionRequired	'keysOrIds' parameter lacks 'description' property. Consider adding a 'description' element. Accurate description is essential for maintaining reference documentation. Location: Azure.Search/preview/2021-04-30-Preview/searchservice.json#L358

️️✔️Avocado succeeded [Detail] [Expand]

Validation passes for Avocado.

️️✔️ModelValidation succeeded [Detail] [Expand]

Validation passes for ModelValidation.

️️✔️SemanticValidation succeeded [Detail] [Expand]

Validation passes for SemanticValidation.

️️✔️Cross-Version Breaking Changes succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️CredScan succeeded [Detail] [Expand]

There is no credential detected.

️️✔️[Staging] SDK Track2 Validation succeeded [Detail] [Expand]

Validation passes for SDKTrack2Validation

The following tags are being changed in this PR

️❌[Staging] PrettierCheck: 3 Errors, 0 Warnings failed [Detail]

Rule	Message
`❌` HowToFix	Code style issues found path: preview/2021-04-30-Preview/examples/SearchServiceCreateIndex.json
`❌` HowToFix	Code style issues found path: Azure.Search/preview/2021-04-30-Preview/searchindex.json
`❌` HowToFix	Code style issues found path: Azure.Search/preview/2021-04-30-Preview/searchservice.json

️️✔️[Staging] SpellCheck succeeded [Detail] [Expand]

Validation passes for SpellCheck.

️️✔️[Staging] Lint(RPaaS) succeeded [Detail] [Expand]

Validation passes for Lint(RPaaS).

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2021-10-01T00:48:39Z

Swagger Generation Artifacts

️️✔️[Staging] ApiDocPreview succeeded [Detail] [Expand]

 Please click here to preview with your @microsoft account.

️️✔️[Staging] SDK Breaking Change Tracking succeeded [Detail] [Expand]

Breaking Changes Tracking

️❌ azure-sdk-for-net-track2 failed [Detail]

❌Failed [Logs] Generate from d34195951093a44fa75bc30cd74c63b2a176b35b. SDK Automation 14.0.0

command	pwsh ./eng/scripts/Automation-Sdk-Init.ps1 ../azure-sdk-for-net_tmp/initInput.json ../azure-sdk-for-net_tmp/initOutput.json
warn	File azure-sdk-for-net_tmp/initOutput.json not found to read
command	pwsh ./eng/scripts/Automation-Sdk-Generate.ps1 ../azure-sdk-for-net_tmp/generateInput.json ../azure-sdk-for-net_tmp/generateOutput.json
cmderr	[Automation-Sdk-Generate.ps1] �[91mWrite-Error: �[91mError occurred while generating artifacts for Azure.Search.Documents
cmderr	[Automation-Sdk-Generate.ps1] �[0m

❌Azure.Search.Documents [View full logs] [Preview SDK Changes]
```
info	[Changelog]
```

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-workflow-bot · 2021-10-01T00:50:27Z

Hi @shmed, Your PR has some issues. Please fix the CI sequentially by following the order of Avocado, semantic validation, model validation, breaking change, lintDiff.

Task	How to fix	Priority	Support (Microsoft alias)
Avocado	Fix-Avocado	High	ruowan
Semantic validation	Fix-SemanticValidation-Error	High	raychen, jianyxi
Model validation	Fix-ModelValidation-Error	High	raychen,jianyxi
LintDiff	Fix-LintDiff	high	jianyxi, ruoxuan

If you need further help, please feedback via swagger feedback."

HeidiSteen

Left some comments

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchindex.json

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json

tjprescott · 2021-10-21T20:13:46Z

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchindex.json

            }
          },
+          {
+            "name": "semanticConfiguration",


In Python this will end up getting exposed as semantic_configuration_name. Is that the case for other languages? If so, perhaps an x-ms-client-name would be appropriate.

I'm not familiar with how names are generated in various languages, though I do want to make sure it's consistent across parameters. For this specific parameter, I was following the footstep of the "scoringProfile" parameter above. If the scoringProfile parameter gets translated into scoring_profile_name in python, then I would want the same for semanticConfiguration.

They don't get translated that way--in our convenience layer we will have to manually rename it. If all languages would expose this as SemanticConfigurationName, then it would be useful to have the x-ms-client-name here. @Mohit-Chakraborty @alzimmermsft @sarangan12

I personally dont have preferences as to whether we should call it SemanticConfigurationName or SemanticConfiguration, all I want to be sure of is that we follow the pattern of the other parameter defined right above this one (ScoringProfile). Right now, ScoringProfile does not have an x-ms-client-name property, so I want to make sure we dont end up in a situation where ScoringProfile and SemanticConfiguration end up with different naming patterns.

At least for Java, this will result in a private property called semanticConfiguration and getter and setter methods getSemanticConfiguration and setSemanticConfiguration. For the most part Java code generation leaves the Swagger name field untouched and uses it directly as-is.

tjprescott · 2021-10-21T20:14:36Z

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json

      },
      "description": "Ranking function based on the Okapi BM25 similarity algorithm. BM25 is a TF-IDF-like algorithm that includes length normalization (controlled by the 'b' parameter) as well as term frequency saturation (controlled by the 'k1' parameter)."
    },
+    "Semantic": {


Should this be SemanticConfigurations?

semanticConfigurations is the name of one of the nested property under "semantic" will contain more property in the future (including defaultSemanticConfiguration and maybe a few others)

There is no issue with a property having the same name as a class, AFAIK.

SemanticConfigurations.semantic_configurations is perfectly valid, if somewhat redundant. We don't want classes generated called Semantic.

openapi-workflow-bot · 2021-10-27T18:54:09Z

NewApiVersionRequired reason:

A service’s API is a contract with customers and is represented by using the api-version query parameter. Changes such as adding an optional property to a request/response or introducing a new operation is a change to the service’s contract and therefore requires a new api-version value. This is critically important for documentation, client libraries, and customer support.

EXAMPLE: if a customer calls a service in the public cloud using api-version=2020-07-27, the new property or operation may exist but if they call the service in a government cloud, air-gapped cloud, or Azure Stack Hub cloud using the same api-version, the property or operation may not exist. Because there is no clear relationship between the service api-version and the new property/operation, customers can’t trust the documentation and Azure customer have difficulty helping customers diagnose issues. In addition, each client library version documents the service version it supports. When an optional property or new operation is added to a service and its Swagger, new client libraries must be produced to expose this functionality to customers. Without updating the api-version, it is unclear to customers which version of a client library supports these new features.

tjprescott · 2021-10-28T19:58:09Z

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json

    },
+    "Semantic": {
+      "properties": {
+        "semanticConfigurations": {


Based on my above comment, I would recommend calling this configurations. Then you would have SemanticConfigurations.configurations which is less rendundant.

tjprescott · 2021-10-28T19:59:18Z

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json

            "url": "https://docs.microsoft.com/azure/search/index-ranking-similarity"
          }
        },
+        "semantic": {


This is not a good property name. It should probably be called semanticConfigurations.

tjprescott

I didn't include my comments in a review (whoops) but they are in there. Mostly naming issues.

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json

tjprescott · 2021-11-02T14:18:08Z

@shmed please reopen this and target the Azure:main branch. Ensure you tag me as the assignee and reference this PR.

shmed · 2021-11-02T23:58:18Z

@shmed please reopen this and target the Azure:main branch. Ensure you tag me as the assignee and reference this PR.

Created this new PR -> #16642

I can't seem to find how to change the "assignee" though. Can you help? @tjprescott

shmed added 4 commits September 30, 2021 17:14

Add semantic skeleton

b502a6c

Add query time semantic configuration parameter

231f4a1

Add example for semantic configuration

28b7b0d

Update descriptions

c9f1567

openapi-pipeline-app bot added the CI-FixRequiredOnFailure label Oct 1, 2021

Fix syntax issue

e335dab

openapi-sdkautomation bot mentioned this pull request Oct 1, 2021

[AutoPR Azure.Search.Documents] Semantic config AzureSDKAutomation/azure-sdk-for-net#6150

Draft

HeidiSteen reviewed Oct 1, 2021

View reviewed changes

dereklegenzoff reviewed Oct 4, 2021

View reviewed changes

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json Outdated Show resolved Hide resolved

dereklegenzoff reviewed Oct 4, 2021

View reviewed changes

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json Outdated Show resolved Hide resolved

tjprescott self-assigned this Oct 18, 2021

tjprescott reviewed Oct 21, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into semantic_config

cef9f7f

openapi-pipeline-app bot added the NewApiVersionRequired label Oct 27, 2021

shmed changed the base branch from main to master October 27, 2021 18:57

shmed added 2 commits October 27, 2021 14:41

Update descriptions + add newly supported languages

39bda0c

Add enum descriptions for languages

d83369c

tjprescott reviewed Oct 28, 2021

View reviewed changes

tjprescott requested changes Oct 28, 2021

View reviewed changes

tjprescott mentioned this pull request Oct 29, 2021

[Search] Add Semantic Configuration to Azure Search Azure/azure-sdk-for-python#21494

Closed

tjprescott reviewed Oct 29, 2021

View reviewed changes

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json Outdated Show resolved Hide resolved

tjprescott reviewed Oct 29, 2021

View reviewed changes

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json Outdated Show resolved Hide resolved

dereklegenzoff reviewed Nov 1, 2021

View reviewed changes

specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json Outdated Show resolved Hide resolved

Fix syntax and spelling

14c5f32

openapi-pipeline-app bot removed the CI-FixRequiredOnFailure label Nov 2, 2021

akning-ms deleted the branch Azure:master November 2, 2021 09:15

akning-ms closed this Nov 2, 2021

shmed mentioned this pull request Nov 2, 2021

Add Semantic configuration to Azure Search specs #16642

Closed

25 tasks

Conversation

shmed commented Oct 1, 2021

Changelog

Contribution checklist:

ARM API Review Checklist

Breaking Change Review Checklist

Uh oh!

openapi-workflow-bot bot commented Oct 1, 2021

Uh oh!

openapi-workflow-bot bot commented Oct 1, 2021

Uh oh!

openapi-pipeline-app bot commented Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Swagger Validation Report

Uh oh!

openapi-pipeline-app bot commented Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Swagger Generation Artifacts

Breaking Changes Tracking

Uh oh!

openapi-workflow-bot bot commented Oct 1, 2021

Uh oh!

HeidiSteen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjprescott Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openapi-workflow-bot bot commented Oct 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjprescott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tjprescott commented Nov 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shmed commented Nov 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

openapi-pipeline-app bot commented Oct 1, 2021 •

edited

Loading

openapi-pipeline-app bot commented Oct 1, 2021 •

edited

Loading

tjprescott Oct 28, 2021 •

edited

Loading

tjprescott commented Nov 2, 2021 •

edited

Loading