Skip to content

Azure Cognitive Search 2021-04-30-Preview dataplane API update #21993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Wenyang Fu (wenyangfu) wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Azure Cognitive Search 2021-04-30-Preview dataplane API update #21993

Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
cd811b9
Add default semantic config to azure search
wenyangfu Nov 1, 2022
854a534
Add default semantic config
wenyangfu Nov 1, 2022
6b0ae1f
DefaultConfiguration nullable
wenyangfu Nov 3, 2022
24ef6fe
Add default configuration
wenyangfu Nov 15, 2022
6e7bf5c
Add semantic error handling parameter to Search API spec
wenyangfu Nov 15, 2022
f6b54c4
Revise and add semanticErrorHandling
wenyangfu Nov 15, 2022
3df7c4b
Add semanticMaxWaitInMilliseconds parameter
wenyangfu Nov 17, 2022
48939a9
Add semantic failure response to search results
wenyangfu Nov 17, 2022
43d2e8d
Add semantic partial response enums
wenyangfu Nov 17, 2022
491ea24
Revise SemanticMaxWait min valid thresholds
wenyangfu Nov 17, 2022
0e24997
Add debug API component to top-level search result response
wenyangfu Nov 18, 2022
fd37d2d
Add debug search request param
wenyangfu Nov 18, 2022
d7edbf9
Add debug info to each document in response
wenyangfu Nov 18, 2022
4fb0d43
Fix typo in AnswersDebugInfo
wenyangfu Nov 18, 2022
d44cc1f
Iterate on feedback, round 1
wenyangfu Nov 30, 2022
ec75980
Wording: semantic queries -> queries
wenyangfu Nov 30, 2022
5ae129a
Description wording updates
wenyangfu Dec 1, 2022
21671ad
Address another round of feedback
wenyangfu Dec 6, 2022
a12c6f4
More iteration on feedback
wenyangfu Dec 6, 2022
ced497b
Wordsmithing for reranker input description
wenyangfu Dec 6, 2022
652d799
Wordsmithing for semantic partial response
wenyangfu Dec 6, 2022
d374f50
Doc clarity updates
wenyangfu Dec 6, 2022
df266a4
Improve description clarity for semantic partial response
wenyangfu Dec 6, 2022
9e7a97e
Semantic debug info clarity
wenyangfu Dec 6, 2022
eb4a7c0
More doc updates
wenyangfu Dec 6, 2022
a028323
Revise debug API docs
wenyangfu Dec 6, 2022
6c79d29
More revisions
wenyangfu Dec 6, 2022
f8e32bb
Fix some spellcheck, semantic validation, and lintdiff errors
Jan 4, 2023
ea6f097
Fix LintDiff missing type:object specs
wenyangfu Jan 6, 2023
d6b00ce
Fix semantic validation error for search index spec
wenyangfu Jan 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions custom-words.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1613,6 +1613,7 @@ phraselists
Pids
pipelineruns
piqd
Piqd
PITR
pixelated
pkcs
Expand Down Expand Up @@ -1815,7 +1816,9 @@ Reprovision
reprovisioned
Reprovisioning
reprovisions
reranked
reranker
Reranked
Reranker
Reregister
resnest
Expand Down
1 change: 1 addition & 0 deletions ...data-plane/Azure.Search/preview/2021-04-30-Preview/examples/SearchServiceCreateIndex.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@
"k1": 1.3
},
"semantic": {
"defaultConfiguration": "semanticHotels",
"configurations": [
{
"name": "semanticHotels",
Expand Down
1 change: 1 addition & 0 deletions ...ne/Azure.Search/preview/2021-04-30-Preview/examples/SearchServiceCreateOrUpdateIndex.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,7 @@
"accessCredentials": null
},
"semantic": {
"defaultConfiguration": "semanticHotels",
"configurations": [
{
"name": "semanticHotels",
Expand Down
290 changes: 289 additions & 1 deletion specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchindex.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,16 @@
"name": "SearchOptions"
}
},
{
"$ref": "#/definitions/Debug",
"name": "debug",
"in": "query",
"type": "string",
"x-ms-client-name": "Debug",
"x-ms-parameter-grouping": {
"name": "SearchOptions"
}
},
{
"name": "facet",
"in": "query",
Expand Down Expand Up @@ -248,6 +258,45 @@
"name": "SearchOptions"
}
},
{
"name": "semanticErrorHandling",
"in": "query",
"type": "string",
"enum": [
"fail",
"partial"
],
"x-ms-enum": {
"name": "SemanticErrorHandling",
"modelAsString": false,
"values": [
{
"value": "fail",
"name": "Fail",
"description": "Default behavior. If there is an exception during semantic processing, the query will fail and return the appropriate HTTP code depending on the error."
},
{
"value": "partial",
"name": "Partial",
"description": "If there is an exception during semantic processing, the query return partial results with a 206 status code. Partial result content depends on which semantic step failed and the reason for the failure."
}
]
},
"description": "A value that specifies which response to return when semantic processing fails.",
"x-ms-parameter-grouping": {
"name": "SearchOptions"
}
},
{
"name": "semanticMaxWaitInMilliseconds",
"in": "query",
"type": "integer",

@alzimmermsft Alan Zimmer (alzimmermsft) Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have a property "format" indicating whether this is an int32 or int64 integer type.

@alzimmermsft Alan Zimmer (alzimmermsft) Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other warnings raised by the LintDiff check should also be looked into

@wenyangfu Wenyang Fu (wenyangfu) Jan 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other warnings raised by LintDiff relate to missing type:object specifications. Our swagger (with three exceptions) has a history of omitting this, so I'll leave those be to remain consistent. I fixed the int32 property.

@jhendrixMSFT Joel Hendrix (jhendrixMSFT) Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please either fix it now or open (and link) to a tracking issue to clean it up. It's technical debt that leads to the same round of questions in future revisions.

@wenyangfu Wenyang Fu (wenyangfu) Jan 6, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All warnings related to LintDiff should be fixed now, though it seems like the CI is blocked by a merge conflict in custom-words.txt

"format": "int32",
"description": "The time in milliseconds to wait before the semantic processing step times out. Must be set to a value >= 750. Only valid when semanticErrorHandling is set to 'partial'.",
"x-ms-parameter-grouping": {
"name": "SearchOptions"
}
},
{
"name": "searchFields",
"in": "query",
Expand Down Expand Up @@ -1180,6 +1229,133 @@
"additionalProperties": true,
"description": "Captions are the most representative passages from the document relatively to the search query. They are often used as document summary. Captions are only returned for queries of type 'semantic'.."
},
"AnswersDebugInfo": {
"properties": {
"queryClassifiedAsQuestion": {
"type": "boolean",
"description": "Specifies whether the given query was classified as a question.",
"readOnly": true
}
},
"x-ms-client-name": "AnswersDebugInfo",
"description": "Provides debug information for an Answers construct in the search result."
},
"SemanticDebugInfo": {
"x-ms-client-name": "SemanticDebugInfo",
"description": "Provides semantic debug information for this the search result.",
"properties": {
"answers": {
"$ref": "#/definitions/AnswersDebugInfo",
"readOnly": true
}
}
},
"RerankerInputUsage": {
"properties": {
"title": {
"type": "string",
"readOnly": true,
"description": "The portion of the title field used for semantic processing."
},
"content": {
"type": "string",
"readOnly": true,
"description": "The portion of the combined content fields used for semantic processing."
},
"keywords": {
"type": "string",
"readOnly": true,
"description": "The portion of the combined keyword fields used for semantic processing."
}
},
"description": "The combined semantic fields used as input for semantic processing. The input string is subject to a maximum length. The values from the provided fields are concatenated until the maximum limit is reached.",
"x-ms-client-name": "RerankerInput"
},
"SemanticFieldUsage": {
"x-ms-client-name": "SemanticFieldUsage",
"properties": {
"name": {
"type": "string",
"readOnly": true,
"description": "The name of this semantic field."
},
"state": {
"type": "string",
"readOnly": true,
"description": "Indicates whether the contents of the semantic field were used in full, partially, or not at all when constructing the input string for semantic processing.",
"enum": [
"used",
"unused",
"partial"
],
"x-ms-enum": {
"name": "SemanticFieldState",
"modelAsString": true,
"values": [
{
"value": "used",
"name": "Used",
"description": "The field was fully used."
},
{
"value": "unused",
"name": "Unused",
"description": "The field was not used."
},
{
"value": "partial",
"name": "Partial",
"description": "The field was partially used."
}
]
}
}
}
},
"DocumentSemanticDebugInfo": {
"x-ms-client-name": "Semantic",
"description": "Semantic debug information for this match.",
"properties": {
"titleField": {
"$ref": "#/definitions/SemanticFieldUsage",
"description": "The title field used in semantic processing.",
"readOnly": true,
"x-ms-client-name": "TitleField"
},
"contentFields": {
"type": "array",
"items": {
"$ref": "#/definitions/SemanticFieldUsage"
},
"description": "The list of content fields used in semantic processing.",
"readOnly": true,
"x-ms-client-name": "ContentFields"
},
"keywordFields": {
"type": "array",
"items": {
"$ref": "#/definitions/SemanticFieldUsage"
},
"description": "The list of keyword fields used in semantic processing.",
"readOnly": true,
"x-ms-client-name": "KeywordFields"
},
"rerankerInput": {
"$ref": "#/definitions/RerankerInputUsage",
"readOnly": true,
"x-ms-client-name": "RerankerInput"
}
}
},
"DocumentDebugInfo": {
"x-ms-client-name": "DocumentDebugInfo",
"description": "Debug information for this match.",
"properties": {
"semantic": {
"$ref": "#/definitions/DocumentSemanticDebugInfo"
}
}
},
"SearchDocumentsResult": {
"properties": {
"@odata.count": {
Expand Down Expand Up @@ -1216,7 +1392,76 @@
"readOnly": true,
"x-ms-client-name": "Answers",
"x-nullable": true,
"description": "The answers query results for the search operation; null if the answers query parameter was not specified or set to 'none'."
"description": "The Answers construction of a search result. This value will be null if Answers was unspecified or set to 'none'."
},
"@search.debugInfo": {
"description": "Debug information for this search result.",
"properties": {
"semantic": {
"$ref": "#/definitions/SemanticDebugInfo"
}
},
"readOnly": true,
"x-ms-client-name": "DebugInfo",
"x-nullable": true
},
"@search.semanticPartialResponseReason": {
"type": "string",
"readOnly": true,
"x-ms-client-name": "SemanticPartialResponseReason",
"enum": [
"transient",
"maxWaitExceeded",
"capacityOverloaded"
],
"x-ms-enum": {
"name": "SemanticPartialResponseReason",
"modelAsString": true,
"values": [
{
"value": "transient",
"name": "Transient",
"description": "Transient issues caused semantic processing to terminate before all results could be processed."
},
{
"value": "maxWaitExceeded",
"name": "MaxWaitExceeded",
"description": "Semantic processing exceeded the maximum wait time. Only a subset of the initial results were reranked."
},
{
"value": "capacityOverloaded",
"name": "CapacityOverloaded",
"description": "The service was overloaded, causing semantic processing to terminate before all results could be processed."
}
]
},
"description": "The cause for semantic processing failure."
},
"@search.semanticPartialResponseType": {
"type": "string",
"readOnly": true,
"x-ms-client-name": "SemanticPartialResponseType",
"enum": [
"baseResults",
"rerankedResults"
],
"x-ms-enum": {
"name": "SemanticPartialResponseType",
"modelAsString": true,
"values": [
{
"value": "baseResults",
"name": "BaseResults",
"description": "Search results were not semantically reranked and do not contain semantic captions, highlights, or answers."
},
{
"value": "rerankedResults",
"name": "RerankedResults",
"description": "Search results are semantically reranked and include semantic captions, but do not include semantic highlights or answers."
}
]
},
"description": "Indicates whether the response contains just the base search result or semantically reranked search results lacking semantic highlights or answers."
},
"@search.nextPageParameters": {
"$ref": "#/definitions/SearchRequest",
Expand Down Expand Up @@ -1284,6 +1529,12 @@
"x-ms-client-name": "Captions",
"x-nullable": true,
"description": "Captions are the most representative passages from the document relatively to the search query. They are often used as document summary. Captions are only returned for queries of type 'semantic'."
},
"@search.documentDebugInfo": {
"$ref": "#/definitions/DocumentDebugInfo",
"readOnly": true,
"x-ms-client-name": "DocumentDebugInfo",
"description": "Debug information for this match."
}
},
"required": [
Expand Down Expand Up @@ -1529,6 +1780,30 @@
},
"description": "This parameter is only valid if the query type is 'semantic'. If set, the query returns captions extracted from key passages in the highest ranked documents. When Captions is set to 'extractive', highlighting is enabled by default, and can be configured by appending the pipe character '|' followed by the 'highlight-<true/false>' option, such as 'extractive|highlight-true'. Defaults to 'None'."
},
"Debug": {
"type": "string",
"enum": [
"disabled",
"semantic"
],
"x-ms-enum": {
"name": "QueryDebugMode",
"modelAsString": true,
"values": [
{
"value": "disabled",
"name": "Disabled",
"description": "Do not return debug information for the query."
},
{
"value": "semantic",
"name": "Semantic",
"description": "Return semantic debug information for the query."
}
]
},
"description": "This parameter determines whether debug information is returned in a query response. Valid values are 'disabled' and 'semantic'. Values are case-insensitive. The default is 'disabled'. Currently, only queries having queryType set to 'semantic' can return debug information."
},
"QueryLanguage": {
"type": "string",
"enum": [
Expand Down Expand Up @@ -2034,6 +2309,10 @@
"description": "A value that specifies whether to fetch the total count of results. Default is false. Setting this value to true may have a performance impact. Note that the count returned is an approximation.",
"x-ms-client-name": "IncludeTotalResultCount"
},
"debug":{
"$ref": "#/definitions/Debug",
"description": "This parameter determines whether debug information is returned in a query response. Valid values are 'disabled' and 'semantic'. Values are case-insensitive. The default is 'disabled'. Currently, only queries having queryType set to 'semantic' can return debug information."
},
"facets": {
"externalDocs": {
"url": "https://docs.microsoft.com/rest/api/searchservice/Search-Documents"
Expand Down Expand Up @@ -2101,6 +2380,15 @@
"type": "string",
"description": "The name of a semantic configuration that will be used when processing documents for queries of type semantic."
},
"semanticErrorHandling": {
"type": "string",
"description": "A value that specifies whether to fail the request when semantic processing is unavailable or to instead return the base results."
},
"semanticMaxWaitInMilliseconds": {
"type": "integer",
"format": "int32",
"description": "The maximum amount of time to wait for semantic enrichment. If semantic processing exceeds this duration, partial results will be returned instead. This duration does not include time spent retrieving documents from the search index. Must be set to a value greater than or equal to 750. Only valid when semanticErrorHandling is set to 'partial'."
},
"search": {
"type": "string",
"description": "A full-text search query expression; Use \"*\" or omit this parameter to match all documents.",
Expand Down
5 changes: 5 additions & 0 deletions specification/search/data-plane/Azure.Search/preview/2021-04-30-Preview/searchservice.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6044,6 +6044,11 @@
},
"SemanticSettings": {
"properties": {
"defaultConfiguration": {

@alzimmermsft Alan Zimmer (alzimmermsft) Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need sign off from the API stewardship board as this is considered a breaking change as it's adding a new optional property that didn't exist before.

@wenyangfu Wenyang Fu (wenyangfu) Jan 4, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that optional, additive changes were okay and do not require a separate sign-off from the API stewardship board. We went by this policy when we last did swagger changes in late 2021: #16777. Has this changed since then?

Adding BevLoh to chime in.

@BevLoh BevLoh Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still need to get signoff on the exception from the API board. See the DP swagger process doc for details.

@jhendrixMSFT Joel Hendrix (jhendrixMSFT) Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC Mike Kistler (@mikekistler) Jeffrey Richter (@JeffreyRichter)

@JeffreyRichter Jeffrey Richter (JeffreyRichter) Jan 4, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, any change to an API contract requires a new api-version. We used to have an exception for somethings but we no longer do as it was causing too much confusion and people were trying to abuse the system.

@BevLoh BevLoh Jan 5, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jeffrey Richter (@JeffreyRichter), this is news to me. ACS has been adding optional properties to existing API versions for quite some time. As additional data point, we did not get any feedback from our customers that this has been causing confusion. Just to extra confirm, the email exception process that we have agreed upon and used in the past is no longer supported. The reason that I'm asking the team will need to figure out how to schedule additional work to add a new api version which long term also carries extra cost maintenance to the code base.

@JeffreyRichter Jeffrey Richter (JeffreyRichter) Jan 6, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, any change to an API contract requires a new api-version - this is just the best thing for customers and customer support services (when working with customers). It simplifies our rules and attempted abuse. I'm sorry this adversely affects ACS but we felt that this was the best choice after weighing all options.

"type": "string",
"description": "The name of the semantic configuration to use if none is specified in the query.",
"x-nullable": true
},
"configurations": {
"type": "array",
"items": {
Expand Down