Skip to content

PII Types for 3.1-Preview.4#12392

Merged
tjprescott merged 11 commits intoAzure:dev-cognitiveservices-TextAnalytics-v3.1-preview.4from
thediris:thediris/pii_filter_dev
Jan 22, 2021
Merged

PII Types for 3.1-Preview.4#12392
tjprescott merged 11 commits intoAzure:dev-cognitiveservices-TextAnalytics-v3.1-preview.4from
thediris:thediris/pii_filter_dev

Conversation

@thediris
Copy link
Contributor

Add support for PII type filters as a request param - takes in a comma separated list of strings describing the types we support

@thediris thediris requested a review from yangyuan as a code owner January 12, 2021 18:07
@openapi-workflow-bot
Copy link

Hi, @thediris Thanks for your PR. I am workflow bot for review process. Here are some small tips.

  • Please ensure to do self-check against checklists in first PR comment.
  • PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.
  • For specs comparison cross API versions, Use API Specs Comparison Report Generator
  • If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?

  • Any feedback about review process or workflow bot, pls contact swagger and tools team. vsswagger@microsoft.com

    @openapi-pipeline-app
    Copy link

    openapi-pipeline-app bot commented Jan 12, 2021

    Swagger Validation Report

    ️️✔️BreakingChange succeeded [Detail] [Expand]
    There are no breaking changes.
    ️❌LintDiff: 1 Errors, 0 Warnings failed [Detail]
    Rule Message
    AutoRest Exception "details":" "Channel": "fatal",
    "Details": null,
    "Text": "Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: [. Path 'parameters.PiiCategories.default',
    line 1,
    position 61348.\n at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)\n at Newtonsoft.Json.JsonTextReader.ReadAsString()\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract,
    Boolean hasConverter)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject,
    JsonReader reader,
    JsonObjectContract contract,
    JsonProperty member,
    String id)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader,
    Type objectType,
    JsonContract contract,
    JsonProperty member,
    JsonContainerContract containerContract,
    JsonProperty containerMember,
    Object existingValue)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary,
    JsonReader reader,
    JsonDictionaryContract contract,
    JsonP",
    "location":"https://github.com/Azure/azure-rest-api-specs/blob/76dd8dc32e145392f1fa14a4295184176dd1342b/specification/cognitiveservices/data-plane/TextAnalytics/readme.md"
    ️️✔️Avocado succeeded [Detail] [Expand]
    Validation passes for Avocado.
    ️️✔️ModelValidation succeeded [Detail] [Expand]
    Validation passes for ModelValidation.
    ️❌SemanticValidation: 23 Errors, 0 Warnings failed [Detail] Only 10 items are listed, please refer to log for more details.
    Rule Message
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: in
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: name
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    INVALID_TYPE Expected type array but found type boolean
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2389
    INVALID_TYPE Expected type array but found type boolean
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2389
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: name
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: in
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    INVALID_TYPE Expected type array but found type object
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L853
    INVALID_TYPE Expected type array but found type boolean
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2389
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: name
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    OBJECT_ADDITIONAL_PROPERTIES Additional properties not allowed: in
    JsonUrl: TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json#L2376
    ️️✔️[Staging] Cross Version BreakingChange (Base on preview version) succeeded [Detail]
    Rule Message
    Runtime Exception "new":"https://github.com/Azure/azure-rest-api-specs/blob/76dd8dc32e145392f1fa14a4295184176dd1342b/specification/cognitiveservices/data-plane/TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json",
    "old":"https://github.com/Azure/azure-rest-api-specs/blob/master/specification/cognitiveservices/data-plane/TextAnalytics/preview/v3.1-preview.3/TextAnalytics.json",
    "details":"Error: Command failed: dotnet /home/vsts/work/1/a/unified-pipeline-runtime/common/temp/node_modules/.pnpm/@azure/oad@0.8.6/node_modules/@azure/oad/dlls/OpenApiDiff.dll -o /tmp/old-resolved.json -n /tmp/new-resolved.json
    Unhandled Exception: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: [. Path 'parameters.PiiCategories.default',
    line 1,
    position 65857.\n at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)\n at Newtonsoft.Json.JsonTextReader.ReadAsString()\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader,
    JsonContract contract,
    Boolean hasConverter)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject,
    JsonReader reader,
    JsonObjectContract contract,
    JsonProperty member,
    String id)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader,
    Type objectType,
    JsonContract contract,
    JsonProperty member,
    JsonContainerContract "
    ️️✔️[Staging] Cross Version BreakingChange (Base on stable version) succeeded [Detail]
    Rule Message
    Runtime Exception "new":"https://github.com/Azure/azure-rest-api-specs/blob/76dd8dc32e145392f1fa14a4295184176dd1342b/specification/cognitiveservices/data-plane/TextAnalytics/preview/v3.1-preview.4/TextAnalytics.json",
    "old":"https://github.com/Azure/azure-rest-api-specs/blob/master/specification/cognitiveservices/data-plane/TextAnalytics/stable/v3.0/TextAnalytics.json",
    "details":"Error: Command failed: dotnet /home/vsts/work/1/a/unified-pipeline-runtime/common/temp/node_modules/.pnpm/@azure/oad@0.8.6/node_modules/@azure/oad/dlls/OpenApiDiff.dll -o /tmp/old-resolved.json -n /tmp/new-resolved.json
    Unhandled Exception: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: [. Path 'parameters.PiiCategories.default',
    line 1,
    position 65857.\n at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)\n at Newtonsoft.Json.JsonTextReader.ReadAsString()\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader,
    JsonContract contract,
    Boolean hasConverter)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject,
    JsonReader reader,
    JsonObjectContract contract,
    JsonProperty member,
    String id)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader,
    Type objectType,
    JsonContract contract,
    JsonProperty member,
    JsonContainerContract "
    Posted by Swagger Pipeline | How to fix these errors?

    @openapi-pipeline-app
    Copy link

    openapi-pipeline-app bot commented Jan 12, 2021

    Swagger Generation Artifacts

    ️❌[Staging] ApiDocPreview: 1 Errors, 0 Warnings failed [Detail]
    Rule Message
    RestBuild error "logUrl":"https://apidrop.visualstudio.com/Content%20CI/_build/results?buildId=189763&view=logs&j=fd490c07-0b22-5182-fac9-6d67fe1e939b",
    "detail":"Run.ps1 failed with exit code 128 "
    ️️✔️ azure-sdk-for-go succeeded [Detail] [Expand]
    • ️✔️Succeeded [Logs] Generate from 2f3f4b1fb990e86053c25fbead37d8f5688d5940. SDK Automation 14.0.0
      command	sh ./initScript.sh ../../../../../azure-sdk-for-go_tmp/initInput.json ../../../../../azure-sdk-for-go_tmp/initOutput.json
      command	go run ./tools/generator/main.go ../../../../../azure-sdk-for-go_tmp/generateInput.json ../../../../../azure-sdk-for-go_tmp/generateOutput.json
    • ️✔️cognitiveservices/v2.1/textanalytics [View full logs]  [Preview SDK Changes]
      info	[Changelog] No exported changes
    • ️✔️cognitiveservices/v2.0/textanalytics [View full logs]  [Preview SDK Changes]
      info	[Changelog] No exported changes
    ️🔄 azure-sdk-for-python inProgress [Detail]
    ️️✔️ azure-sdk-for-js succeeded [Detail] [Expand]
    • ️✔️Succeeded [Logs] Generate from 2f3f4b1fb990e86053c25fbead37d8f5688d5940. SDK Automation 14.0.0
      warn	Skip initScript due to not configured
      command	autorest --version=V2 --typescript --license-header=MICROSOFT_MIT_NO_VERSION --use=@microsoft.azure/autorest.typescript@4.4.4 --typescript-sdks-folder=/home/vsts/work/1/s/azure-sdk-for-js/azure-sdk-for-js ../../azure-rest-api-specs/specification/cognitiveservices/data-plane/TextAnalytics/readme.md
    • ️✔️@azure/cognitiveservices-textanalytics [View full logs]  [Preview SDK Changes]
      cmderr	[npmPack] loaded rollup.config.js with warnings
      cmderr	[npmPack] (!) Unused external imports
      cmderr	[npmPack] default imported from external module 'rollup' but never used
      cmderr	[npmPack] ./esm/textAnalyticsClient.js → ./dist/cognitiveservices-textanalytics.js...
      cmderr	[npmPack] created ./dist/cognitiveservices-textanalytics.js in 179ms
    ️⚠️ azure-sdk-for-java warning [Detail]
    • ⚠️Warning [Logs] Generate from 2f3f4b1fb990e86053c25fbead37d8f5688d5940. SDK Automation 14.0.0
      command	./eng/mgmt/automation/init.sh ../azure-sdk-for-java_tmp/initInput.json ../azure-sdk-for-java_tmp/initOutput.json
      command	./eng/mgmt/automation/generate.py ../azure-sdk-for-java_tmp/generateInput.json ../azure-sdk-for-java_tmp/generateOutput.json
      cmderr	[generate.py] 2021-01-22 18:04:15 ERROR [Skip] readme path does not format as specification/*/resource-manager/readme.md
      warn	No file changes detected after generation
      warn	No package detected after generation
    Posted by Swagger Pipeline | How to fix these errors?

    @openapi-workflow-bot
    Copy link

    Hi @thediris, Your PR has some issues. Please fix the CI sequentially by following the order of Avocado, semantic validation, model validation, breaking change, lintDiff.

    TaskHow to fixPrioritySupport (Microsoft alias)
    AvocadoFix-AvocadoHighruowan
    Semantic validationFix-SemanticValidation-ErrorHighraychen, jianyxi
    Model validationFix-ModelValidation-ErrorHighraychen,jianyxi
    LintDiffFix-LintDiffhighjianyxi, ruoxuan
    If you need further help, please feedback via swagger feedback."

    @assafi
    Copy link
    Contributor

    assafi commented Jan 12, 2021

    Could you also update the PII response example?

    Copy link
    Contributor

    @abhahn abhahn left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Looks good to me. :)

    @maririos
    Copy link
    Member

    @iscai-msft could you help get this PR merged?

    Copy link
    Contributor

    @iscai-msft iscai-msft left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    looks like there are still a couple of ci errors that seem valid. addressed some of them in my comments, but can help get it merged once it's fixed. thanks for doing this @thediris !

    @suhas92
    Copy link
    Member

    suhas92 commented Jan 22, 2021

    @tjprescott can we get this merged ?

    Copy link
    Member

    @suhas92 suhas92 left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    💯

    @tjprescott tjprescott merged commit 87f79f1 into Azure:dev-cognitiveservices-TextAnalytics-v3.1-preview.4 Jan 22, 2021
    anuchandy added a commit that referenced this pull request Feb 12, 2021
    * Adds base for updating TextAnalytics from version preview/v3.1-preview.3 to version v3.1-preview.4
    
    * Updates readme
    
    * Updates API version in new specs and examples
    
    * add entity linking task (#12366)
    
    * Opinion Mining API changes for Preview.4 (#12344)
    
    * Adding changed example
    
    * json change for om part 1
    
    * Change API structure as well
    
    * refine text again
    
    * Getting back to false by default
    
    * word chagne
    
    * .
    
    * final wording change
    
    * fix for readme.md
    
    Co-authored-by: josuh <josuh@microsoft.com>
    
    * Analyze: made task results optional (#12373)
    
    * Made results array optional since tasks can be in a failed state with no results
    
    * Fixing a syntax issue in the README
    
    * Attempting to address an Autorest error in the README
    
    * Attempting to address an Autorest error in the README
    
    * Added some info about v3.1-preview.4 to the python readme file; attempting to correct syntax issues in readme.md
    
    * Attempting to fix generation for the Python SDK
    
    * Attempting to fix generation for the Python SDK
    
    Co-authored-by: Abigail Hartman <Abigail.Hartman@microsoft.com>
    
    * add stringtype parameter to entitylinking task (#12467)
    
    * TA v3.1-preview.4 - updated examples and definitions for TA for Health (#12340)
    
    * updated examples and health definitions for new preview version
    
    * fixed linting issue in readme
    
    * removed old properties in ta.json
    
    * undo changes to sentiment portion of response, update health response
    
    * fixed spelling issues
    
    * fixed more typose
    
    * remove linting issue in readme
    
    * added enums to assertions
    
    * add in ms-enum extension
    
    * ran prettier json linter over files
    
    * made assertion field singular
    
    * PII Types for 3.1-Preview.4 (#12392)
    
    * based off of correct branch now
    
    * change the request from enum to a generic list of strings
    
    * cleanup some other comments
    
    * add enum type
    
    * change type --> category
    
    * add categories to the swagger
    
    * update the types to make them read better
    
    * add in the NER types
    
    * add in address type as well
    
    * fix some typos
    
    * one last typo
    
    * Health descritpion and naming updates (#12616)
    
    * updated examples and health definitions for new preview version
    
    * fixed linting issue in readme
    
    * removed old properties in ta.json
    
    * undo changes to sentiment portion of response, update health response
    
    * fixed spelling issues
    
    * fixed more typose
    
    * remove linting issue in readme
    
    * added enums to assertions
    
    * add in ms-enum extension
    
    * ran prettier json linter over files
    
    * made assertion field singular
    
    * addressed comments and updated description
    
    * TA v3.1-preview.4 - Minor syntax changes to address comments on PR 12338  (#12650)
    
    * updated examples and health definitions for new preview version
    
    * fixed linting issue in readme
    
    * removed old properties in ta.json
    
    * undo changes to sentiment portion of response, update health response
    
    * fixed spelling issues
    
    * fixed more typose
    
    * remove linting issue in readme
    
    * added enums to assertions
    
    * add in ms-enum extension
    
    * ran prettier json linter over files
    
    * made assertion field singular
    
    * addressed comments and updated description
    
    * addressed comments for minor syntax changes
    
    * replaced all instances of well-known
    
    * fixed linting issue - pii category was set to list instead of string
    
    * [text analytics] Fixed Swagger validation issues for v3.1-preview.4 (#12660)
    
    * Trying to solve swagger validation issues
    
    * Removed x-ms-enum property from PiiCategories
    
    * Created a definition for PiiCategories in addition to a parameter in order to support passing the same thing in the body of a Task object
    
    * Ran 'prettier' to, uh... make it prettier.
    
    * Updated custom-words.txt
    
    * Corrected typos in Swagger and removed unneeded entries in custom-words.txt
    
    * Changed 'default' to 'x-ms-client-default' in order to pass both model validation and linting
    
    * Fixed a misspelled PII category
    
    * Reran prettier check
    
    * Removed an experimental line in the Swagger
    
    * Adding collectionFormat to fix an autorest error
    
    * Removing x-ms-client-default
    
    Co-authored-by: Abigail Hartman <Abigail.Hartman@microsoft.com>
    
    * Added Extensible Enum for Health Relation Types - Dev cognitiveservices text analytics v3.1 preview.4 (#12737)
    
    * updated examples and health definitions for new preview version
    
    * fixed linting issue in readme
    
    * removed old properties in ta.json
    
    * undo changes to sentiment portion of response, update health response
    
    * fixed spelling issues
    
    * fixed more typose
    
    * remove linting issue in readme
    
    * added enums to assertions
    
    * add in ms-enum extension
    
    * ran prettier json linter over files
    
    * made assertion field singular
    
    * addressed comments and updated description
    
    * addressed comments for minor syntax changes
    
    * replaced all instances of well-known
    
    * fixed linting issue - pii category was set to list instead of string
    
    * revert change to pii default
    
    * remove default value as this is an optional parameter
    
    * added extensible list to relation types for healthcare
    
    * revert change to pii default
    
    * changed enum type and made extensible via the modelAsString param
    
    * ran npm prettier check and fixed spelling error (#12947)
    
    Co-authored-by: Chris Tufts <christopher.tufts25@gmail.com>
    Co-authored-by: Mikael <mikael.sitruk@outlook.com>
    Co-authored-by: Joohee Suh <58048046+josuhazure@users.noreply.github.com>
    Co-authored-by: josuh <josuh@microsoft.com>
    Co-authored-by: Abigail Hartman <Abigail.Hahn@microsoft.com>
    Co-authored-by: Abigail Hartman <Abigail.Hartman@microsoft.com>
    Co-authored-by: Thaminda Edirisooriya <61993993+thediris@users.noreply.github.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Projects

    None yet

    Development

    Successfully merging this pull request may close these issues.

    8 participants