-
Notifications
You must be signed in to change notification settings - Fork 5.1k
Split out analyze methods for supervised and unsupervised models #10562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
annelo-msft
merged 36 commits into
Azure:master
from
annelo-msft:fr-split-supervised-unsupervised
Mar 20, 2020
Merged
Changes from 31 commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
cebfd1e
work in progress
annelo-msft fe1ee4f
interim commit
annelo-msft 8f2bef2
update api
annelo-msft fc8e7ea
interim split
annelo-msft 0d75a39
handle multiple receipts
annelo-msft 948520c
api update and bug fix
annelo-msft 0978cbf
api tweaks
annelo-msft b85cce7
update api
annelo-msft 59c3a1e
Merge remote-tracking branch 'upstream/master' into fr-split-supervis…
annelo-msft 09ccbf5
Merge new CodeGen updates.
annelo-msft d66c034
Merge remote-tracking branch 'upstream/master' into fr-split-supervis…
annelo-msft 87bf81d
regen
annelo-msft 9f587c8
api update
annelo-msft 66c81fd
api tweaks
annelo-msft 8e116fa
move to custom namespace
annelo-msft 167e715
client docstrings
annelo-msft a3d105b
README
annelo-msft 1f8e764
README nits
annelo-msft 805b55f
README nits
annelo-msft 432ea3c
README nit
annelo-msft fcdbe1d
README nit
annelo-msft a9f228d
some docstrings
annelo-msft ee7533f
enable docstrings
annelo-msft aca397a
docstrings
annelo-msft 63645db
fix readme
annelo-msft 0cea404
another README fix
annelo-msft 2bc47bf
docstring updates
annelo-msft 875d4d1
UX study updates
annelo-msft 46e08ed
updates from ux study
annelo-msft 1831700
api update
annelo-msft 2fbf2a5
learned form-> page
annelo-msft 70429dd
api updates
annelo-msft dae7f5a
Merge remote-tracking branch 'upstream/master' into fr-split-supervis…
annelo-msft 2998efc
updates for rebase
annelo-msft f099fbf
Merge remote-tracking branch 'upstream/master' into fr-split-supervis…
annelo-msft 6408aa1
pr feedback
annelo-msft File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,197 @@ | ||
| # Azure Cognitive Services Form Recognizer client library for .NET | ||
| Azure Cognitive Services Form Recognizer is a cloud service that uses machine learning to extract text and table data from form documents. It allows you to train custom models using your own forms, to extract field names and values, and table data from them. It also provides a prebuilt models you can use to extract values from receipts, or tables from any form. | ||
|
|
||
| [Source code][formreco_client_src] | <!--[Package (NuGet)]() | [API reference documentation]() |--> [Product documentation][formreco_docs] <!--| [Samples]()--> | ||
|
|
||
| ## Getting started | ||
|
|
||
| ### Prerequisites | ||
| * An [Azure subscription][azure_sub]. | ||
| * An existing Form Recognizer resource. If you need to create the resource, you can use the [Azure Portal][azure_portal] or [Azure CLI][azure_cli]. | ||
| <!-- | ||
| If you use the Azure CLI, replace `<your-resource-group-name>`, `<your-resource-name>`, `<location>`, and `<sku>` with your values: | ||
|
|
||
| ```PowerShell | ||
| az cognitiveservices account create --kind TextAnalytics --resource-group <your-resource-group-name> --name <your-resource-name> --location <location> --sku <sku> | ||
| ``` --> | ||
| <!-- | ||
| ### Install the package | ||
| Install the Azure Text Analytics client library for .NET with [NuGet][nuget]. To use the [.NET CLI](https://docs.microsoft.com/en-us/nuget/consume-packages/install-use-packages-dotnet-cli), run the following command from PowerShell in the directory that contains your project file: | ||
|
|
||
| ```PowerShell | ||
| dotnet add package Azure.AI.TextAnalytics --version 1.0.0-preview.1 | ||
| ``` | ||
| For other installation methods, please see the package information on [NuGet][nuget]. | ||
| --> | ||
|
|
||
| ### Authenticate a Form Recognizer client | ||
| In order to interact with the Form Recognizer service, you'll need to select either a `ReceiptClient`, `FormLayoutClient`, or `CustomFormClient`, and create an instance of this class. In the following samples, we will use CustomFormClient as an example. You will need an **endpoint**, and either a **subscription key** or ``TokenCredential`` to instantiate a client object. For more information regarding authenticating with cognitive services, see [Authenticate requests to Azure Cognitive Services][cognitive_auth]. | ||
annelo-msft marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| #### Get Subscription Key | ||
|
|
||
| You can obtain the endpoint and subscription key from the resource information in the [Azure Portal][azure_portal]. | ||
|
|
||
| Alternatively, you can use the [Azure CLI][azure_cli] snippet below to get the subscription key from the Form Recognizer resource. | ||
|
|
||
| ```PowerShell | ||
| az cognitiveservices account keys list --resource-group <your-resource-group-name> --name <your-resource-name> | ||
| ``` | ||
|
|
||
| #### Create CustomFormClient with Subscription Key Credential | ||
| Once you have the value for the subscription key, create a `FormRecognizerApiKeyCredential`. This will allow you to update the subscription key by using the `UpdateCredential` method without creating a new client. | ||
|
|
||
| With the value of the endpoint and a `FormRecognizerApiKeyCredential`, you can create the [CustomFormClient][formreco_custom_client_class]: | ||
|
|
||
| ```C# | ||
| string endpoint = "<endpoint>"; | ||
| string subscriptionKey = "<subscriptionKey>"; | ||
| var credential = new FormRecognizerApiKeyCredential(subscriptionKey); | ||
| var client = new CustomFormClient(new Uri(endpoint), credential); | ||
| ``` | ||
|
|
||
| <!-- #### Create CustomFormClient with Azure Active Directory Credential | ||
|
|
||
| Client subscription key authentication is used in most of the examples in this getting started guide, but you can also authenticate with Azure Active Directory using the [Azure Identity library][azure_identity]. Note that regional endpoints do not support AAD authentication. Create a [custom subdomain][custom_subdomain] for your resource in order to use this type of authentication. | ||
|
|
||
| To use the [DefaultAzureCredential][DefaultAzureCredential] provider shown below, | ||
| or other credential providers provided with the Azure SDK, please install the Azure.Identity package: | ||
|
|
||
| ```PowerShell | ||
| Install-Package Azure.Identity | ||
| ``` | ||
|
|
||
| You will also need to [register a new AAD application][register_aad_app] and [grant access][aad_grant_access] to Text Analytics by assigning the `"Cognitive Services User"` role to your service principal. | ||
|
|
||
| Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET. | ||
|
|
||
| ``` | ||
| string endpoint = "<endpoint>"; | ||
| var client = new TextAnalyticsClient(new Uri(endpoint), new DefaultAzureCredential()); | ||
| ``` --> | ||
|
|
||
| ## Key concepts | ||
|
|
||
| ### ReceiptClient | ||
| A `ReceiptClient` is the Form Recognizer interface to use for analyzing receipts. It provides operations to extract receipt field values and locations from receipts from the United States. | ||
|
|
||
| ### FormLayoutClient | ||
| A `FormLayoutClient` is the Form Recognizer interface to extract layout items from forms. It provides operations to extract table data and geometry. | ||
|
|
||
| ### CustomFormClient | ||
| A `CustomFormClient` is the Form Recognizer interface to use for creating, using, and managing custom machine-learned models. It provides operations for training models on forms you provide, and extracting field values and locations from your custom forms. It also provides operations for viewing and deleting models, as well as understanding how close you are to reaching subscription limits for the number of models you can train. | ||
|
|
||
| ### Long-Running Operations | ||
| Long-running operations are operations which consist of an initial request sent to the service to start an operation,followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has succeeded, to get the result. | ||
|
|
||
| Methods that train models or extract values from forms are modeled as long-running operations. The client exposes a `Start<operation-name>` method that returns an `Operation<T>`. Callers should wait for the operation to complete by calling `WaitForCompletionAsync()` on the operation returned from the `Start<operation-name>` method. A sample code snippet is provided to illustrate using long-running operations [below](#extracting-receipt-values-with-a-long-running-operation). | ||
|
|
||
| ### Training models | ||
| Using the `CustomFormClient`, you can train a machine-learned model on your own form type. The resulting model will be able to extract values from the types of forms it was trained on. | ||
|
|
||
| #### Training without labels | ||
| A model trained without labels uses unsupervised learning to understand the layout and relationships between field names and values in your forms. The learning algorithm clusters the training forms by type and learns what fields and tables are present in each form type. | ||
|
|
||
| This approach doesn't require manual data labeling or intensive coding and maintenance, and we recommend you try this method first when training custom models. | ||
|
|
||
| #### Training with labels | ||
| A model trained with labels uses supervised learning to extract values you specify by adding labels to your training forms. The learning algorithm uses a label file you provide to learn what fields are found at various locations in the form, and learns to extract just those values. | ||
|
|
||
| This approach can result in better-performing models, and those models can work with more complex form structures. | ||
|
|
||
| ### Extracting values from forms | ||
| Using the `CustomFormClient`, you can use your own trained models to extract field values and locations, as well as table data, from forms of the type you trained your models on. The output of models trained with and without labels differs as described below. | ||
|
|
||
| #### Using models trained without labels | ||
| Models trained without labels consider each form page to be a different form type. For example, if you train your model on 3-page forms, it will learn that these are three different types of forms. When you send a form to it for analysis, it will return a collection of three pages, where each page contains the field names, values, and locations, as well as table data, found on that page. | ||
|
|
||
| #### Using models trained with labels | ||
| Models trained with labels consider a form as a single unit. For example, if you train your model on 3-page forms with labels, it will learn to extract field values from the locations you've labeled across all pages in the form. If you sent a document containing two forms to it for analysis, it would return a collection of two forms, where each form contains the field names, values, and locations, as well as table data, found in that form. Fields and tables have page numbers to identify the pages where they were found. | ||
|
|
||
| ### Managing Custom Models | ||
| Using the `CustomFormClient`, you can get, list, and delete the custom models you've trained. You can also view the count of models you've trained and the maximum number of models your subscription will allow you to store. | ||
|
|
||
| ## Examples | ||
| The following section provides several code snippets illustrating common patterns used in the Form Recognizer .NET API. | ||
|
|
||
| ### Extracting receipt values with a long-running operation | ||
| ```C# | ||
annelo-msft marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| string endpoint = "<endpoint>"; | ||
| string subscriptionKey = "<subscriptionKey>"; | ||
| var credential = new FormRecognizerApiKeyCredential(subscriptionKey); | ||
| var client = new ReceiptClient(new Uri(endpoint), credential); | ||
|
|
||
| using (FileStream stream = new FileStream(@"C:\path\to\receipt.jpg", FileMode.Open)) | ||
| { | ||
| var extractReceiptOperation = client.StartExtractReceipts(stream, FormContentType.Jpeg); | ||
| await extractReceiptOperation.WaitForCompletionAsync(); | ||
| if (extractReceiptOperation.HasValue) | ||
| { | ||
| IReadOnlyList<ExtractedReceipt> result = extractReceiptOperation.Value; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Contributing | ||
|
|
||
| This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla]. | ||
|
|
||
| When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. | ||
|
|
||
| This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [[email protected]][coc_contact] with any additional questions or comments. | ||
|
|
||
|  | ||
|
|
||
|
|
||
| <!-- LINKS --> | ||
| [formreco_client_src]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/formrecognizer/Azure.AI.FormRecognizer/src | ||
| [formreco_docs]: https://docs.microsoft.com/en-us/azure/cognitive-services/form-recognizer/ | ||
| [formreco_refdocs]: https://aka.ms/azsdk-net-textanalytics-ref-docs | ||
| <!-- [formreco_nuget_package]: https://www.nuget.org/packages/Azure.AI.TextAnalytics --> | ||
| <!-- [formreco_samples]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/textanalytics/Azure.AI.TextAnalytics/tests/samples --> | ||
| [formreco_rest_api]: https://westus2.dev.cognitive.microsoft.com/docs/services/form-recognizer-api-v2-preview | ||
| [cognitive_resource]: https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account | ||
|
|
||
| <!-- [language_detection]: https://docs.microsoft.com/en-us/azure/cognitive-services/Text-Analytics/how-tos/text-analytics-how-to-language-detection | ||
| [sentiment_analysis]: https://docs.microsoft.com/en-us/azure/cognitive-services/Text-Analytics/how-tos/text-analytics-how-to-sentiment-analysis | ||
| [key_phrase_extraction]: https://docs.microsoft.com/en-us/azure/cognitive-services/Text-Analytics/how-tos/text-analytics-how-to-keyword-extraction | ||
| [named_entity_recognition]: https://docs.microsoft.com/en-us/azure/cognitive-services/Text-Analytics/how-tos/text-analytics-how-to-entity-linking --> | ||
|
|
||
|
|
||
| [formreco_custom_client_class]: src/CustomFormClient.cs | ||
| [azure_identity]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/identity/Azure.Identity | ||
| [cognitive_auth]: https://docs.microsoft.com/en-us/azure/cognitive-services/authentication | ||
| [register_aad_app]: https://docs.microsoft.com/azure/cognitive-services/authentication#assign-a-role-to-a-service-principal | ||
| [aad_grant_access]: https://docs.microsoft.com/azure/cognitive-services/authentication#assign-a-role-to-a-service-principal | ||
| [custom_subdomain]: https://docs.microsoft.com/azure/cognitive-services/authentication#create-a-resource-with-a-custom-subdomain | ||
| [DefaultAzureCredential]: ../../identity/Azure.Identity/README.md | ||
|
|
||
| <!-- [detect_language_sample0]: tests/samples/Sample1_DetectLanguage.cs | ||
| [detect_language_sample1]: tests/samples/Sample1_DetectLanguageBatchConvenience.cs | ||
| [detect_language_sample2]: tests/samples/Sample1_DetectLanguageBatch.cs | ||
| [detect_language_sample_async]: tests/samples/Sample1_DetectLanguageAsync.cs | ||
| [analyze_sentiment_sample0]: tests/samples/Sample2_AnalyzeSentiment.cs | ||
| [analyze_sentiment_sample1]: tests/samples/Sample2_AnalyzeSentimentBatchConvenience.cs | ||
| [analyze_sentiment_sample2]: tests/samples/Sample2_AnalyzeSentimentBatch.cs | ||
| [extract_key_phrases_sample0]: tests/samples/Sample3_ExtractKeyPhrases.cs | ||
| [extract_key_phrases_sample1]: tests/samples/Sample3_ExtractKeyPhrasesBatchConvenience.cs | ||
| [extract_key_phrases_sample2]: tests/samples/Sample3_ExtractKeyPhrasesBatch.cs | ||
| [recognize_entities_sample0]: tests/samples/Sample4_RecognizeEntities.cs | ||
| [recognize_entities_sample1]: tests/samples/Sample4_RecognizeEntitiesBatchConvenience.cs | ||
| [recognize_entities_sample2]: tests/samples/Sample4_RecognizeEntitiesBatch.cs | ||
| [recognize_entities_sample_async]: tests/samples/Sample4_RecognizeEntitiesAsync.cs | ||
| [recognize_pii_entities_sample0]: tests/samples/Sample5_RecognizePiiEntities.cs | ||
| [recognize_pii_entities_sample1]: tests/samples/Sample5_RecognizePiiEntitiesBatch.cs | ||
| [recognize_pii_entities_sample2]: tests/samples/Sample5_RecognizePiiEntitiesBatchConvenience.cs | ||
| [recognize_linked_entities_sample0]: tests/samples/Sample6_RecognizeLinkedEntities.cs | ||
| [recognize_linked_entities_sample1]: tests/samples/Sample6_RecognizeLinkedEntitiesBatch.cs | ||
| [recognize_linked_entities_sample2]: tests/samples/Sample6_RecognizeLinkedEntitiesBatchConvenience.cs --> | ||
|
|
||
| [azure_cli]: https://docs.microsoft.com/cli/azure | ||
| [azure_sub]: https://azure.microsoft.com/free/ | ||
| [nuget]: https://www.nuget.org/ | ||
| [azure_portal]: https://portal.azure.com | ||
|
|
||
| [cla]: https://cla.microsoft.com | ||
| [code_of_conduct]: https://opensource.microsoft.com/codeofconduct/ | ||
| [coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/ | ||
| [coc_contact]: mailto:[email protected] | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Complete README TBD: #10483