Skip to content

[translation] poller design #19041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kristapratico merged 8 commits into from
Jun 3, 2021
Merged

[translation] poller design #19041

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7f187e5
add polling implementation
kristapratico Jun 2, 2021
7003ea4
add polling generation transforms and regenerate
kristapratico Jun 2, 2021
c50e949
add tests and recordings
kristapratico Jun 2, 2021
9016aff
samples, readme, changelog
kristapratico Jun 2, 2021
2ae44df
update changelog
kristapratico Jun 2, 2021
f175eda
pylint
kristapratico Jun 2, 2021
6181411
some docs updates
kristapratico Jun 2, 2021
b8bc3de
missed an await
kristapratico Jun 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions eng/.docsettings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ known_content_issues:
- ['sdk/monitor/azure-monitor-opentelemetry-exporter/README.md', '#4554']
- ['sdk/digitaltwins/azure-digitaltwins-core/swagger/README.md', '#4554']
- ['sdk/textanalytics/azure-ai-textanalytics/swagger/README.md', '#4554']
- ['sdk/translation/azure-ai-translation-document/swagger/README.md', '#4554']
- ['sdk/purview/azure-purview-catalog/swagger/README.md',  '#4554']
- ['sdk/purview/azure-purview-scanning/swagger/README.md',  '#4554']
- ['sdk/agrifood/azure-agrifood-farming/swagger/README.md',  '#4554']
Expand Down
11 changes: 11 additions & 0 deletions sdk/translation/azure-ai-translation-document/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@

This version of the SDK defaults to the latest supported service version, which currently is v1.0

**Breaking changes**

- `create_translation_job` was renamed to `begin_translation` and now returns a `DocumentTranslationPoller`
Comment thread
kristapratico marked this conversation as resolved.
Outdated
(or `AsyncDocumentTranslationPoller`) to begin the long-running operation. Call `.result()` on the poller object
Comment thread
kristapratico marked this conversation as resolved.
Outdated
to wait until the translation is complete. See the README for more information about LROs.
Comment thread
kristapratico marked this conversation as resolved.
Outdated
- Upon completion of the LRO, `begin_translation` now returns a pageable of `DocumentStatusResult`. All job-level metadata can still
be found on `poller.details`.
- `has_completed` has been removed from `JobStatusResult` and `DocumentStatusResult`. Use `poller.done()` to check if the

@maririos maririos Jun 2, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhh interesting. So if a user wants to know if a specific document is done, they will have to check the status of the document and "calculate it" or just wait until the poller finishes, right?
why?

@kristapratico kristapratico Jun 2, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

People found this confusing in the user study - they thought it was a dynamic property that would update the status in python. Moving to the poller allows us to actually have that dynamic property with done(). I feel it's less important to have on DocumentStatusResult, at that point you're actually interested in checking what the status is - "failed", "succeeded" etc to see what to do (at least that's how all our samples are written). Checking with johan on this one.

@kristapratico kristapratico Jun 2, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Johan said to remove it

translation has completed.
- Client method `wait_until_done` has been removed. Use `poller.result()` to wait for the LRO to complete.

**New features**

- Authentication using `azure-identity` credentials now supported.
Expand Down
112 changes: 42 additions & 70 deletions sdk/translation/azure-ai-translation-document/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Azure Cognitive Services Document Translation is a cloud service that translates
and dialects while preserving document structure and data format. Use the client library for Document Translation to:

* Translate numerous, large files from an Azure Blob Storage container to a target container in your language of choice.
* Check the translation status and progress of each document in the translation job.
* Check the translation status and progress of each document in the translation operation.
* Apply a custom translation model or glossaries to tailor translation to your specific case.

[Source code][python-dt-src] | [Package (PyPI)][python-dt-pypi] | [API reference documentation][python-dt-ref-docs] | [Product documentation][python-dt-product-docs] | [Samples][python-dt-samples]
Expand Down Expand Up @@ -130,14 +130,14 @@ the service documentation:
Interaction with the Document Translation client library begins with an instance of the `DocumentTranslationClient`.
The client provides operations for:

- Creating a translation job to translate documents in your source container(s) and write results to you target container(s).
- Checking the status of individual documents in the translation job and monitoring each document's progress.
- Enumerating all past and current translation jobs with the option to wait until the job(s) finish.
- Creating a translation operation to translate documents in your source container(s) and write results to you target container(s).
- Checking the status of individual documents in the translation operation and monitoring each document's progress.
- Enumerating all past and current translations operations.
- Identifying supported glossary and document formats.

### Translation Input

To create a translation job, pass a list of `DocumentTranslationInput` into the `create_translation_job` client method.
To begin translating your documents, pass a list of `DocumentTranslationInput` into the `begin_translation` client method.
Constructing a `DocumentTranslationInput` requires that you pass the SAS URLs to your source and target containers (or files)
and the target language(s) for translation.

Expand Down Expand Up @@ -191,20 +191,22 @@ my_input = [

See the service documentation for all [supported languages][supported_languages].

### Return value
### Long-Running Operations
Long-running operations are operations which consist of an initial request sent to the service to start an operation,
followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has
succeeded, to get the result.

There are primarily two types of return values when checking on the result of a translation job - `JobStatusResult` and `DocumentStatusResult`.
* A `JobStatusResult` will contain the details of the entire job, such as it's status, ID, any errors, and status summaries of the documents in the job.
* A `DocumentStatusResult` will contain the details of an individual document, such as it's status, translation progress, any errors,
and the URLs to the source document and translated document.
Methods that translate documents are modeled as long-running operations.
The client exposes a `begin_<method-name>` method that returns an `DocumentTranslationPoller` or `AsyncDocumentTranslationPoller`. Callers should wait
Comment thread
kristapratico marked this conversation as resolved.
Outdated
for the operation to complete by calling `result()` on the poller object returned from the `begin_<method-name>` method.
Sample code snippets are provided to illustrate using long-running operations [below](#examples "Examples").

## Examples

The following section provides several code snippets covering some of the most common Document Translation tasks, including:

* [Translate your documents](#translate-your-documents "Translate Your Documents")
* [Check status on individual documents](#check-status-on-individual-documents "Check Status On Individual Documents")
Comment thread
maririos marked this conversation as resolved.
* [List translation jobs](#list-translation-jobs "List Translation Jobs")
* [List translation operations](#list-translation-operations "List Translation Operations")

### Translate your documents
Translate the documents in your source container to the target containers.
Expand All @@ -221,7 +223,7 @@ target_container_sas_url_fr = "<sas-url-fr>"

document_translation_client = DocumentTranslationClient(endpoint, credential)

job = document_translation_client.create_translation_job(
poller = document_translation_client.begin_translation(
[
DocumentTranslationInput(
source_url=source_container_sas_url_en,
Expand All @@ -231,59 +233,32 @@ job = document_translation_client.create_translation_job(
],
)
]
) # type: JobStatusResult

job_result = document_translation_client.wait_until_done(job.id) # type: JobStatusResult

print("Job created on: {}".format(job_result.created_on))
print("Job last updated on: {}".format(job_result.last_updated_on))
print("Total number of translations on documents: {}".format(job_result.documents_total_count))

print("Of total documents...")
print("{} failed".format(job_result.documents_failed_count))
print("{} succeeded".format(job_result.documents_succeeded_count))

if job_result.status == "Succeeded":
print("Our translation job succeeded")

if job_result.status == "Failed":
print("All documents failed in the translation job")

# check document statuses... see next sample
```

### Check status on individual documents
Check status and translation progress of each document under a job.

```python
from azure.core.credentials import AzureKeyCredential
from azure.ai.translation.document import DocumentTranslationClient

endpoint = "https://<resource-name>.cognitiveservices.azure.com/"
credential = AzureKeyCredential("<api_key>")
job_id = "<job-id>"

document_translation_client = DocumentTranslationClient(endpoint, credential)
)

documents = document_translation_client.list_all_document_statuses(job_id) # type: ItemPaged[DocumentStatusResult]

for doc in documents:
if doc.status == "Succeeded":
print("Document at {} was translated to {} language".format(
doc.translated_document_url, doc.translate_to
))
if doc.status == "Running":
print("Document ID: {}, translation progress is {} percent".format(
doc.id, doc.translation_progress*100
))
if doc.status == "Failed":
print("Document ID: {}, Error Code: {}, Message: {}".format(
doc.id, doc.error.code, doc.error.message
))
result = poller.result()

print("Status: {}".format(poller.status()))
print("Created on: {}".format(poller.details.created_on))
print("Last updated on: {}".format(poller.details.last_updated_on))
print("Total number of translations on documents: {}".format(poller.details.documents_total_count))

print("\nOf total documents...")
print("{} failed".format(poller.details.documents_failed_count))
print("{} succeeded".format(poller.details.documents_succeeded_count))

for document in result:
print("Document ID: {}".format(document.id))
print("Document status: {}".format(document.status))
if document.status == "Succeeded":
print("Source document location: {}".format(document.source_document_url))
print("Translated document location: {}".format(document.translated_document_url))
print("Translated to language: {}\n".format(document.translate_to))
else:
print("Error Code: {}, Message: {}\n".format(document.error.code, document.error.message))
```

### List translation jobs
Enumerate over the translation jobs submitted for the resource.
### List translation operations
Enumerate over the translation operations submitted for the resource.

```python
from azure.core.credentials import AzureKeyCredential
Expand All @@ -297,9 +272,6 @@ document_translation_client = DocumentTranslationClient(endpoint, credential)
jobs = document_translation_client.list_submitted_jobs() # type: ItemPaged[JobStatusResult]

for job in jobs:
if not job.has_completed:
Comment thread
maririos marked this conversation as resolved.
job = document_translation_client.wait_until_done(job.id)

print("Job ID: {}".format(job.id))
print("Job status: {}".format(job.status))
print("Job created on: {}".format(job.created_on))
Expand Down Expand Up @@ -348,7 +320,7 @@ The following section provides several code snippets illustrating common pattern
These code samples show common scenario operations with the Azure Document Translation client library.

* Client authentication: [sample_authentication.py][sample_authentication]
* Create a translation job: [sample_create_translation_job.py][sample_create_translation_job]
* Begin translating documents: [sample_begin_translation.py][sample_begin_translation]
* Check the status of documents: [sample_check_document_statuses.py][sample_check_document_statuses]
* List all submitted translation jobs: [sample_list_all_submitted_jobs.py][sample_list_all_submitted_jobs]
* Apply a custom glossary to translation: [sample_translation_with_glossaries.py][sample_translation_with_glossaries]
Expand All @@ -361,7 +333,7 @@ first install an async transport, such as [aiohttp](https://pypi.org/project/aio
are found under the `azure.ai.translation.document.aio` namespace.

* Client authentication: [sample_authentication_async.py][sample_authentication_async]
* Create a translation job: [sample_create_translation_job_async.py][sample_create_translation_job_async]
* Begin translating documents: [sample_begin_translation_async.py][sample_begin_translation_async]
* Check the status of documents: [sample_check_document_statuses_async.py][sample_check_document_statuses_async]
* List all submitted translation jobs: [sample_list_all_submitted_jobs_async.py][sample_list_all_submitted_jobs_async]
* Apply a custom glossary to translation: [sample_translation_with_glossaries_async.py][sample_translation_with_glossaries_async]
Expand Down Expand Up @@ -416,8 +388,8 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con

[sample_authentication]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_authentication.py
[sample_authentication_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_authentication_async.py
[sample_create_translation_job]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_create_translation_job.py
[sample_create_translation_job_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_create_translation_job_async.py
[sample_begin_translation]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_begin_translation.py
[sample_begin_translation_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_begin_translation_async.py
[sample_check_document_statuses]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_check_document_statuses.py
[sample_check_document_statuses_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_check_document_statuses_async.py
[sample_list_all_submitted_jobs]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_list_all_submitted_jobs.py
Expand Down
2 changes: 2 additions & 0 deletions sdk/translation/azure-ai-translation-document/azure/ai/translation/document/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
StorageInputType,
)
from ._api_version import DocumentTranslationApiVersion
from ._polling import DocumentTranslationPoller
from ._models import (
TranslationTarget,
JobStatusResult,
Expand All @@ -34,4 +35,5 @@
"JobStatusResult",
"DocumentStatusResult",
"DocumentTranslationError",
"DocumentTranslationPoller"
]
Loading