-
Notifications
You must be signed in to change notification settings - Fork 2k
Protocol Methods Quickstart with AutoRest
- Before you Start
- Generate SDK
- Build
- Install for Local Test
- Improve SDK Documentation
- Feature Checklist
- Review API Documentation
- Improve SDK Code
- Code Review
- Package Release
- Appendix
Java Azure SDK Design Guidelines is the overall design guideline of the data-plane SDK. Make sure you are familiar with concepts such as "Service Client" and "Packaging".
Make sure you are familiar with Git and Maven, especially the Build Lifecycle.
Please also take a look at Protocol Methods on the overview of the typical protocol methods from minimal data-plane client SDK.
You can choose from
- Python script, Java developer environment required
- Azure pipeline
to create your first data-plane client SDK.
☑️ If you have access to the Azure pipeline, it is recommended to use the pipeline.
If you already have SDK generated, but want to update it (for example, when swagger specification updated, or new api-version added), please see Follow-up code generation.
A Python script in Azure SDK for Java can help generate a minimal data-plane client SDK from OpenAPI 2.0 specification.
Please see Working with AutoRest for the installation of autorest
CLI.
It also requires the installation of Node.js 16 or above, Java 8 or above, and Python 3.
Python package PyYAML
and requests
is required to be installed, e.g. via pip3 install --upgrade PyYAML requests
.
If you are new to Java development environment, see Java Development Environment for details.
☑️ Required options from user:
-
spec-readme
: URL to README.md, which have OpenAPI 2.0 specification JSON asinput-file
in defaulttag
. It is recommended over theinput-file
option below, as the README.md potentially includes more configurations likesecurity
andtitle
. -
input-file
: URL to OpenAPI 2.0 specification JSON as input file. -
service
: Service name undersdk/
, sample:storage
. -
module
: Module name undersdk/<service>/
, sample:azure-storage-blob
.
Optional options:
-
security
: Credential types, sample:AADToken
for AAD credential for OAuth 2.0 authentication. See Authentication in AutoRest, and we recommend service to define them in OpenAPI. -
security-scopes
: OAuth 2.0 scopes, required if security includesAADToken
. -
title
: Client name. The name should always ends with "Client", sample:BlobClient
, which results inBlobServiceVersion
as service version class.
For service
, it is usually your service name in REST API specifications. For instance, if the JSON is under specification/storage
, the service would normally be storage
.
For module
, please refer to Namespace in Java Guideline. For example, if the namespace is com.azure.storage.blob
, the module should be azure-storage-blob
.
☑️ Namespace is decided by SDK arch board.
As an example,
python eng/mgmt/automation/generate_data.py --spec-readme https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/purview/data-plane/Azure.Analytics.Purview.Account/readme.md --service=purview --module=azure-analytics-purview-account
This will generate the SDK under sdk/purview/azure-analytics-purview-account
. And your follow-up work will mostly happen in this SDK folder.
Tips: The Python script also builds and verifies the generated project. It is possible that the verification fails due to Java code quality requirements. See Build section on how to fix.
There is an internal pipeline java - data-plane generator, which invokes the the Python script, generate the code, and create pull request of the generated SDK.
It takes similar input parameters as the script, e.g. SPEC_README
/INPUT_FILE
variable for spec-readme
/input-file
option, SERVICE
and MODULE
variable for service
and module
option.
A draft pull request will be created at pull requests, named as [Automation] Generate Data-plane SDK for <module>
.
Here is an example PR created by the pipeline: https://github.com/Azure/azure-sdk-for-java/pull/28336
An AutoRest configuration file is generated at sdk/<service>/<module>/swagger/readme.md
during the first-time code generation. You can now edit it, and generate code through this configuration file.
If you have not installed the AutoRest, please see Working with AutoRest for the installation of autorest
CLI.
It requires the installation of Node.js 16 or above.
As an example,
autorest --java --use=@autorest/java@latest sdk/<service>/<module>/swagger/readme.md
☑️ It is advised to use the latest version of @autorest/java
extension (see AutoRest.Java Release).
See AutoRest flags, and Java Data-Plane Client for details.
All possible Java flags can be found either from autorest --help --java
, or from AutoRest.Java readme.
See Building and Adding a Module.
See the contributing guide.
Here is the list of files that you are required to modify/confirm for your new module.
-
eng/versioning/version_client.txt
for versioning -
sdk/<service>/pom.xml
for project -
sdk/<service>/ci.yml
for CI
The Azure pipeline or Python script automatically created/modified these files for you. However, it is still advised that you double check them.
See Building.
Install SDK to local maven repository:
mvn install -f sdk/<service>/<module>/pom.xml -Dgpg.skip -Drevapi.skip -DskipTests
Tips: Javadoc might fail if other than Java 11. If you see error message like "Error while generating Javadoc", you can add option -Dmaven.javadoc.skip
to skip Javadoc.
See Maven Getting Start Guide, if you would like to create a client that uses your SDK.
See sdk/<service>/<module>/README.md
for adding the SDK to dependency.
There is many information about the SDK that AutoRest will never know. These can only be provided by a contributor.
A minimal README.md
is generated by AutoRest. Your first task will be improving it.
☑️ You would want to modify [product_documentation]
to point to your service documentation.
A blank Java sample file is generated as ReadmeSamples.java
, which is synchronized with your README (see Codesnippet Maven Plugin).
Add your sample code between // BEGIN: ...
and // END: ...
in ReadmeSamples.java
, and build the project, you will find your Examples section in README updated with the code.
A collection of generated Java sample files (based on x-ms-examples
in OpenAPI specs) is generated under <module>.generated
package. They might not be directly usable as runnable sample codes, however you should get the basic idea on how code works, and may even modify some to be runnable.
An example of adding samples is demonstrated in this commit.
A <client>TestBase
file and a collection of generated Java test files (based on x-ms-examples
in OpenAPI specs) is generated under <module>.generated
package.
See Adding Tests.
AutoRest supports OAuth2 and ApiKey authentication.
However, if service requires advanced solution (for example, JWT claim or HMAC-SHA256 authentication scheme), you are responsible to provide the implementation (usually as a subclass of HttpPipelinePolicy
), and customize the ClientBuilder
.
AutoRest.Java supports a default chained polling strategy for long-running operation.
It supports status monitor LRO and simple Location
-header-based LRO.
It does not support the outdated resource based LRO.
If service uses non-supported or non-standard LRO, you can contact Java SDK for guidance on the implementation of a specialized polling strategy, and customization in swagger/readme.md
.
AutoRest.Java supports application/octet-stream
, but it does not support multipart/form-data
. An issue in azure-core
tracks the status of the feature.
Generate Javadoc:
mvn javadoc:javadoc -f sdk/<service>/<module>/pom.xml
Review the generated API documentation sdk/<service>/<module>/target/site/apidocs/index.html
ClientBuilder
, Client
, AsyncClient
is the user-facing API surface that you might be interested to improve.
See Partial Update and Customization.
Code review happens on 2 places.
One is the usual GitHub pull request, for developers.
Another is the apiview, for architects. GitHub pull request should automatically trigger the apiview. If this does not happen, upload one manually:
- Build the SDK.
- Login apiview with GitHub account.
- "Create review" at top-left corner.
- "Browse" and upload
sdk/<service>/<module>/target/<module>-<version>-sources.jar
, and comment the result link in PR. If you did not find the source jar in step 4, runmvn source:jar
.
Comment @Azure/dpg-devs for awareness
in PR to loop in SDK developers for review.
See Release Checklist.
In particular, you need to follow "Prepare Release Script" and make another "prepare release" PR.
Before running the script, make sure your CHANGELOG.md is correct. Except for first release, it should contain following text on dependency updates.
### Other Changes
#### Dependency Updates
- Upgraded `azure-core` from `1.32.0` to version `1.33.0`.
- Upgraded `azure-core-http-netty` from `1.12.5` to version `1.12.6`.
After running the script, check the "release date" and make sure it is your scheduled release date.
After package release, one will see an "Increment versions for releases" PR. Please ping the PR to SDK developer to approve and merge.
Download Java, install.
Download Maven, unpack, set to PATH.
Download Node.js, install.
Install AutoRest with npm install -g autorest
in command line.
Optional for Git and Python:
sudo apt -y install openjdk-17-jdk
sudo apt -y install maven
curl -sL https://deb.nodesource.com/setup_16.x | sudo bash -
sudo apt -y install nodejs
npm install -g autorest
sudo apt -y install python3
sudo apt -y install python3-pip
pip3 install --upgrade wheel
pip3 install --upgrade PyYAML requests
Prepare the same AutoRest configuration at sdk/<service>/<module>/swagger/readme.java.md
.
In addition to the required flags, do add input-file
flag pointing to your OpenAPI specs.
Then use Use Python script for first-time code generation to generate the SDK.
Adding runnable tests requires both a good understanding of the service, and the knowledge of the client and test framework. Feel free to contact SDK developers, if you encountered issues on client or test framework.
Before testing the client, it is recommended that you prepare a Bicep or ARM template (or code, or CLI script, equivalent to the ARM template) that creates all required Azure resources and assigns necessary RBAC for your client. This ARM template will be very helpful for you to re-run the tests after version update, or for other developers to create their test environment.
If your service does not support connection string or API key authentication, it would likely use AzureAD OAuth2 token for authentication. A common solution is to provide an application and its service principal and to provide RBAC to the service principal for the access to the Azure resource of your service.
Client requires following 3 environment variables for the service principal using client ID/secret for authentication:
AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
On Linux, you can call export
:
export AZURE_TENANT_ID=<tenant_id>
export AZURE_CLIENT_ID=<client_id>
export AZURE_CLIENT_SECRET=<client_secret>
On Windows, it is set
.
Live tests require one more environment variable:
export AZURE_TEST_MODE=LIVE
AutoRest.Java already provided you the generated Java test files at sdk/<service>/<module>/src/test/java/<namespace>/generated
folder, and a <client>TestBase
class.
You should double check the beforeTest()
method in <client>TestBase
class, and modify it if necessary.
It is recommended to copy the test files you would like to modify and use to sdk/<service>/<module>/src/test/java/<namespace>
folder.
It is common for you to put configuration from your test environment to the code, as long as they are not a secret.
Alternatively, you can keep rely on Configuration.getGlobalConfiguration().get("<env_variable>")
and define a few additional environment variables.
After you settle down the builder
, you can decide which REST APIs you want to test, and write the code and verification in the testClient()
method.
The generated samples under <module>.generated
package can be used as reference for your test code.
After you finish the test code, call mvn test
under sdk/<service>/<module>
to execute the tests.
If everything works fine, you will see following log.
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.776 s - in com.azure.messaging.webpubsub.ClientTests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
An example of adding live tests is demonstrated in this commit.
The live tests only run when environment variable AZURE_TEST_MODE
is set to LIVE
, and hence it is usually run manually.
Next step, you will convert the live tests to record/playback tests.
TestBase.setPlaybackSyncPollerPollInterval
on the SyncPoller
instance, if your test involves long-running operation API.
Firstly, the tests need to be recorded.
Remove @DoNotRecord(skipInPlayback = true)
annotation in ClientTests.java
.
And set environment variable AZURE_TEST_MODE
to RECORD
:
export AZURE_TEST_MODE=RECORD
Then call mvn test
as if this is live tests.
Please follow this guide on updating the test recordings.
☑️ Permission is required for updating the asserts repository. If you failed in test-proxy push
, please join a partner write team.
These recorded tests can now be run in playback mode automatically (it can be run locally via the same mvn test
call, by setting environment variable AZURE_TEST_MODE
to PLAYBACK
).
Be aware that in playback mode during continuous integration, there is no test environment nor environment variables available. Hence if you rely on Configuration.getGlobalConfiguration().get("<env_variable>")
, make sure you provide a second parameter as the default value, for the case when <env_variable>
is not available.
Modify pom.xml to enable test coverage.
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <jacoco.skip>true</jacoco.skip>
</properties>
At beta, one can use a lower minimum coverage.
Since autorest/[email protected]
and with partial-update
option, you can in-place modify the generated code, and AutoRest.Java will keep the hand-written code intact when it re-generates the SDK.
The user-facing API is of pattern:
FooClient client = new FooClientBuilder()
.endpoint(...)
.credential(new DefaultAzureCredentialBuilder().build())
.buildClient();
FooAsyncClient asyncClient = new FooClientBuilder()
.endpoint(...)
.credential(new DefaultAzureCredentialBuilder().build())
.buildAsyncClient();
And the classes involved is FooClient
, FooAsyncClient
, and FooClientBuilder
.
All of these classes can be customized in-place, to enhance the usability of the SDK, or to add new feature.
You can add new class variables and new class methods to these classes, and with partial-update
option in swagger/README.md
, new code generated via AutoRest will not override your change.
For modifying the signature or the implementation of a class method, please remove the @Generated
annotation, so AutoRest.Java can recognize that you have modified the method.
If you would like to remove a class method, simply make the method package private, so it is not exposed to customer. Also you need to remove the @Generated
annotation.
It is advised that you keep the modifications simple.
If certain feature requires complicated code, it is better to put the implementation code in some other classes within implementation
package, and only add the public class method and invocation in these Client
or ClientBuilder
classes.
Check "Build" and "Analyze" first, the "Test"s.
It helps to run mvn clean verify
locally, before checking CI failure.
See the contributing guide on versioning.
TLDR: run python eng/versioning/update_versions.py --ut library --bt client --sr
Fix spelling in code or in markdown.
Suppression at .vscode/cspell.json
.
Fix corresponding files.
Fix code for SpotBugs, Checkstyle, and Javadoc.
If you run it locally, make sure your build tool is re-installed.
Avoid breaking changes for RevApi.
Suppression at eng/code-quality-reports/src/main/resources
.
Besides test failure, it could also be caused by insufficient test coverage.
[WARNING] Rule violated for bundle azure-ai-anomalydetector: lines covered ratio is 0.21, but expected minimum is 0.40
[WARNING] Rule violated for bundle azure-ai-anomalydetector: branches covered ratio is 0.27, but expected minimum is 0.30
For beta SDK, you are allowed to lower the minimum coverage, by adding following Jacoco settings in pom.xml.
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <jacoco.min.linecoverage>0.2</jacoco.min.linecoverage>
+ <jacoco.min.branchcoverage>0.2</jacoco.min.branchcoverage>
</properties>
For GA SDK, it is expected to follow the default coverage, which is 0.40 for line coverage, 0.30 for branch coverage.
- Frequently Asked Questions
- Azure Identity Examples
- Configuration
- Performance Tuning
- Android Support
- Unit Testing
- Test Proxy Migration
- Azure Json Migration
- New Checkstyle and Spotbugs pattern migration
- Protocol Methods
- TypeSpec-Java Quickstart