Microsoft Azure IoT SDKs for .NET v2 Preview Release 2022-10-14

[drwill-ms]

This is the first preview of the v2 clients. You can read about the changes in this migration guide.

---

This discussion was created from the release Microsoft Azure IoT SDKs for .NET v2 Preview Release 2022-10-14.

[drwill-ms]

Web documentation for API surface can be found at https://learn.microsoft.com/dotnet/api/microsoft.azure.devices.client?view=azure-dotnet-preview.

[drwill-ms]

I wanted to share what our goals are with shipping a v2 of our .NET SDKs, because with major version update comes breaking changes, and as that comes at a cost to customers who currently use the current SDKs, we want you to understand the 'why' of these changes.

Preface

First, most of the changes come as a direct result of feedback from customers, namely when we're handling GitHub issues and live site incidents. We've even had hour-long meetings with customers to hear from them about their struggles and complaints. We appreciate the time customers take to share what didn't work for them and how they wish the SDKs worked. We also lament the trouble customers have and how that sets them back. In other words, we didn't make these changes for no reason; they are based on the usage and feedback of the community.

Secondly, we've taken the lessons learned in this SDK, and in others in Azure, and applied them across the entire API surface. If the client library APIs look or work different in various areas or across different clients, it can confuse and befuddle. Any problem that causes hiccups makes using the SDK unpleasant and frustrating. We want to avoid that.

Finally, one approach we've taken is in looking at our most thorough SDK sample called Device reconnection sample and questioned anything required by the user that we thought was not obvious or a pain to write, and worked to make the client library easier for that area.

An overview of changes

Our primary goal is that the API is intuitive and easy to use. We know that isn't always possible because APIs stand in front of concepts which aren't always evident to a newer user. In those cases, we rely on informative documentation comments. We've beefed up the doc comments with extra information, links to helpful web topics, and declared which exceptions one can expect to see and when. Still on our to-do list is to add code examples as well. Since doc comments get translated into web documentation, our aim is not only enable an exceptional in-Visual Studio experience but also to make it easy to search for and share that documentation.

One of the broadest problems affecting the SDKs are shared libraries. Some are implicit dependencies and not too awkward to use, but others one has to know about in order to get the device app to work correctly. Having to know about the existence of is a barrier to entry and generally requires someone to read docs or samples. This has been eliminated. Each library has what it needs to operate, and all the types a user should need should be in the base namespace. We've gone from 9 libraries to 4. Another problem with the shared library was that it had super types that met all the requirements of all libraries which meant for any given scenario it had more members that are needed. That causes confusion (do I need this?) and problems (I set this but nothing changed!). Now each library has its own copy of the formerly shared objects, trimmed down to only what is supported.

We added a lot of name disambiguation. When using libraries together (namely DPS and Hub device clients), we want to avoid requiring using a full namespace to disambiguate. This may make some type names longer and arguably more than a mouthful, but we believe that clarity and lack of naming conflicts is better than the alternative. Even though we try, we don't always pick the best names and if you have suggestions on how to name something better, please let us know.

Client objects weren't always obviously named, and especially on the Hub service SDK side, it was painful having to initialize so many different kinds of clients, let alone finding the right client for the operation you wanted. That has all been simplified. All clients end with the suffix "Client" and are aptly named. The Hub and DPS service clients are now a single client each with aptly named sub-clients to make discovery easier. Additionally, all clients can be created using a constructor; factory methods are mostly gone. This enables easier discovery of client initialization as well as unit test mocking.

The clients have many dependencies, some of which can throw exceptions during an operation. We've strived to cut down on the number of exception types that can be thrown. While you'll still see various exceptions one would expect to see when a method's parameters are evaluated (i.e., ArgumentException (and its derivatives like ArgumentNullException and ArgumentOutOfRangeException), FormatException for when the contents of an argument don't work (e.g., a connection string's key/value pairs), InvalidOperationException when the state of the client is invalid for a given operation, ObjectDisposedException when an operation is started on a disposed object, and OperationCanceledException when the provided cancelation token has been signaled and the operation ends prematurely. Finally, any operation that fails due to an error reported from the service or fails due to network issues, will result in a single exception type (per client library) with the type of error being distinguishable by several properties, for example, IsTransient and ErrorCode, that an app can use to determine how to react to the error, with a human readable exception message to provide more information.

Another big change is in the IoT hub device client for operations that start from the service: direct methods, twin property updates, and cloud-to-device messaging. These have been a mix of callback and async methods. For callbacks, the return type has been a specific value or void, where a user must know what action to take for a properly behaving device application. We've consolidated approaches choosing the callback approach, as we find it requires less code and makes the programming model clearer, for example, one doesn't have to set up a loop or catch exceptions for the async call(s). Also, we've updated the callback definition to include the information expected to be returned, so it is evident to the device app developer what should be done to complete that operation.

An example of making the client better by looking at our samples, is in staying up to date on the connection state. The device client has a callback that includes the new state and a reason for that state change. A device app would need to cache that for the rest of the app to make use of it. Now that information is cached for you on the client. Secondly, the action (or inaction) suggested for you was commented in the sample in a nested switch statement, but that required reading the sample to know what to do. Now that advice is embedded in the client as an enum called RecommendedAction. You can choose to follow the recommendation, or break with it when that makes sense for your scenario.

Other changes

I could go on about all the other changes, but we have all this documented in our [migration guide](../../blob/previews/v2/SDK v2 migration guide.md). In that guide, you'll find lists of breaking changes and additions. There is also a mapping table of former names to new names, for when you update the SDK reference to v2 and see a variety of build errors you can simply search that table and find the replacement.

Have you found a change we didn't document? Please let us know, or feel free to submit a PR to change the migration guide.

Conclusion

We hope you find the v2 SDK API surface easier to understand and use. We're still taking feedback and considering changes while we're still in preview. Once we close on this API surface and release a non-preview version, we won't make breaking changes again.

[sandervandevelde]

Path to [migration guide](../../blob/previews/v2/SDK v2 migration guide.md) is not a link. Please fix.

Should point to https://github.com/Azure/azure-iot-sdk-csharp/blob/previews/v2/SDK%20v2%20migration%20guide.md

[drwill-ms]

Fixed.

[ahernandezdelta]

How should we provide feedback or ask questions on the preview SDK?

[drwill-ms]

GitHub issues are good. More comments in this discussion is also fine. It probably depends on how much discussion you expect to have; if a lot, then better segregated in an issue instead.

[ahernandezdelta]

I created a separate discussion over here: #3075

I cannot tell yet if this is an issue with the V2 SDK. I will research some more and compare with the older SDK.

[drwill-ms]

Another discussion is just fine. I realize we get pretty good threading in this discussion too, so here is also acceptable. We're not too picky - just pleased to be getting your feedback. :)

[ahernandezdelta]

I'm confused about why the updated jobs client is called ScheduledJobs. It implies there is a concept of a normal "job" and one that is "scheduled". It's also confusing when thinking in terms of what the old SDK used to refer to: JobClient, GetJobAsync(), JobProperties, etc. Also, the REST API documentation refers to it being a "Scheduled Job" but the resource endpoint itself is /jobs. It's also not really clear whether or not there is a separate construct of a job that's already completed (i.e., it's not longer "scheduled"). Is that result set also a ScheduledJob? That would not be intuitive.

These kinds of inconsistencies are littered throughout Azure SDKs and APIs and is partly what made the original SDK confusing.

Also, what is the rationale behind using a new construct QueryResponse<T>? Why not just re-use existing constructs we see elsewhere in Azure SDKs or keep IEnumerable<T> or IAsyncEnumerable?

[drwill-ms]

Hi @ahernandezdelta -

Thank you for taking the time to look at our v2 API and leave a comment. We seriously appreciate it!

Jobs

There are jobs that are scheduled and jobs you run on demand. The way the service exposed them via REST endpoints doesn't make it less confusing. The jobs run on demand are in the /jobs REST API path, and the scheduled jobs are in the /jobs/v2 REST API path. Are the latter a version 2 of the first? No, they aren't.

Scheduled jobs include:

• Device method
• Update twin

This used to be available on the JobsClient and in v2 is available on IotHubServiceClient.ScheduledJobs to remove the ambiguity of it not being a client for all jobs.

On demand jobs include:

• Import devices (and modules and configurations)
• Export devices (and modules and configurations)

This used to be available on the RegistryManager client and in v2 is available on IotHubServiceClient.Devices as ImportAsync(...) and .ExportAsync(...). We felt there was more affinity with the operations in IotHubServiceClient.Devices as it is another way to add/remove/update devices - the only difference is how bulk it is as the normal bulk API limits one to 100 devices.

It wasn't 100% clear to us how to group them up. We could have moved import/export devices job over to the jobs client and kept the name, but aside from being jobs they do have little affinity. If you have feedback to share on how you think it would most make sense to group them, we're happy to hear it.

Query

I don't remember why we chose to use QueryResponse<T>. We've worked in track2 where IAsyncEnumerable is the norm. Perhaps we weren't aware it was available outside of track2. I'll check with our dev who redesigned that piece and see if he recalls any obstacles we encountered that influenced that decision.

Regards,
David

[drwill-ms]

Ah, IAsyncEnumerable is defined in .NET Standard 2.1 and we're bound to .NET Standard 2.0 (by guideline), but that doesn't mean we can't bring in a special NuGet package to get it. We'll look into it!

[timtay-microsoft]

Also, what is the rationale behind using a new construct QueryResponse? Why not just re-use existing constructs we see elsewhere in Azure SDKs or keep IEnumerable or IAsyncEnumerable?

We've made it so the query APIs in both the IoT hub and DPS service client now return AsyncPageable as of #3176 FYI. Thanks for the feedback on this point

[drwill-ms]

Fourth release of v2 preview is out: https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2023-03-13-v2preview

[gagwithgaffer]

V2 TPM Sample not yet available? whilst migrating over to V2 SDK, am getting squigglys underneath DeviceAuthenticationWithTpm

Tried updating all the NuGets, then also tried changing DeviceAuthenticationWithTpm --> ClientAuthenticationWithTpm but still get the squigglys.

Are there any plans to support TPM in the near future? I wasnt able to find a TPM sample within the Preview-005 Solution.

[drwill-ms]

Correct, TPM support is gone from v2 and is unlikely to return. Instead, we recommend using certificate authentication for the best security, followed by SAS auth.

https://github.com/Azure/azure-iot-sdk-csharp/blob/previews/v2/SDK%20v2%20migration%20guide.md#dps-device-client

[drwill-ms]

Fifth preview is out: https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2023-04-26-v2preview

[tmahmood-microsoft]

We've written apps to rapidly exercise the SDK over long periods of time to ensure stability, measure sustained performance, and identify memory issues. You can find the V2 long haul test results here.