SDK and ScheduledJobs (JobResponse) - Different results from devices.jobs query

[ahernandezdelta]

It's not clear to me (and it doesn't seem documented anywhere) how using the JobsClient in the SDK (IotHubServiceClient.ScheduledJobs in V2 of the SDK) is supposed to be different or identical to using the REST API or the SDK Query API.

For instance, if I run IotHubServiceClient.ScheduledJobs.CreateQueryAsync() I get a set of results that's different from the interactive query in IoT Hub (SELECT * FROM devices.jobs) with no filter criteria. When I use IotHubServiceClient.Query.CreateAsync() the results are identical to the interactive query. It's not at all clear where the QueryResponse<ScheduledJob> (formerly JobResponse) is coming from. My understanding is the SDK is just a pass-thru to the REST API and that they should be referring to the same data:

https://learn.microsoft.com/en-us/rest/api/iothub/service/jobs/query-scheduled-jobs

I am currently using the V2 preview version of the SDK:

https://github.com/Azure/azure-iot-sdk-csharp/blob/previews/v2/SDK%20v2%20migration%20guide.md

As a side note as feedback for the V2 preview SDK, it's weird that the new client is called ScheduledJobs and not just Jobs to be consistent.

[timtay-microsoft]

My understanding is the SDK is just a pass-thru to the REST API and that they should be referring to the same data

Your understanding is correct here. The different sets of results you get from the different service client APIs is a result of the service client calling different IoT Hub service APIs.

IotHubServiceClient.ScheduledJobs.CreateQueryAsync() is identical to IotHubServiceClient.Query.CreateJobsQueryAsync() as they are both a GET on /jobs/v2/query with some filters on job type and job state.

IotHubServiceClient.Query.CreateAsync() with SELECT * FROM devices.jobs is actually a POST on /devices/query

Because these are different service requests, IoT hub has slightly different implementations on what it returns. I agree that the service returning different data from these seemingly identical requests is confusing, but it isn't something the SDK can control unfortunately.

As a side note as feedback for the V2 preview SDK, it's weird that the new client is called ScheduledJobs and not just Jobs to be consistent.

We appreciate the feedback! We decided on the name "ScheduledJobs" to make clear that the client does not handle all types of jobs. Import/export jobs in particular are wildly different, so we wanted to make these distinct. Does that help contextualize the name choice at all?

[ahernandezdelta]

I agree that the service returning different data from these seemingly identical requests is confusing, but it isn't something the SDK can control unfortunately.

If you're designing the SDK how is it you can say the SDK can't control this? For instance, why can't CreateJobsQueryAsync() just be a passthrough to POST /devices/query or Query.CreateAsync()?

Regardless, what are developers supposed to do with this information? This isn't documented anywhere as far as I can tell and it doesn't help clarify what I'm expected to do as as solutions developer when I'm building a back-end solution for managing jobs. At the end of the day I posted this discussion/question because the data are completely different. I expect to see the same # of records from the above calls and I don't. Is there a standard explanation for this or does it require in-depth troubleshooting? As far as I can tell, one shows the top-level job and the other shows the results of the job as it was run against each device. But the documentation is not at all clear if this is the case. I would expect there to be a section that clarifies: "If you want to manage the job itself, use X. If you want to see the results of the job as it was run against each device, use Y". Also, it wouldn't be clear why there are different ways of getting to this information.

We appreciate the feedback! We decided on the name "ScheduledJobs" to make clear that the client does not handle all types of jobs. Import/export jobs in particular are wildly different, so we wanted to make these distinct. Does that help contextualize the name choice at all?

Sure, but Import/Export jobs seem to be the exception here and you've already moved those to the new *.Devices client. Why not be consistent and either have them both on the *.Devices client or have them both on a *.Job client with sub-clients for import/export/scheduled jobs or better yet just call it a jobs client and have clearly named methods? It's just really odd that they're still spread out so it doesn't really clarify the confusion from the old SDK when I had to know whether I needed the RegistryManager, ServiceClient, or DeviceClient.

[timtay-microsoft]

If you're designing the SDK how is it you can say the SDK can't control this? For instance, why can't CreateJobsQueryAsync() just be a passthrough to POST /devices/query or Query.CreateAsync()?

The SDK's job is to expose all the service APIs through a C# package, so we don't want to hide certain APIs (such as GET on /jobs/v2/query) even if it would make for a cleaner SDK overall. Especially since we don't know how many users preferred one of these service APIs over the other in v1.

Regardless, what are developers supposed to do with this information? This isn't documented anywhere as far as I can tell and it doesn't help clarify what I'm expected to do as solutions developer when I'm building a back-end solution for managing jobs.

I understand your frustration, and we'll take it upon ourselves to improve the documentation around these query APIs to help users in the future.

Sure, but Import/Export jobs seem to be the exception here and you've already moved those to the new *.Devices client. Why not be consistent and either have them both on the *.Devices client or have them both on a *.Job client with sub-clients for import/export/scheduled jobs or better yet just call it a jobs client and have clearly named methods?

We've had a lot of confusion from users in the past stemming from having import/export jobs on the same client as scheduled jobs. Users would get confused when they failed to cancel an import/export job without realizing the API they called was specifically for scheduled jobs.

It's just really odd that they're still spread out so it doesn't really clarify the confusion from the old SDK when I had to know whether I needed the RegistryManager, ServiceClient, or DeviceClient.

The subclient structure we went with in v2 was done so that you wouldn't need to instantiate so many clients, and to break apart the behemoth that was the registry manager. Do you find navigating the subclients more painful than instantiating and owning multiple clients yourself?

//v2
IotHubServiceClientOptions options = new(...);
IotHubServiceClient serviceClient = new(..., options);
serviceClient.Devices.AddAsync(...);
serviceClient.ScheduledJobs.ScheduleTwinUpdateAsync(...);

//v1
RegistryManagerOptions rmOptions = new(...);
RegistryManager registryManager = new(..., rmOptions);
registryManager.AddDeviceAsync(...);

JobClientOptions jcOptions = new(...);
JobClient jobClient = new JobClient(..., jcOptions);
jobClient.ScheduleTwinUpdateAsync(...);

[ahernandezdelta]

The pain was never having to manage multiple clients - I would expect that behavior as a natural part of Open/Closed and SoC design patterns. What was a problem with the old design was the fact that we had to manage multiple libraries and namespaces and with that different patterns and ways of constructing them. I would still expect there to either be different clients for managing different types of jobs (or for there to be a separate client for import/export functionality) but for their implementation to be seamless and housed within a cleaner namespace/packaging scheme.