Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MQTT Device is auto rejecting Cloud to Device messages for first time #98930

Closed
willyt150 opened this issue Sep 23, 2022 · 14 comments
Closed

MQTT Device is auto rejecting Cloud to Device messages for first time #98930

willyt150 opened this issue Sep 23, 2022 · 14 comments
Assignees

Comments

@willyt150
Copy link

willyt150 commented Sep 23, 2022

Hello,

This document specifically calls out that MQTT device communication cannot Reject a Cloud to Device message, and I can confirm that when I try to call the "RejectAsync" method on my DeviceClient, with MQTT, it throws an exception saying it's not supported.

However, we are experiencing for every brand new IoT Device, the first time we send a Cloud to Device message to it, that when the device code connects and attempts to receive the messages, those messages are auto rejected and never received.

Why's it doing that? Is this expected behavior or a bug?

This is occurring on multiple IoT Hub instances

Thanks

Additional Details

  1. IoT Hub instances are all running in South Central US region. (3 instances)
  2. IoT Hubs are all S1 - Standard with 2 IoT Hub units
  3. MQTT connection is made using C#, Microsoft.Azure.Devices.Client v1.41.2.0 NuGet Package DeviceClient.Create with TransportType.Mqtt
  4. I do not have any log traces, and when calling deviceClient.ReceiveAsync(TimeSpan.FromSeconds(5)) there isn't any error, I just receive a null Message.
    1. When sending the Cloud to Device message we set the Ack value to DeliveryAcknowledgement.Full, so once the C# code attempts the ReceiveAsync call, we then receive feedback that it was Rejected. FeedbackRecord object contains
      • StatusCode: FeedbackStatusCode.Rejected
      • Description: Message rejected

Once the first message has been auto rejected, every following message is received by the C# client.


Document Details

Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

@RamanathanChinnappan-MSFT
Copy link
Contributor

@willyt150

Thanks for your feedback! We will investigate and update as appropriate.

@willyt150
Copy link
Author

@RamanathanChinnappan-MSFT thanks for the response, do you have any idea or guess as to when you'll have more information?

We're trying to get these new cloud to device message features released for our projects, however this issue arising is going to limit/prevent us being able to do that and I'm going to need to be able to provide some idea on timeline.

Thanks!

@AshokPeddakotla-MSFT
Copy link
Contributor

@willyt150

Can you please add more info on the issue?

  1. Which Region you have created Azure IoT Hub
  2. Azure IoT Hub Tier?
  3. How did you make the MQTT connection, any specific client used?
  4. Any screenshots of the error or the logs would be more helpful to help you further....
  5. Any log traces?

@willyt150
Copy link
Author

@AshokPeddakotla-MSFT Hello, I've updated the original issue w/ Additional Details

Please let me know what else I can provide.

@AshokPeddakotla-MSFT
Copy link
Contributor

@willyt150 We are checking this issue internally with our team. Please stay tuned for further updates. Appreciate your time and patience.

@AshokPeddakotla-MSFT AshokPeddakotla-MSFT added the escalated-product-team used when escalating an issue to product team label Sep 28, 2022
@AshokPeddakotla-MSFT
Copy link
Contributor

@willyt150 Below is the response from our team. Do let us know if that helps.

We have not heard similar issue before, but it will be helpful if you can confirm

  • whether your IoT Hub is on the new platform (GWV2 = GatewayV2) and whether it's specific to GWV2

  • You can check this from your IoT Hub's properties as follows

image

@willyt150
Copy link
Author

willyt150 commented Sep 29, 2022

@AshokPeddakotla-MSFT Thanks for the response.

  1. All of the IoT Hubs have None listed under Features
    • image
  2. I will see what I can do about trying to reproduce the issue with raw MQTT. The issue doesn't occur until my .NET C# code attempts to Receive the message, so honestly it makes me believe it's a bug in the .NET SDK.

I'll keep you posted on my progress/findings

@AshokPeddakotla-MSFT
Copy link
Contributor

@willyt150 Any update/progress on this?

@willyt150
Copy link
Author

@AshokPeddakotla-MSFT unfortunately, I've been given higher priority items than trying to reproduce the issue with raw MQTT (which I'll have to learn/guess at). At this point I do not have an ETA on when I'll be able to get to it.

I was however able to document the exact flow with the .Net C# SDK that results in the issue. (Not sure if anyone on the MS side could attempt the same flow with an already setup raw MQTT system)

  1. IoT Device gets created, in the IoT Hub, via DPS registration from a Symmetric Key Individual enrollment
  2. BEFORE the physical IoT code ever calls DeviceClient.ReceiveAsync a C2D message must be sent
  3. Then have the physical device call DeviceClient.ReceiveAsync, the message gets auto rejected somewhere behind the scenes and the call to ReceiveAsync returns a null Message
  4. All subsequent C2D messages that are sent can be received successfully by the device

The moment the ReceiveAsync call has been made, any C2D messages sent after all work.

@AshokPeddakotla-MSFT
Copy link
Contributor

@willyt150 We have not observed such issues.
Sure, we will definitely continue the further troubleshooting once you are available. Incase if you are blocked with this,

Please file a support request @ https://aka.ms/azsupt? If you do not have access to a support plan, please reach out @ [email protected] with a link to this Issue as well as your subscription ID and we can help get the support ticket opened for this issue.

Thread URL: #
Your Subscription ID:
Email Subject: Attn Ashok Peddakotla

Please let me know once you have done the same. Also, for further Technical help, please post to Microsoft Q&A

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

@bastyuchenko
Copy link

I've faced with the same issue. I use MQTT and .NET SDK previews/v2 branch. I receive Rejected on backend side.

@willyt150
Copy link
Author

willyt150 commented Jun 15, 2023

We ended up just building a workaround in our server side code to resend "Rejected" C2D messages up to a configured amount (e.g. 3). After the first rejection, it doesn't auto-reject again.

We keep track of the resend count of the message id ourselves.

It still appears to happen only the very first time for a brand new IoT Device. If we delete an existing IoT device that has been interacting with C2D messages and recreate it, everything works just fine and it's never rejected.

@bastyuchenko
Copy link

bastyuchenko commented Jun 19, 2023

@willyt150 or @AshokPeddakotla-MSFT or @RamanathanChinnappan-MSFT please reopen the issue, I can reproduce this issue.

I use Azure IoT SDK .NET previews/v2 and communicate through MQTT.

Steps for reproduce:

  • Create Azure IoT Hub
  • Create a fresh device identity (or it can be an existing device identity, but no session should left from a previous connection in IoT Hub for you ClientId, otherwise messages from this session will be received on device and stored to "offline queue" on SDK side and will be completed)
  • run a device code (see Console App below)
            X509Certificate2 x509Certificate = Helper.LoadProvisioningCertificate();
            var security = new AuthenticationProviderX509(x509Certificate);
            var registrationId = security.GetRegistrationId();
            IAuthenticationMethod auth = new ClientAuthenticationWithX509Certificate(x509Certificate, registrationId);
            var deviceClient = new IotHubDeviceClient(Helper.IotHubHostName, auth, new IotHubClientOptions(new IotHubClientMqttSettings()));

            deviceClient.ConnectionStatusChangeCallback = (info) =>
            {
                Console.WriteLine($"Device Status {info.Status}; Change reason: {info.ChangeReason}; Recommended action {info.RecommendedAction}.");
            };

            await deviceClient!.OpenAsync(CancellationToken.None);
            await Task.Delay(-1);
  • run Console App with MessageFeedbackProcessor
 async static Task Main(string[] args)
        {
            var iotHub = ConfigurationManager.AppSettings["IoTHubConnectionString"];
            _serviceClient = new IotHubServiceClient(iotHub);

            await ReceiveMessageFeedbacksAsync(CancellationToken.None);
            await Console.Out.WriteLineAsync("feedback listener started");

            await Task.Delay(-1);
        }

        private static async Task ReceiveMessageFeedbacksAsync(CancellationToken token)
        {
            _serviceClient.MessageFeedback.MessageFeedbackProcessor =
            (FeedbackBatch batch) =>
            {
                Console.WriteLine("New Feedback received:");
                Console.WriteLine(JsonConvert.SerializeObject(
                    new
                    {
                        batch.EnqueuedOnUtc,
                        batch.Records
                    }, Formatting.Indented));
                return Task.FromResult(AcknowledgementType.Complete);
            };
            await _serviceClient.MessageFeedback.OpenAsync(token);
        }
  • run Console App to push c2d messages
        async static Task Main(string[] args)
        {
            _serviceClient = new IotHubServiceClient(ConfigurationManager.AppSettings["IoTHubConnectionString"]);

            while (true)
            {
                await Console.Out.WriteLineAsync("Send message: ");
                Console.ReadKey();
                await SendC2dMessagesAsync(CancellationToken.None);
            }
        }

        private static async Task SendC2dMessagesAsync(CancellationToken cancellationToken)
        {
            var messageId = Guid.NewGuid().ToString("D");
            var correlationId = Guid.NewGuid().ToString("D");
            await Console.Out.WriteLineAsync($"MessageId: {messageId};");
            var message = new OutgoingMessage("test message")
            {
                MessageId = messageId,
                CorrelationId = correlationId,
                // An acknowledgment is sent on delivery success or failure.
                Ack = DeliveryAcknowledgement.Full
            };
            message.Properties.Add("custom-prop-device-id", Helper.DeviceId);

            Console.WriteLine($"Sending C2D message with Id {message.MessageId} to {Helper.DeviceId}.");

            await _serviceClient.Messages.OpenAsync();
            await _serviceClient.Messages.SendAsync(Helper.DeviceId, message, CancellationToken.None);
           Console.WriteLine($"Sent message with Id {message.MessageId} to {Helper.DeviceId}.");
        }
  • send message from the device (Console App above)

Result:

I have 4 Azure IoT Hubs:

  • MessageFeedbackProcessor receives "statusCode": "Rejected", "description": "Message rejected" regarding the sent message for three IoT Hub. Features flag for these Azure IoT Hub
    image

  • MessageFeedbackProcessor receives "statusCode": "Success" regarding the sent message for 1 IoT Hub. Features flag for these Azure IoT Hub
    image

  1. I want to understand is it a bug in Azure IoT Hub or in Azure IoT SDK .NET or in MQTTNet (that is used in Azure IoT .NET SDK).
  2. Is it possible to update Azure IoT Hub Features to GWV2 somehow from my side?
  3. Should I share with you any other information to shed light on the situation? Maybe somehow export schema of IoT Hub and share with you?
  4. Could you upgrade my Azure IoT Hub instance Features to GWV2 to confirm that this upgrade really changes the situation?

@willyt150
Copy link
Author

@willyt150 or @AshokPeddakotla-MSFT please reopen the issue, I can reproduce this issue.

I am unable to reopen the issue, I think it would need to be @AshokPeddakotla-MSFT, @RamanathanChinnappan-MSFT or another collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants