[BUG] TooManyRequests when audit parameter is enabled

[fmdkrenepedersen]

Describe the bug
Two different tenants are getting "TooManyRequests" during the backup in the past two weeks, if the audit parameter is specified. Not on every run, but often 8 out of 10 runs.

To Reproduce
Steps to reproduce the behavior:

Add the audit parameter to the backup.

Expected behavior
A clear and concise description of what you expected to happen.

Retry or ignore the error.

Screenshots
If applicable, add screenshots to help explain your problem.

Run type (please complete the following information):

• Mode: 1 (backup)
• Client: Pipeline
• Version: Current public

Additional context
Add any other context about the problem here.

[almenscorner]

Since 429 is throttling in graph which we are managing in code (not forever though) can you see in the run output that it retries a couple of times before the error is thrown?

[almenscorner]

Are you still seeing this on each run or has it improved?

[snodecoder]

I saw this behaviour only on an initial backup, my workaround was to run an initial backup without audit, and then on subsequent runs include audit

[almenscorner]

I found an issue with audit logs and Graph throttling while setting things up in 2.4.0 so this should be fixed in the upcoming version. You can already try the beta if you want to verify.

[nyuware]

Hey,

I'm seeing this behavior also, here are the logs

Tue Mar 25 15:14:04 2025 [ERROR] Error making Graph request to https://graph.microsoft.com/v1.0/deviceManagement/auditEvents: 'retry-after'
Tue Mar 25 15:14:04 2025 [ERROR] Error processing Management Intents data: Request failed with 429 - {"error":{"code":"TooManyRequests","message":"{\r\n  \"_version\": 3,\r\n  \"Message\": \"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 9c3e581b-8053-479b-93f5-4c6ee6ee023e - Url: [https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-8881-0601-032415301251/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceIntent%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a04.218565Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a04.218604Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\",\r\n](https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-8881-0601-032415301251/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceIntent%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a04.218565Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a04.218604Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\%22,\r\n)  \"CustomApiErrorPhrase\": \"\",\r\n  \"RetryAfter\": null,\r\n  \"ErrorSourceService\": \"\",\r\n  \"HttpHeaders\": \"{}\"\r\n}","innerError":{"date":"2025-03-25T15:14:04","request-id":"9c3e581b-8053-479b-93f5-4c6ee6ee023e","client-request-id":"9c3e581b-8053-479b-93f5-4c6ee6ee023e"}}}
Tue Mar 25 15:14:05 2025 [ERROR] Error making Graph request to https://graph.microsoft.com/v1.0/deviceManagement/auditEvents: 'retry-after'
Tue Mar 25 15:14:05 2025 [ERROR] Error processing Management Partner data: Request failed with 429 - {"error":{"code":"TooManyRequests","message":"{\r\n  \"_version\": 3,\r\n  \"Message\": \"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: f99dff2e-280a-4c0d-8179-4ab9475c0783 - Url: [https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-8837-0601-032416243340/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27Enrollment%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a05.046432Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a05.046476Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\",\r\n](https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-8837-0601-032416243340/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27Enrollment%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a05.046432Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a05.046476Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\%22,\r\n)  \"CustomApiErrorPhrase\": \"\",\r\n  \"RetryAfter\": null,\r\n  \"ErrorSourceService\": \"\",\r\n  \"HttpHeaders\": \"{}\"\r\n}","innerError":{"date":"2025-03-25T15:14:05","request-id":"f99dff2e-280a-4c0d-8179-4ab9475c0783","client-request-id":"f99dff2e-280a-4c0d-8179-4ab9475c0783"}}}

sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 319, in start
    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 272, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 199, in backup_intune
    results.append(SettingsCatalogBackupModule(**params).main())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup/Intune/SettingsCatalog.py", line 57, in main
    self.audit_data = self.make_audit_request(self.audit_filter)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 199, in make_audit_request
    data = self.make_graph_request(endpoint=endpoint, params=q_param)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 156, in make_graph_request
    raise requests.exceptions.HTTPError(
requests.exceptions.HTTPError: Request failed with 429 - {"error":{"code":"TooManyRequests","message":"{\r\n  \"_version\": 3,\r\n  \"Message\": \"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 9a6cc5e2-8fc2-4912-8cbc-be7047520fab - Url: [https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-0589-0601-032416063889/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceConfiguration%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a10.573148Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a10.573193Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\",\r\n](https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-0589-0601-032416063889/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceConfiguration%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a10.573148Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a10.573193Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\%22,\r\n)  \"CustomApiErrorPhrase\": \"\",\r\n  \"RetryAfter\": null,\r\n  \"ErrorSourceService\": \"\",\r\n  \"HttpHeaders\": \"{}\"\r\n}","innerError":{"date":"2025-03-25T15:14:10","request-id":"9a6cc5e2-8fc2-4912-8cbc-be7047520fab","client-request-id":"9a6cc5e2-8fc2-4912-8cbc-be7047520fab"}}}

##[error]Bash exited with code '1'.
##[error]Bash wrote one or more lines to the standard error stream.
##[error]Traceback (most recent call last):
  File "/home/vsts/.local/bin/IntuneCD-startbackup", line 8, in <module>

##[error]    sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 319, in start
    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 272, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 199, in backup_intune
    results.append(SettingsCatalogBackupModule(**params).main())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup/Intune/SettingsCatalog.py", line 57, in main
    self.audit_data = self.make_audit_request(self.audit_filter)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 199, in make_audit_request
    data = self.make_graph_request(endpoint=endpoint, params=q_param)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 156, in make_graph_request
    raise requests.exceptions.HTTPError(
requests.exceptions.HTTPError: Request failed with 429 - {"error":{"code":"TooManyRequests","message":"{\r\n  \"_version\": 3,\r\n  \"Message\": \"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 9a6cc5e2-8fc2-4912-8cbc-be7047520fab - Url: [https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-0589-0601-032416063889/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceConfiguration%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a10.573148Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a10.573193Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\",\r\n](https://fef.amsub0302.manage.microsoft.com/StatelessAuditingFEService/98cf933b-ffff-0589-0601-032416063889/deviceManagement/auditEvents?api-version=2022-06-14&$filter=componentName+eq+%27DeviceConfiguration%27+and+(activityDateTime+gt+2025-03-24T15%3a14%3a10.573148Z)+and+(activityDateTime+le+2025-03-25T15%3a14%3a10.573193Z)+and+activityOperationType+ne+%27Get%27&$orderby=activityDateTime+desc&$select=actor%2cactivityDateTime%2cactivityOperationType%2cactivityResult%2cresources\%22,\r\n)  \"CustomApiErrorPhrase\": \"\",\r\n  \"RetryAfter\": null,\r\n  \"ErrorSourceService\": \"\",\r\n  \"HttpHeaders\": \"{}\"\r\n}","innerError":{"date":"2025-03-25T15:14:10","request-id":"9a6cc5e2-8fc2-4912-8cbc-be7047520fab","client-request-id":"9a6cc5e2-8fc2-4912-8cbc-be7047520fab"}}}

They are a bit long!

I used the pipeline without audit for two weeks, and then decided to try adding --audit and now it fails

I will try the beta (2.4.0b6) and report back

[snodecoder]

Using 2.4.0b6 fixed it for me ;)

[nyuware]

I can confirm it fixed it also for me! ;)