[qualys_gav] Fix rate limit headers missing error

[brijesh-elastic]

Proposed commit message

qualys_gav: fix rate limit headers missing error

This removes the rate limit calculations for the `/auth` API call, because we don't receive
the necessary headers from it. It is also mentioned in the [documentation](1) that
rate limits do not apply to the "auth" API (JWT Token Generation API).

[1] https://docs.qualys.com/en/csam/api/get_started/api_rate_limits.htm

Note

After resolving the rate limit missing rate error, I was unable to collect all events from qualys_gav. It only ingested events for 2 intervals. After investigation, it appears that rate was set too low or more throttled than what the server's rate limit headers indicate. (See issue.)

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/qualys_gav directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Relates [qualys_gav]: rate limit missing rate #16517

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[ShourieG]

LGTM, nit.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: aa0c325

History

• 💚 Build #36135 succeeded 023a139
• 💚 Build #36133 succeeded 4643ec7

cc @brijesh-elastic

[clement-fouque]

I tested version ⁠0.7.1 but still couldn’t complete a full ingest. The process halted after just four iterations, with a delay of seven seconds between the first two and the last two.

I tested with a batch of 100.

[brijesh-elastic]

Hi @clement-fouque ,

This PR will fix the error you're getting about rate limit missing rate.

I tested version ⁠0.7.1 but still couldn’t complete a full ingest. The process halted after just four iterations, with a delay of seven seconds between the first two and the last two.

During testing, I also came across the issue that the full batch of ingestion never completes. There seems to be an issue with rate calculation. The current behavior of API calls is more throttled than what the server's rate limit headers seem to indicate. Simply put, it seems like due to wrong rate calculation, the qualys API calling rate is 60 times lower than what it should be. We're looking into it in this issue.

[clement-fouque]

When can we expect a resolution? The current version has issues, so I’m considering whether reverting the entire rate-limiting functionality would be the best approach to allow us to address the problem.