Speed-up riemann-http-check with resolver and worker threads
#254
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IP addresses resolution and HTTP resources fetching are I/O bound. When testing a large amount of HTTP endpoints, this can be an issue. Some measurement we did for a use case show that checking 112 URI (most of them being serviced by 2 IP addresses) took 95 seconds. Implement an address resolution queue and a working queue in order to parallelize tasks in multiple threads. With a default of 5 resolvers and 20 workers, our test case now completes in about 4 seconds, which is more than 23 times faster. These values can be tuned using the --resolvers and --workers new parameters. While here, add metrics to track the resolvers and workers utilization and saturation.
Some tools parse ARGV, and will not cope with parameters passed to RSpec. This allows runnning rspec with random parameters, e.g. `--fail-fast`.
smortex
changed the title
Style/RedundantRegexpEscape: Redundant escape inside regexp literal
Reusing the same port to run a second server just after the previous one has been terminated sometimes fail with Errno::EADDRINUSE Address already in use. Start the server without setting a port so that the system pick up a random one for us and use this port after. This should make the test suite more reliable.
smortex
changed the title
riemann-http-check
smortex
changed the title
riemann-http-checkriemann-http-check with resolver and worker threads
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IP addresses resolution and HTTP resources fetching are I/O bound. When testing a large amount of HTTP endpoints, this can be an issue. Some measurement we did for a use case show that checking 112 URI (most of them being serviced by 2 IP addresses) took 95 seconds.
Implement an address resolution queue and a working queue in order to parallelize tasks in multiple threads. With a default of 5 resolvers and 20 workers, our test case now completes in about 4 seconds, which is more than 23 times faster. These values can be tuned using the --resolvers and --workers new parameters.
While here, add metrics to track the resolvers and workers utilization and saturation.