Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adaptive queue for staging dials #237

Merged
merged 11 commits into from
Jan 30, 2019
Merged

Commits on Jan 29, 2019

  1. introduce adaptive queue for DHT dials.

    This patch introduces an adaptive dial queue that spawns a dynamically sized
    set of goroutines to preemptively stage dials for later handoff to the DHT
    protocol for RPC. It identifies backpressure on both ends (dial consumers and
    dial producers), and takes compensating action by adjusting the worker pool.
    
    We start with `DialQueueMinParallelism` number of workers (6), and scale up
    and down based on demand and supply of dialled peers.
    
    The following events trigger scaling:
    - we scale up when we can't immediately return a successful dial to a new
      consumer.
    - we scale down when we've been idle for a while waiting for new dial
      attempts.
    - we scale down when we complete a dial and realise nobody was waiting for it.
    
    Dialler throttling (e.g. FD limit exceeded) is a concern, as we can easily
    spin up more workers to compensate, and end up adding fuel to the fire. Since
    we have no deterministic way to detect this for now, we hard-limit concurrency
    to `DialQueueMaxParallelism` (20).
    raulk committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    5e74c4a View commit details
    Browse the repository at this point in the history
  2. adjust godoc.

    raulk committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    f044043 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    95a0975 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    24f0a2d View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7cd14f5 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    abacfe5 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    8366041 View commit details
    Browse the repository at this point in the history
  8. add godoc on global vars.

    raulk committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    72f9d4c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f2df3ec View commit details
    Browse the repository at this point in the history
  10. harden tests.

    raulk committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    bf4b91c View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2019

  1. Configuration menu
    Copy the full SHA
    74d22f3 View commit details
    Browse the repository at this point in the history