Throttle max-concurrency if bandwidth limit is exceeded

[mortenjoenby]

Feature Request

Which version of blobfuse was used?

2.1.0

Which OS distribution and version are you using?

Oracle Linux 8.8

If relevant, please share your mount command.

Using configuration file. See comment below.

What was the issue encountered?

Ref.: #1230

When exceeding the VM network bandwidth limit, we are getting "connection reset by peer" and "cancelled context".

Have you found a mitigation/solution?

azstorage: max-concurrency defaults to 32 threads (TCP connections) per file being handled.
This seem to max out the VM network bandwidth.
When manually "throttling" this to 8 or 16 we no longer seem to get errors.

Suggestion

We would like Blobfuse to somehow discover that it runs into this limitation and then automatically throttle the concurrency to avoid errors resulting in the client application (in this case Oracle RMAN backup) failing.

[mortenjoenby]

Configuration file:

allow-other: false

logging:
  type: base
  file-path: /var/log/blobfuse/blobfuse2.log
  level: log_debug
  max-file-size-mb: 50
  file-count: 50

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage

libfuse:
  default-permission: 0644
  attribute-expiration-sec: 240
  entry-expiration-sec: 240
  negative-entry-expiration-sec: 240
  # Required for Oracle RMAN
  ignore-open-flags: true

file_cache:
  # Pre-create dir, should be done at startup if not already existing
  path: /mnt/blobfusetmp
  timeout-sec: 300
  allow-non-empty-temp: true
  cleanup-on-start: true
  # Max size: 150GB
  #max-size-mb: 153600

attr_cache:
  # Default: 120
  #timeout-sec: 7200

azstorage:
  type: block
  account-name: xxxxx
  account-key: xxxxxx
  endpoint: https://xxxxxx.blob.core.windows.net
  mode: key
  container: blob-backup
  # max-retries default: 5
  max-retries: 100
  block-size-mb: 16
  sdk-trace: true
  max-concurrency: 16   <<<<<<=======

health_monitor:
  enable-monitoring: true
  stats-poll-interval-sec: 10
  process-monitor-interval-sec: 30
  output-path: /tmp
  monitor-disable-list: