feat: Support Store S3

[speeddragon]

This PR adds S3 support to hb_store.

• This is under the s3 profile.
• Required keys are:
  • bucket
  • priv_access_key_id
  • priv_secret_access_key
• Optional keys are:
  • region
  • endpoint
  • force_path_style
  • retry-attempts
  • min-retry-delay
  • max-retry-delay
  • retry-mode
  • scope
• Added MinIO via docker-compose to test S3 behaviour.
• 2 new dependencies added:
  • erlcloud_s3 dependency added to interface with S3 protocol.
  • meck dependency added to test error behaviour from erlcloud_s3.

Store Configuration

To test this store, create the config.flat file and ensure the MinIO service is running.

store/ao-types: .=list

% S3 Config
store/1/ao-types: store-module=atom
store/1/store-module: hb_store_s3
store/1/bucket: hb-s3
store/1/priv_access_key_id: minioadmin
store/1/priv_secret_access_key: minioadmin
store/1/endpoint: http://localhost:9000
store/1/force_path_style: true
store/1/region: us-east-1

% Gateway with S3 Config
store/2/ao-types: store-module=atom
store/2/store-module: hb_store_gateway
store/2/remote-store/ao-types: .=list
store/2/remote-store/1/ao-types: store-module=atom
store/2/remote-store/1/store-module: hb_store_s3
store/2/remote-store/1/bucket: hb-s3
store/2/remote-store/1/priv_access_key_id: minioadmin
store/2/remote-store/1/priv_secret_access_key: minioadmin
store/2/remote-store/1/endpoint: http://localhost:9000
store/2/remote-store/1/force_path_style: true
store/2/remote-store/1/region: us-east-1

A configuration with LMDB as the first store would be:

store/ao-types: .=list

% LMDB
store/1/ao-types: store-module=atom
store/1/store-module: hb_store_lmdb
store/1/name: cache-mainnet/lmdb

% S3 Config
store/2/ao-types: store-module=atom
store/2/store-module: hb_store_s3
store/2/bucket: hb-s3
store/2/priv_access_key_id: minioadmin
store/2/priv_secret_access_key: minioadmin
store/2/endpoint: http://localhost:9000
store/2/force_path_style: true
store/2/region: us-east-1

% Gateway with S3 Config
store/3/ao-types: store-module=atom
store/3/store-module: hb_store_gateway
store/3/local-store/ao-types: .=list
store/3/local-store/1/ao-types: store-module=atom
store/3/local-store/1/store-module: hb_store_s3
store/3/local-store/1/bucket: hb-s3
store/3/local-store/1/priv_access_key_id: minioadmin
store/3/local-store/1/priv_secret_access_key: minioadmin
store/3/local-store/1/endpoint: http://localhost:9000
store/3/local-store/1/force_path_style: true
store/3/local-store/1/region: us-east-1

Decisions

• Sharded key on the first key element. Keys with a size equal to or below SHARD_CUT (currently set to 4) aren't valid.

Performance

Storing a message in S3 using one file per message attributes make storing a complete message quite slow.

hb_store: -generate_test_suite/2-fun-1- (hb_store_s3: benchmark message read write)...
        Generated 25 messages (size 8,192 bits)... 
        Wrote 25 messages in 3,103ms (8 messages/s)... 
        Read 25 messages in 4,580ms (5 messages/s)... [8.138 s] ok

Future work

• Extract exponential backoff from hb_store_s3 to hb_store.

[speeddragon]

This is necessary to use hb_http_client in hb_store_s3 requests.