feat: configurable block brokers #280

achingbrain · 2023-10-09T17:14:12Z

Adds configurable block providers to allow using bitswap but also other methods such as trustless gateways and any yet-to-be-invented way of resolving a CID to a block.

There are two block providers out of the box - bitswap and a trustless gateway provider that tries to download the block from one of a set of preconfigured gateways.

The block is verified after the provider has provided it. This lets users call helia.blockstore.put without verification for performance but anything coming from the network will be verified.

achingbrain · 2023-10-09T17:17:08Z

packages/helia/src/block-providers/bitswap-block-provider.ts

+      hashLoader: {
+        getHasher: async (codecOrName: string | number): Promise<MultihashHasher<number>> => {
+          const hasher = hashers.find(hasher => {
+            return hasher.code === codecOrName || hasher.name === codecOrName
+          })
+
+          if (hasher != null) {
+            return hasher
+          }
+
+          throw new Error(`Could not load hasher for code/name "${codecOrName}"`)
+        }
+      }


Bitswap already verifies blocks pulled from the network. If we like the approach in this PR we should PR bitswap to skip verification and let it be handled here.

I think if we did remove verification from bitswap, we should only do it via configuration flag (verifyBlocks (default true)) so current users don't have a hard migration to do validation themselves.

We'd release a major. Despite it being published for consumption outside IPFS projects realistically I don't anyone else is using it bar a few weekend hackers.

achingbrain · 2023-10-09T17:18:43Z

packages/helia/src/block-providers/trustless-gateway-block-provider.ts

+
+  async get (cid: CID, options: AbortOptions & ProgressOptions<TrustlessGatewayGetBlockProgressEvents> = {}): Promise<Uint8Array> {
+    // choose a gateway
+    const url = this.gateways[Math.floor(Math.random() * this.gateways.length)]


Randomly chosen gateway. If a gateway returns a 5xx we may wish to temporarily remove it from rotation and try another?

we definitely should enable some ability to retry other given gateways. IIRC, we discussed in https://pl-strflt.notion.site/Helia-reliable-retrieval-technical-design-golden-path-255-15c4d3c25a404a85b6db8bf3f8d1f310?pvs=4 that just spamming all gateways would be fine for now because we're trying to optimize for successful retrieval.

True, but I had a moment of doubt when I realised we'd be making 50 HTTP requests for a 10 block DAG.

achingbrain · 2023-10-09T17:21:36Z

packages/interface/src/blocks.ts

+  /**
+   * Notify a block provider that a new block is available
+   */
+  notify(cid: CID, block: Uint8Array, options?: NotifyProgressOptions): void


Right now this is only used by bitswap to fulfil want list requests from other peers. I guess you could argue that this BlockProvider is now Providing Blocks to the network? Or we could factor it out.

I couldn't mentally reconcile a BlockProvider with a notify method. This is why I didn't pull out bitswap in my PR, but naming is hard and I think supporting generic bitswap functionality is more important than getting the naming perfect.

I guess you could argue that this BlockProvider is now Providing Blocks to the network

This line of thought makes sense to me. We should make sure it's clear to consumers that not all blockProviders need to support a "notify" method.

Adds configurable block providers to allow using bitswap but also other methods such as trustless gateways and any yet-to-be-invented way of resolving a CID to a block..

SgtPooki

This is definitely more in the direction I wanted to go, but I wasn't sure how to reconcile bitswap as a "blockProvider". comments on that on appropriate lines.

I think a better name would help future devs and consumers, and I'm more inclined to unblock Helia-retrieval now rather than prevent Helia API difficulties in the future, but we should decide on a good name now since we’re exposing that to consumers.

A few other things I’ve thought of w.r.t. naming and future-proofing:

We could have a more generic BlockExchanger interface that doesn’t imply a direction; A BlockProvider that implies 1. provide to others (notify) and 2. provides to you (get) is confusing to me.
We could use naming similar to go-bitswap and js-ipfs-bitswap (“data trading module”), i.e. BlockTrader, BlockBroker
BlockBroker (my preferred name) could consist of both a BlockRetriever (get agent) and a BlockAnnouncer (notify agent) interface, and NetworkedStorage should deal only in BlockBrokers. (this could be done in future PRs)
We could use BlockProviderStrategy, RemoteBlockStrategies, or BlockStrategies to align with the description you wrote in packages/helia/src/index.ts

I'll put up a PR with some of the changes I've mentioned in comments so they can be pulled in without much edit if desired.

SgtPooki · 2023-10-09T17:32:45Z

packages/helia/src/block-providers/bitswap-block-provider.ts

+      hashLoader: {
+        getHasher: async (codecOrName: string | number): Promise<MultihashHasher<number>> => {
+          const hasher = hashers.find(hasher => {
+            return hasher.code === codecOrName || hasher.name === codecOrName
+          })
+
+          if (hasher != null) {
+            return hasher
+          }
+
+          throw new Error(`Could not load hasher for code/name "${codecOrName}"`)
+        }
+      }


I think if we did remove verification from bitswap, we should only do it via configuration flag (verifyBlocks (default true)) so current users don't have a hard migration to do validation themselves.

SgtPooki · 2023-10-09T18:07:13Z

packages/interface/src/blocks.ts

+  /**
+   * Notify a block provider that a new block is available
+   */
+  notify(cid: CID, block: Uint8Array, options?: NotifyProgressOptions): void


I couldn't mentally reconcile a BlockProvider with a notify method. This is why I didn't pull out bitswap in my PR, but naming is hard and I think supporting generic bitswap functionality is more important than getting the naming perfect.

I guess you could argue that this BlockProvider is now Providing Blocks to the network

This line of thought makes sense to me. We should make sure it's clear to consumers that not all blockProviders need to support a "notify" method.

packages/helia/src/index.ts

packages/helia/src/utils/networked-storage.ts

packages/helia/src/block-providers/trustless-gateway-block-provider.ts

SgtPooki · 2023-10-09T18:25:51Z

packages/helia/src/block-providers/trustless-gateway-block-provider.ts

+
+  async get (cid: CID, options: AbortOptions & ProgressOptions<TrustlessGatewayGetBlockProgressEvents> = {}): Promise<Uint8Array> {
+    // choose a gateway
+    const url = this.gateways[Math.floor(Math.random() * this.gateways.length)]


we definitely should enable some ability to retry other given gateways. IIRC, we discussed in https://pl-strflt.notion.site/Helia-reliable-retrieval-technical-design-golden-path-255-15c4d3c25a404a85b6db8bf3f8d1f310?pvs=4 that just spamming all gateways would be fine for now because we're trying to optimize for successful retrieval.

packages/helia/src/block-providers/trustless-gateway-block-provider.ts

packages/helia/src/index.ts

SgtPooki

LGTM, thanks Alex 👍

achingbrain requested a review from a team as a code owner October 9, 2023 17:14

achingbrain mentioned this pull request Oct 9, 2023

feat: helia supports additional byteProviders #276

Closed

achingbrain commented Oct 9, 2023

View reviewed changes

achingbrain force-pushed the feat/block-providers branch from 5a8029e to 8ee0646 Compare October 9, 2023 17:19

achingbrain commented Oct 9, 2023

View reviewed changes

achingbrain force-pushed the feat/block-providers branch from 8ee0646 to d264154 Compare October 9, 2023 17:22

feat: configurable block providers

4dd83b7

Adds configurable block providers to allow using bitswap but also other methods such as trustless gateways and any yet-to-be-invented way of resolving a CID to a block..

achingbrain force-pushed the feat/block-providers branch from d264154 to 4dd83b7 Compare October 9, 2023 17:28

SgtPooki reviewed Oct 9, 2023

View reviewed changes

This was referenced Oct 9, 2023

Update Helia to fallback to trustless HTTP gateways #274

Closed

feat: TrustlessGatewayBlockBroker prioritizes & tries all gateways #281

Merged

achingbrain added 4 commits October 10, 2023 08:17

chore: fix build

29ed06d

chore: fix build

783e989

chore: accept strings or urls

33bf15c

chore: rename providers to brokers

1962eb4

achingbrain changed the title ~~feat: configurable block providers~~ feat: configurable block brokers Oct 10, 2023

SgtPooki mentioned this pull request Oct 10, 2023

feat: stop automatically verifying bytes ipfs/js-ipfs-bitswap#603

Open

SgtPooki approved these changes Oct 10, 2023

View reviewed changes

achingbrain merged commit 0749cbf into main Oct 10, 2023
15 checks passed

achingbrain deleted the feat/block-providers branch October 10, 2023 16:59

SgtPooki mentioned this pull request Oct 10, 2023

feat: BlockBroker factory support #284

Merged

This was referenced Jan 8, 2024

chore: release main #360

Closed

chore(main): release 1.0.0 #361

Closed

SgtPooki mentioned this pull request Apr 9, 2024

docs: fixing changelogs SgtPooki/helia#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: configurable block brokers #280

feat: configurable block brokers #280

achingbrain commented Oct 9, 2023

achingbrain Oct 9, 2023

SgtPooki Oct 9, 2023

achingbrain Oct 10, 2023

achingbrain Oct 9, 2023

SgtPooki Oct 9, 2023

achingbrain Oct 10, 2023

achingbrain Oct 9, 2023

SgtPooki Oct 9, 2023

SgtPooki left a comment

SgtPooki Oct 9, 2023

SgtPooki Oct 9, 2023

SgtPooki Oct 9, 2023

SgtPooki left a comment

feat: configurable block brokers #280

feat: configurable block brokers #280

Conversation

achingbrain commented Oct 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SgtPooki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SgtPooki left a comment

Choose a reason for hiding this comment