feat: BlockBroker factory support #284

SgtPooki · 2023-10-10T19:42:15Z

Adds on to changes from #280 and #281

This makes it much easier to support constructing helia with custom blockBrokers while still allowing consumers easy access to the default BitswapBlockBroker. One particular use-case where this comes in handy: our tests.

Take the code in packages/helia/test/fixtures/create-helia.ts for example. Without these changes, you have to do something like:

import { webSockets } from '@libp2p/websockets'
import * as Filters from '@libp2p/websockets/filters'
import { MemoryBlockstore } from 'blockstore-core' // new import
import { MemoryDatastore } from 'datastore-core' // new import
import { circuitRelayTransport } from 'libp2p/circuit-relay'
import { identifyService } from 'libp2p/identify'
import { BitswapBlockBroker } from '../../src/block-brokers/index.js' // new import
import { createHelia as createNode } from '../../src/index.js'
import { defaultHashers } from '../../src/utils/default-hashers.js' // new import
import { createLibp2p } from '../../src/utils/libp2p.js' // new import
import type { Helia } from '@helia/interface'

export async function createHelia (): Promise<Helia> {
  const datastore = new MemoryDatastore()
  const blockstore = new MemoryBlockstore()
  const libp2p = await createLibp2p(datastore, {
    addresses: {
      listen: [
        `${process.env.RELAY_SERVER}/p2p-circuit`
      ]
    },
    transports: [
      webSockets({
        filter: Filters.all
      }),
      circuitRelayTransport()
    ],
    connectionGater: {
      denyDialMultiaddr: async () => false
    },
    services: {
      identify: identifyService()
    }
  })
  return createNode({
    blockstore,
    blockBrokers: [
      new BitswapBlockBroker(libp2p, blockstore, defaultHashers())
    ],
    datastore,
    libp2p
  })
}

However, with these changes, we can now do this:

import { webSockets } from '@libp2p/websockets'
import * as Filters from '@libp2p/websockets/filters'
import { circuitRelayTransport } from 'libp2p/circuit-relay'
import { identifyService } from 'libp2p/identify'
import { BitswapBlockBrokerFactory } from '../../src/block-brokers/bitswap-block-broker.js' // new import
import { createHelia as createNode } from '../../src/index.js'
import type { Helia } from '@helia/interface'

export async function createHelia (): Promise<Helia> {
  return createNode({
    blockBrokers: [
      BitswapBlockBrokerFactory
    ],
    libp2p: {
      addresses: {
        listen: [
        `${process.env.RELAY_SERVER}/p2p-circuit`
        ]
      },
      transports: [
        webSockets({
          filter: Filters.all
        }),
        circuitRelayTransport()
      ],
      connectionGater: {
        denyDialMultiaddr: async () => false
      },
      services: {
        identify: identifyService()
      }
    }
  })
}

Some key benefits:

No need to construct libp2p ourselves and pass to BlockBrokers
No need to construct our own blockstore for passing to BitswapBlockBroker
No need to construct our own datastore for passing to createLibp2p
additional hashers can be added easily to heliaInit config and passed to all blockBrokers.

SgtPooki

self review

SgtPooki · 2023-10-10T19:56:37Z

packages/helia/src/helia.ts

@@ -19,6 +20,7 @@ const log = logger('helia')
 interface HeliaImplInit<T extends Libp2p = Libp2p> extends HeliaInit<T> {
  libp2p: T
  blockstore: Blockstore
+  blockBrokers: BlockBroker[]


needed to ensure it's not using the Array<BlockBroker | BlockBrokerFactoryFunction> type from HeliaInit

SgtPooki · 2023-10-10T20:00:31Z

packages/helia/src/index.ts

+    return blockBroker({
+      blockstore,
+      datastore,
+      libp2p,
+      hashers
+    }) satisfies BlockBroker


are there other things that blockBrokers may need?

SgtPooki · 2023-10-10T20:01:13Z

packages/helia/src/index.ts

@@ -98,7 +98,7 @@ export interface HeliaInit<T extends Libp2p = Libp2p> {
   * A list of strategies used to fetch blocks when they are not present in
   * the local blockstore
   */
-  blockBrokers?: BlockBroker[]
+  blockBrokers?: Array<BlockBroker | BlockBrokerFactoryFunction>


do we just want to accept BlockBrokerFactoryFunction[] instead of BlockBrokers directly?

SgtPooki · 2023-10-10T20:01:35Z

packages/helia/test/fixtures/create-helia.ts

  return createNode({
+    blockBrokers: [
+      BitswapBlockBrokerFactory
+    ],


We should not be making external network requests (i.e. not calling default trustlessGateways) during tests.

SgtPooki · 2023-10-10T20:02:40Z

packages/interface/src/index.ts

+/**
+ * A function that receives some {@link Helia} components and returns a
+ * {@link BlockBroker}.
+ *
+ * This is needed in order to re-use some of the internal components Helia
+ * constructs without having to hoist each required component into the top-level
+ * scope.
+ */
+export interface BlockBrokerFactoryFunction {
+  (heliaComponents: BlockBrokerFactoryComponents): BlockBroker
+}


seemed like the best place for this but lmk if somewhere else is better.

SgtPooki · 2023-10-10T20:03:45Z

packages/helia/test/utils/networked-storage.spec.ts

@@ -15,7 +15,7 @@ import type { BitswapBlockBroker } from '../../src/block-brokers/bitswap-block-b
 import type { Blockstore } from 'interface-blockstore'
 import type { CID } from 'multiformats/cid'

-describe('storage', () => {
+describe('networked-storage', () => {


this has the same describe namespace as packages/helia/test/storage.spec.ts and it bugs me

SgtPooki · 2023-10-10T20:04:43Z

packages/helia/test/pins.recursive.spec.ts

@@ -25,6 +25,7 @@ describe('pins (recursive)', () => {
    dag = await createDag(codec, blockstore, 2, 3)

    helia = await createHelia({
+      blockBrokers: [],


Prevent blockBroker networking when not needed

SgtPooki · 2023-10-10T20:04:48Z

packages/helia/test/pins.depth-limited.spec.ts

@@ -27,6 +27,7 @@ describe('pins (depth limited)', () => {
    dag = await createDag(codec, blockstore, MAX_DEPTH, 3)

    helia = await createHelia({
+      blockBrokers: [],


Prevent blockBroker networking when not needed

This approach is consistent with libp2p components, unixfs, ipns, etc. Since #281 and #284 were merged without review, this PR implements suggsestions that would have been in the review of those PRs. 1. Creation of block brokers is done by exported function. If your broker takes arguments, pass them to the factory function. The factory then returns a function that accepts helia components and returns the broker. 2. Removes BitswapBrokerFactory as it is redunant The internal API may need some more work but the external API should be relatively stable.

SgtPooki added 2 commits October 10, 2023 10:40

chore: cleanup trustless-gateway-block-broker

e91f917

feat: block-broker factory support

a0eff4d

SgtPooki requested a review from a team as a code owner October 10, 2023 19:42

SgtPooki changed the base branch from main to feat/block-providers-sgtpooki October 10, 2023 19:44

fix: tests and lint

2fc1043

SgtPooki commented Oct 10, 2023

View reviewed changes

SgtPooki added 2 commits October 10, 2023 13:19

fix: export BitswapBbFactory from /block-brokers

e09d09f

fix: helia/interop tests dont use gateway blockBrokers

9ae76ce

This was referenced Oct 10, 2023

feat: blockBrokers can control block validation #285

Merged

Update Helia to fallback to trustless HTTP gateways #274

Closed

SgtPooki self-assigned this Oct 12, 2023

feat: blockBrokers can control block validation (#285)

cc3e9ce

SgtPooki merged commit a3260ff into feat/block-providers-sgtpooki Oct 13, 2023
15 checks passed

SgtPooki deleted the fix/block-broker-testing branch October 13, 2023 02:04

achingbrain mentioned this pull request Oct 13, 2023

refactor: use functions for block broker creation #286

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: BlockBroker factory support #284

feat: BlockBroker factory support #284

SgtPooki commented Oct 10, 2023

SgtPooki left a comment

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

SgtPooki Oct 10, 2023

feat: BlockBroker factory support #284

feat: BlockBroker factory support #284

Conversation

SgtPooki commented Oct 10, 2023

SgtPooki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment