Update Helia to fallback to trustless HTTP gateways

[SgtPooki]

discussed in https://pl-strflt.notion.site/Helia-reliable-retrieval-technical-design-golden-path-255-15c4d3c25a404a85b6db8bf3f8d1f310?pvs=4

Should do #272 first

Tasks

Beta Give feedback

1. create ByteProvider interface
2. add byteProviders config key to Helia configuration object
3. Emit onProgress events that specify which provider (dht, gateway url, etc..) is returning bytes
4. implement default fallback byteProvider for a trustless http gateway
5. clarify with @achingbrain whether we need to verify bytes or if this is already handled
6. feat: helia supports additional byteProviders #276
   +0
   
7. feat: configurable block brokers #280
   +0
8. feat: TrustlessGatewayBlockBroker prioritizes & tries all gateways #281
   +0
   
9. feat: BlockBroker factory support #284
   +0
   
10. feat: blockBrokers can control block validation #285
    +0
    
11. Marketing for BlockBroker API changes and retrievability improvements #287
    +0
    

[SgtPooki]

#276 is still in progress, but I could use some input from @achingbrain that the direction i'm heading in works before I flesh everything out

[SgtPooki]

some of this work was moved to #280 and also #281 on top of that

[SgtPooki]

The work here is done and merged into the main branch but is not released on @latest tag on npm yet. You can experiment with the changes made on the @next tag.

[BigLep]

@SgtPooki : what are the conditions for closing this?

1. Helia release?
2. Any additional docs we need to add? (or is that already in the API docs)? (If already in the API docs, can you point me to the relevant ones so I can look at them from a user's point of view?)
3. ?

Thanks

[SgtPooki]

@SgtPooki : what are the conditions for closing this?

1. Helia release?

Yep. I discussed this with Alex and we talked about how we would let this sit in @next tagged release and request devs try it out for a bit before merging it. There's no guarantee that devs will try it, but I plan to.

2. Any additional docs we need to add? (or is that already in the API docs)? (If already in the API docs, can you point me to the relevant ones so I can look at them from a user's point of view?)

Docs are not published yet but should be in https://helia.io/interfaces/helia.HeliaInit.html & https://helia.io/modules/_helia_interface.blocks.html when they are. You can see the docs that will be mentioned in the source here:

• helia/packages/helia/src/index.ts

  Lines 95 to 99 in 43932a5

  	/**
  	* A list of strategies used to fetch blocks when they are not present in
  	* the local blockstore
  	*/
  	blockBrokers?: Array<(components: any) => BlockBroker>

• helia/packages/interface/src/blocks.ts

  Lines 66 to 90 in 43932a5

  	export type BlockRetrievalOptions<GetProgressOptions extends ProgressOptions = ProgressOptions> = AbortOptions & GetProgressOptions & {
  	/**
  	* A function that blockBrokers should call prior to returning a block to ensure it can maintain control
  	* of the block request flow. e.g. TrustedGatewayBlockBroker will use this to ensure that the block
  	* is valid from one of the gateways before assuming it's work is done. If the block is not valid, it should try another gateway
  	* and WILL consider the gateway that returned the invalid blocks completely unreliable.
  	*/
  	validateFn?(block: Uint8Array): Promise<void>
  	}
  	export interface BlockRetriever<GetProgressOptions extends ProgressOptions = ProgressOptions> {
  	/**
  	* Retrieve a block from a source
  	*/
  	retrieve(cid: CID, options?: BlockRetrievalOptions<GetProgressOptions>): Promise<Uint8Array>
  	}
  	export interface BlockAnnouncer<NotifyProgressOptions extends ProgressOptions = ProgressOptions> {
  	/**
  	* Make a new block available to peers
  	*/
  	announce(cid: CID, block: Uint8Array, options?: NotifyProgressOptions): void
  	}
  	export type BlockBroker = BlockRetriever | BlockAnnouncer

It could be worth adding some info to the wiki or README including the diagrams at #272 once we decide we will release these changes.

[pcfreak30]

I can actually confirm the trustless gateway works. The only issue are some of the gateways don't properly allow CORS which causes issues in browser when fetching CAR data.

[SgtPooki]

@pcfreak30 thanks for the update! Which gateways were causing issues?

[pcfreak30]

@pcfreak30 thanks for the update! Which gateways were causing issues?

I would honestly test them all. I am using a rather complex system that is powered by webworkers inside iframes. The CORS should be such it allows everything.

[SgtPooki]

Yea, will do. FYI, you can limit the gateway urls you enable by setting blockBrokers in the config object passed to createHelia:

import { createHelia, type Helia } from 'helia'
import { bitswap, trustlessGateway } from 'helia/block-brokers'

await createHelia({
      blockBrokers: [
        bitswap(),
        trustlessGateway({ gateways: ['https://dweb.link', 'https://ipfs.io'] })
      ]
    })

[SgtPooki]

FYI that I threw a quick codepen together that shows this working fine for all default gateways: https://codepen.io/SgtPooki/pen/JjxYpZX

[pcfreak30]

I will update if I can replicate in my setup, thanks.

[pcfreak30]

@SgtPooki i was able to get a screenshot from a tester.

[BigLep]

@SgtPooki : what is needed for closing out this issue?

[SgtPooki]

I believe this is done except for #374. The spec work for other gateways is at ipfs/specs#423.

closing this