Gateway improvements

[lidel]

This a placeholder for gathering references to create a more fleshed out plan.

We have distinct Gateway (/ipfs, /ipns) and API (/api/v0) endpoints today, and only the most basic read operations can be executed via Gateway.

Exposing /api/v0 on the gateway was a mistake, and we need to work towards removing the need for it.

The goal here would be to shift what is possible on the Gateway endpoints natively, so most popular operations like manipulating IPLD work without touching the advanced API.

---

Ordered improvements

Lower effort, high impact

Things well understood, with clear implementation plan:

• smart HTTP cache invalidation for /ipns/
  "dropping only the branches that changed, not entire DAG"
  (feat: X-Ipfs-Roots for smarter HTTP caches kubo#8720)
• block and car stream support
  (wip: feat(gateway): Block and CAR response formats kubo#8758)
• ipld selector support (for car stream download)
  (https://github.com/ipfs/go-ipfs/issues/8769)
• dag-json and dag-cbor support (IPLD support on Gateways #182)

High impact maintenance work

Things that do not add anything new, but need to happen to solve production issues, improve devexp, and create solid foundations for wider community to participate in Gateway work beyond MVP features listed in the section above.

• SPECS
• Proper cache-control for /ipns/ based on TTL
  (https://github.com/ipfs/go-ipfs/issues/1818#issuecomment-1015849462)
• HTTP headers cleanup (HTTP Headers Cleanup: API and Gateway #132)
• extracting gateway code into a standalone library (https://github.com/ipfs/boxo/tree/main/gateway)
• compliance test suite

Higher effort, high impact

Things that have extremely high impact, but need some additional research / design to happen.

• writable gateways (IPIP: Data Onboarding via HTTP POST (and future ipfs:// POST|PUT) specs#375)
  • POST /ipfs/ with support for IPLD content types (as a template for ipfs:// behavior in browsers)
• mountable gateways, Stage 0: read-only /ipfs/{cid}
  "Mount any gateway CID root as a native read-only drive"
  (WebDAV (HTTP Extensions for Distributed Authoring) #146)
• mountable gateways, Stage 1: read-write MFS
  "Mount Files from ipfs-desktop as a native, writable drive"
  (WebDAV (HTTP Extensions for Distributed Authoring) #146)

---

Old Brainstorm

Below is the landscape of things worth considering.

No regrets

Things worth doing

• ?format=car|block|... – Gateway support for /ipfs/{cid}?format=car|raw|... kubo#8234
  • better IPLD support (IPLD support on Gateways #182)
  • support IPLD, at very leasst DAG-CBOR/JSON (feat: serve CBOR encoded DAG nodes from the gateway kubo#8037)
  • JSON/CBOR dir listings (Gateway: client opt-in JSON responses for directory listing kubo#7552)
  • ability to fetch a single block
  • ability to fetch arbitrary DAG as CAR
• Revisit the concept of Writable Gateways – IPIP: Data Onboarding via HTTP POST (and future ipfs:// POST|PUT) specs#375
  👉 what if:
  • not limited to unixfs (IPLD support on Gateways #182)
  • supports MFS on localhost
  • supports IPNS publishing via HTTP POST
  • and/or supports DAG imports (Make ipfs.dag.export built-in feature of HTTP gateways #170)
  • can be mounted as network drive (WebDAV: move fs-specific objects into own pkg kubo#146)
• HTTP headers cleanup (HTTP Headers Cleanup: API and Gateway #132)
• Improved image when social sharing a directory, similar to what github does (Gateway: dynamic image for social sharing kubo#8811)
• DNSLink/IPNS fallback cache (Known DNSLink and IPNS should work in airplane mode kubo#8245)

Other ideas

Big complexity and low impact for user running it on localhost.

• resumable uploads

  • prompt: importing huge file (DAG, or just a 4K video) over janky connection
  • prior art: https://tus.io/faq.html, [WIP] feat: support chunked add requests js-ipfs#1540

• access controls?

  • writable aspect, and also pinning both need some access controls to limit capabilities to specific Origins or requests with correct access token

[RangerMauve]

Regarding being able to POST to a gateway, I think it would be useful to support multipart/form-data in order to upload a bunch of files at once.

E.g. using the FormData API in browsers: https://developer.mozilla.org/en-US/docs/Web/API/FormData

I've done some tests in the Agregore Browser and in the ipfs-protocol-compliance-suite and it's been pretty nice for ergonomics.

Here's what it could look like from a user's perspective:

const formData = new FormData()

formData.append('file', new Blob(['hello world!']), 'example1.txt')
formData.append('file', new Blob(['Goodbye World?']), 'example2.txt')

// Post to empty directory CID
const postResponse = await fetch('ipfs://bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354/', {
  method: 'POST',
  body: formData
})

const newURL = await postResponse.text()

// newURL will be a directory with `example1.txt` and `example2.txt`

This will also play well with people using file inputs or drag and drop to upload data.

[lidel]

Created ipfs/specs#375 for tracking ideas around "Writable Gateways"