[OPEN GRANT] Fierro IPNS Pinning Service

[mrodriguez3313]

Draft open grant application

Edit: Link to repository https://github.com/Fierro-Labs/Fierro

[mishmosh]

@mrodriguez3313 Thank you for submitting this draft proposal. The overall project goal and milestone/budget structure look good.

I'm tagging @autonome @aschmahmann @lidel as IPNS experts to weigh in on the proposed technical design. (cc @ErinOCon @realChainLife)

[RangerMauve]

Just an aside, I'm doing a bunch of IPNS stuff in the Agregore project right now and finding some pain points that might be relevant.

Feel free to ping me if you ever wanna chat about it. 😁 I'd also love to integrate the IPNS pinning stuff you're doing into the web extension we're making for Agregore to make pinning easier.

[realChainLife]

Hi @mrodriguez3313 thank you for your proposal. We'd like to fund the work outlined here with a few recommendations:

• For the OpenAPI specs, can you submit these as improvements to IPNS. @aschmahmann and @lidel are available to work with you as technical sponsors, and provide additional guidance you mght need here.

• From a business viability standpoint, we recommend integrating IPNS pinning support as a feature to broader IPFS Pinning services, thanks @RangerMauve for the suggestion as well. It'd be great to explore this after the initial development work.

Please email [email protected] so we can discuss next steps.

[lidel]

(moving technical feedback from email thread to this public issue)

Pinning Service API Specs

Clarification: when it comes to devgrant and IPNS specs, my understanding is that we won't be changing IPNS specs as part of this project, but we are interested in extending Pinning Service API specs to document a convention for pinning IPNS records using this already existing API (right now, only immutable identifiers can be pinned).

The rationale is that it is better for the wider ecosystem if you base your work on an already existing pinning spec than to create something new.

Pinning the immutable IPNS record and data

User should be able to use ipfs pin remote add (which is bult-in in go-ipfs) to pin IPNS names.

This should be possible with a small spec extension:

0. User sends pin request for IPNS identifier (CIDv1 with libp2p-key codec: k2k4r8muw54003tchliewoz52my5qxq2yyvo5k0z8pj02twx7c91mul5)
1. Pinning service resolves /ipns/{libp2p-key}, finds the latest IPNS record on DHT or pubsub and pins data behind its CID + starts reproviding it to the network (both the data and IPNS record).
2. Service follows IPNS record updates: if a new version is published at /ipns/{libp2p-key}, service downloads and pins the new data, and unpins the old one + starts republishing updated IPNS record.

I think this is what "startFollowing" method does in your initial API. What I propose here is to avoid inventing new API for add/update/remove pin, and reuse existing methods instead.

Spec for adding/removing pins exists at https://ipfs.github.io/pinning-services-api-spec/, it is already supported across ecosystem, go-ipfs ships with client. If we want to add support for "IPNS pinning", then leveraging the existing spec is the way to go.

Note: this existing spec is focused on pinning only.
A bit more controversial things like key import/export could be something additional, provided by Fierro, but not part of official spec.

Trustless IPNS pinning vs Shared publishing keys

You may notice that there is no mention of sharing keys – that is because we always want to support "trustless pinning" where IPNS publishing keys are never shared with a pinning service. In many cases, delegating trust to pinning service is not acceptable, and we always need to support trustless scenario, where records are kept alive for longer periods of time (record lifetime/ttl set to more than 24h, service keeping record alive beyond 36h DHT window).

Ability to fully delegate IPNS key management, signing and publishing is useful, fine to work on it as part of the devgrant, but "delegating trust" should be optional, not mandatory.

@mrodriguez3313 this is my initial high level feedback + below answers to your questions from the email thread.

Q&A

• Are we thinking this [Pinning Service API] spec also include details about the trustless ipns pinning method? In other words, if the trustless method is a zk-snark or UCAN or another method, would we layout its details in the spec and how it operates in communicating to said service?

  • Yes, we want to document "how pinning arbitrary IPNS identifiers work", similar to how we documented Provider hints.
  • Details depend on that you mean by "trustless".
    • In my mind "trustless IPNS publishing" means "republishing signed records". Service is unable to publish modified IPNS record on its own, and user is able to pack their bags and move on, keep publishing updates even when the service goes away.
    • It is ok to experiment with "trusted IPNS published" which has private keys, but that would not be part of the official [Pinning Service API] until we figure a way of removing the need for trust somehow. (not a problem, we do want experimentation around this)

• can you explain more of what you mean by [pinning CIDv1 with libp2p-key codec]?

  • IPNS is addressed using /ipns/{libp2p-key}. It uses the same key representation as Peerids, which can be represented as CIDv1 with libp2p-key codec. The codec included inside CIDv1 makes it self-describing, allowing pinnign service to tell it is IPNS identifier that require special handling described above in "Pinning the immutable IPNS record and data".

• Currently, the project handles record following (point 2) by implementing a circular queue. I actually made a short 5-6 minute video on this: How to pin IPNS Records with Fierro | Fierro. Let me know what you think?

  • That is good! IIUC the queue would remain, that is fine for mvp.
  • The only change you need to apply, to be interoperable with IPFS Pinning Service API, would be:
    • Renaming startFollowing and stopFollowing – you would use existing add pin and remove pin endpoints.
  • Private key import would be optional – existing spec has no endpoint for this, you can experiment with "post record" and "put record" in Fierro as "extra" feature of your service. Experimentation is good.

[mrodriguez3313]

Thank you for the feedback again! Something that I thought about while reading was that a user may want to publish multiple records to a remote node. so having something like ipfs pin remote add --key=<key_name> would be great to have as well since they are already able to publish to their local node in a similar fashion. (with ipfs name publish --key=<key_name> <cid>).

In my mind "trustless IPNS publishing" means "republishing signed records"

So if I understand correctly, ultimately until a solution is crafted on the ipfs side, it will be upto the service to implement like a zk-snark, UCAN, etc... to support republishing in a decentralized & trustless manner? Or are you saying that you forsee that even with these methods it won't be possible to allow that kind of republishing? Therefore that won't be included in the spec and the tracking is all that is needed to be spec'd out as of right now?

The only change you need to apply, to be interoperable with IPFS Pinning Service API, would be: Renaming startFollowing and stopFollowing – you would use existing add pin and remove pin endpoints.

But atleast the remote tracking is 100% possible which this is a good start 👍

[lidel]

Yes, being able to pin local IPNS names creates nice UX.

So if I understand correctly, ultimately until a solution is crafted on the ipfs side, it will be upto the service to implement like a zk-snark, UCAN, etc... to support republishing in a decentralized & trustless manner?

Yes, I believe doing this in userland is the easiest path to enable experimentation.
As long the service produces a valid (current) IPNS record format, you can experiment with various trust schemes and additional endpoints.

We are open to improving IPNS record format, but if someone proposes a change to the spec of IPNS record format, that is much bigger endeavor, because it requires additional reviews, approval, and then every IPFS implementation to support new version to become interoperable.

[RangerMauve]

Just reviewed this, trustless pinning of IPNS as part of the spec seems great!

One thing to note will be that folks should increase the expiration time in their IPNS records when they do the IPNS publish so that they can get the maximum utility out of the pinning service IPNS republeshing.

Not sure if there's a CLI flag for that but any examples should have that in there with a note to make people aware of the limitations.

Is Fierro only going to support the case where the private key is sent to the pinning service? Or would it be possible for it to support both cases as part of this grant?

[mrodriguez3313]

Is Fierro only going to support the case where the private key is sent to the pinning service? Or would it be possible for it to support both cases as part of this grant?
@RangerMauve

My plan was originally to include it later after beta, but I think it will have to be milestone 2 or 3 instead. I don't know if it has to be its own grant due to some obscure grant acceptance logistics?? Maybe they can confirm 👇?

What it looks like what should happen is, the grant would be altered/rearranged like this:

M1) IPNS pinning spec as described by lidel in his latest comment. *This will only be to do part 1 in his comment for now, as we briefly discussed on Tuesday
M2) Alter current Fierro project to implement pinning spec (Pin && Pin Status Objects, fixing code&endpoints to work with those changes, return proper error responses)
M3) trustless pinning research, collaboration, & experimentation
M4) the rest of the grant from the original 1st milestone

How does that sound? @lidel @realChainLife @mishmosh. This would add time and money to the project, but I am super glad that I'm able to have this conversation with more people and especially core IPFS contributor to get this idea rolling for the community!

[lidel]

M1 and M2 sound good. tbd if we do IPNS pinning under existing /pins endpoint or prototype a new one (as noted in ipfs/pinning-services-api-spec#85 (comment)). These are cosmetics, should not impact your estimates, but it changes deliverables. Devgrant should clearly state, if it is scoped to "pinning libp2p-key CIDs with existing /ipns endpoint", or "creating and implementing a new /follows endpoint".

My vote would be for latter – exploring how /follows endpoint (with IPNS, DNSLink and time-bound pins) could work sounds more valuable to IPFS ecosystem.

As for M3, personally, I would suggest extracting M3 into a separate devgrant, or just focusing on it and coming up with a spec for delegated IPNS republishing (e.g. where republisher is only able to bump expiration time, and is not able to modify the target CID). Trying to do both "Fierro - a polished end product pinning service" and "low level IPNS specs change for delegated publishing" under a single grant feels a bit distracting, and makes it harder to review.

[mrodriguez3313]

Trying to do both "Fierro - a polished end product pinning service" and "low level IPNS specs change for delegated publishing" under a single grant feels a bit distracting, and makes it harder to review.

Yes that is what I was thinking as well. I will change up the current devgrant to reflect these additions. I like the idea of having a new endpoint, to the points you make in that thread. Ill ping you when I have the edits and see how you approve.

[RangerMauve]

One thing I'd like to just throw out there, would it be make sense to extend the pinning service spec to support pinning a url in place of pinning a cid?

e.g. /pin ipfs://bafywhatever or /pin ipns://ipfs.io/subfolder/

[mrodriguez3313]

One thing I'd like to just throw out there, would it be make sense to extend the pinning service spec to support pinning a url in place of pinning a cid?

So I roughly understand what would be the benefit of adding this support? That's not to say I don't think its a bad idea. Just that I really don't know what you mean by pinning a url? And what scenarios would that bring value to the proposition? Im also confused/thinking about how that request would look like 🤔

[RangerMauve]

Yeah, I talked about it a bit at the IPFS Implementors call yesterday: https://youtu.be/XZDXAKbpwoc?t=1228

I think right now CID make it hard to think about pinning things that aren't easily repesented as "an entire tree of IPFS data". e.g. IPNS with DNSLink or public keys seems pretty different from the CID use case. Similarly, there's been some mention that it would be useful if there was a way to pin data that's more specific like ADLs.

URLs make this a bit more straightforward in that rather than pinning a CID, one can pin either an ipfs:// URL for static data, or ipns:// URLs with either a public key or DNSLink domain.

Similarly, this could open the door for being able to pin subpaths of a URL, like say I only want to pin all the data in ipns://ipfs.io/example/ rather than the full dir. I'm not saying such a thing should be included right away, but it could be a useful thing to discuss in the future to improve the flexibility for end-users.

Finally, I'm working on an ipld:// URL spec (and integration with the gateway) and some of the plans will include supporting ADLs and Selectors for more nuanced pathing which could help applications further.

For the time being though, having ipfs://CID, ipns://public_key and ipns://DNS URLs in place of cid would make it so that clients using the pinning service API wouldn't need to import libraries to figure out how to convert DNS hostnames to CIDs and the extra code needed to figure them out.

This could look like:

POST `https://myservice.com/pins

{
  "url": "ipns://my.dns.site.com",
  "name: "My site",
  "meta": {
    "wow": "It's all the same"
  }
}

--- response

{
  requestId: "same old"
  pin: {
    "url": "same as above"
  }
}

It's pretty much the same as the existing spec, but with a url string instead of a pin CID string