IPNS with multiple resolvers

[aschmahmann]

Context

As noted in the IPNS spec there are currently two components of IPNS:

• The IPNS record spec (related to the similarly proposed IPRS spec)
• The IPNS protocol

The spec currently does not specify the routing system by which a user that wishes to publish or resolve an IPNS record may do so, instead it says:

a peer can use whatever routing system it supports to make the record accessible to the remaining peers of the network

However, in actual IPNS implementations (e.g. in go-ipfs and js-ipfs) there is an implicit assumption that addresses such as /ipns/$key or dweb:/ipns/$key will resolve in some standardized way. There's a dive into the world of dweb addressing by @lgierth and @lidel here.

Thus far we've run into minimal problems here despite having both DHT and PubSub based resolution systems. This is because the PubSub system relies on the DHT for initial content resolution (i.e. only using PubSub for publishing/retrieving updates that occur while the daemon is running) and we always publish updates to both the DHT and PubSub. This effectively makes the DHT the authoritative store of IPNS information, with PubSub just being used for faster propagation.

Upcoming Issues

Some of the current endeavors to increase IPNS performance will involve creating record delivery mechanisms that are not reliant on using the DHT to propagate the IPNS records (e.g. a persistent PubSub channel, or IPNS over DNS). As a result, it's time for us to put together a proposal that adds some clarity to issues including:

• What is the protocol used when users lookup /ipns/$key?
• If that routing mechanism isn't going to be declared in the spec will we have a specified way to declare which routing mechanism we're using? (e.g. /ipns/pubsub-backed-by-dht/$key or /ipns/$key-with-routing-info)
• If not should we explicitly call out that /ipns like /ipld refers to content and pathing, but not to record retrieval or propagation?

Many of these issues were surfaced in a meeting on ongoing IPNS endeavors with notes at: ipfs/team-mgmt#948.

Thanks for your help and looking forward to hearing your thoughts: @Stebalien @pgte @satazor @vasco-santos @hugomrdias @jimpick @lidel @momack2

[lidel]

Some notes I had:

• /ipns/$key refers to content and pathing, routing should be invisible when content addressing
  • gist: we really don't want people to hardcode /ipns/ links that resolve only over dht and don't work over pubsub
  • ~/.ipfs/config should have options to enable/disable resolvers, their priority, order, and tweak DoH server for DNS lookups
  • ipfs name resovle could have a --resolver dns|pubsub|dht to enable ad-hoc diagnostics or resolving via specific method
• I feel there is no silver bullet. Different environments will have different means of resolving IPNS, but DHT should be the canonical store of IPNS information that is not the fastest, but acts as the ultimate fallback. I don't think we want to be too specific about lookup methods and order, but we should ensure DHT is listed there as a hard requirement for IPNS:
  • act of publishing should always update the DHT, not matter if other publishing methods are used in parallel
  • resolving IPNS name can be delegated and parallelized, with fallback to DHT if other methods fail or produce expired/invalid/conflicting records
    • spec should include resolver-agnostic details on when result from "delegated resolver" is ignored and we do the fallback to DHT

Question around (delegated) lookup protocol:

• What is our protection against scenario when "delegated resolver" goes offline for some time and misses some IPNS updates? When it goes back online it will return an old record, DNS replay attack style.
  • What if there is a "delegation freshness" threshold over which we force DHT lookup to ensure we got the latest record?

[aschmahmann]

Some thoughts on /ipns addressing brought on by the above issues as well as:
ipfs/notes#348, ipfs/kubo#6448, and ipfs/kubo#3942

Name resolution and IPNS

Naming resolution schemes have three components:

1. How do I identify my data? (public key, ASCII name, random 64bit integer)
2. How do I transport my data? (Server, DHT, PubSub, carrier pigeon)
3. What data format do I return? (IPNS record, DID Document, CID, IPLD Object, Raw binary, text)

IPNS (as specified in the spec) has the matching components:

1. IPNS Key (Multihash of a public key)
2. Transport (put/get) of IPNS record (DHT now, PubSub coming soon)
3. IPNS record Value, which is an arbitrary path
   • Inconsistency: ipfs name resolve you get a path, but a web-browser based dweb:/ipns/ results in a binary blob resolved from the path.

DNSLink

/ipns/DNSlink confuses people and on the outset seems ridiculous because DNSlink does not support properties 1 or 2 of IPNS. However, it does actually satisfy the (somewhat inconsistent) property 3.

Putting dweb:/dns/ as it's own namespace makes a lot of sense since people tend to identify with properties 1 and 2 more than 3. However, we should make clear which of these properties is the one that merits getting a new namespace.

New Namespaces

Some examples requirements for when to create new namespaces (and their implications):

• different identifiers (e.g. property 1)
  • dns and ens share a namespace, even worse all systems that use multihashes share a namespace (including ipfs and ipns)
• different transports (e.g. property 2)
  • ipfs, dns, ens get there own namespaces, but it's likely ipns-pubsub and ipns-dht get their own namespaces as well
• different returned data (e.g. property 3)
  • ipns, dns, ens, ... should all share the same namespace (ipfs just barely gets it's own namespace because it never returns a path)

Therefore, the Transport seems to be the only really useful piece of information in deciding whether or not to allocate a new namespace; new Transport => new Namespace.

Implications for DHT and PubSub IPNS

If we really don't want ipns-dht and ipns-pubsub to be different transports (or at least to only be different transports) we can take @lidel 's suggestion above to the extreme and modify the IPNS protocol so that it is not router pluggable, but instead checks both the PubSub and DHT routers.

To be clear, any solution that says "check PubSub and if that fails check the DHT" is based on the fact that they share properties 1 + 2. However, DNS and ENS share those properties as well and so we could just have a /dns namespace with ENS as a configurable routing mechanism and that would share properties with an IPNS with PubSub as a configurable router.

[lidel]

I agree that current IPNS+DNS situation at /ipns/ is unsustainable and extremely confusing.

At the same time, I am pretty sure "Transport" is not the best abstraction for drawing the line.
The purpose of content paths is to be transport-agnostic: we don't want to expose internals of IPNS resolver via /ipns paths.

What if we frame the concept of namespaces as path Resolvers.
Resolver takes a path and returns a path:

• /ipfs resolves to itself (ends resolving)
• /ipns resolves libp2p-keys and returns a path
• /dns resolves DNSLink over regular DNS and returns a path
• /ens resolves "ENSLink" over EthDHT and returns a path

Some clarifying notes behind this way of thinking:

• /ipns paths don't care about IPNS backend.
  Internals such as details of used "ipns transport" should be tweakable via ipfs config Ipns
  • We already have Ipns.ResolveCacheSize, adding things like Ipns.ResolverOrder or Ipns.ResolverStrategy feels like a safe way to go.
• /dns don't care about which DNS server(s) were used, nor their caching strategies. DNS servers are set in OS and/or provided by ISP.
• We don't care about which Ethereum network is behind /ens, nor which EthDNS server was used. EthDNS servers can be tweaked via Ens.Nameservers array.

Other notes:

• In theory, /ipns/{libp2p-key} could point at /ipfs/, /dns or /ens path. When accessing content using a path IPFS should resolve it recursively using a Resolver specific to the namespaces it discovers until it produces an immutable /ipfs/ path or hits the recursion limit.
  This is how /ipns works today, we would just extend it to multiple namespaces
• Backward-compatibility for /ipns/{fqdn}: it would be an alias for /dns/{fqdn}, and in case of HTTP gateway produce the HTTP 301 Redirect to /dns path

[aschmahmann]

@lidel Do you think you could clarify what makes DHT and PubSub routers of IPNS records similar, and yet DNS and ENS different?

I understand that IPNS over PubSub/DHT uses the same record type, but that's almost incidental. As a comparison think about the DID methods, even though they all have the same base record type the different transports/methods get different names did:ipid:bafyxyz, did:stack:me.stack, etc.

It's also worth noting that the transports inherently have security properties embedded in them which, while not relevant for immutable data/content-addressing like IPFS, are relevant with mutable data. For example, the IPNS PubSub router will allow third parties to persist IPNS records, but the DHT router will not.

I guess what I'm trying to do here is tease apart the difference between a Transport and a Resolver.

It's too high level to say "Resolvers map path => path (or sometimes path => data as I noted above), and every resolver gets its own namespace" because that means any two schemes that differ at all in the path => path process get their own namespace. That implies that because the DHT and PubSub routers of IPNS records do resolution differently from each other they should get their own namespaces (unless we define a protocol P that encompasses both routers). I tried Transport as a way of showing what our current plans seem to imply, but it doesn't have to be the only option. Any answer we come up with here could theoretically apply to the DID space as well, they seem to have settled on DID Method's as the implication of what needs a new namespace, but perhaps there are lessons we can learn.

[lidel]

clarify what makes DHT and PubSub routers of IPNS records similar, and yet DNS and ENS different?

In my mind IPNS remains a single namespace because while its internal "transports" can store and return different records for the same libp2p-key, the client can always pick and verify the latest record from across all available transports.

DNS<>ENS are not like this:

• TLD namespaces may diverge: .eth may produce conflicts when Ethiopia is added to DNS
• Once you import DNSSEC-backed domain to ENS, you can update it to something else in ENS while old value remains in DNS. When that happens, you get different value in DNS and ENS, but BOTH are valid in respective namespaces and you can't just pick the "latest record".

think about the DID methods, even though they all have the same base record type the different transports/methods get different names did:ipid:bafyxyz, did:stack:me.stack, etc.

DIDs made me think about concepts of "Control" and "Equivalence".
I feel those are useful in defining what content-addressed path namespaces are:

In my mind DID methods translate to sub-namespaces, because {foo} in /did/ipid/{foo} and /did/stack/{foo} is not guaranteed to mean the same or to be controlled/managed by the same entity.

/ipns/{foo} is different in that {foo} means the same across all transports ({libp2p-key}) and operates under guarantee of control by the same entity, no matter which IPNS transport/store you use.

[aschmahmann]

@lidel @hugomrdias I was talking with @Stebalien about how we're currently using Pubsub + DHT as IPNS routers.

What do you think about publishing to all routing systems simultaneously, but allowing the Publish function to return successfully as soon as even one routing system has sent out the message? This would make it so IPNS over PubSub users wouldn't have to wait around for a DHT publish to complete before ipfs name publish returns.

[lidel]

@aschmahmann I'd be ok with such fast default, as long there is an opt-in way (eg. --wait-for dht,pubsub parameter, or API to check status of pending publishing tasks) to wait for confirmation from specific system.

I imagine contexts where fast UI is built on top of backed that does not care about speed and wants to block until DHT publish succeeds.