Allowing URI values in DNSLink TXT records

[lidel]

Context

I've been talking with @sjoerdvangroning about https://github.com/pdsinterop/simply-domain (originally created with SOLID in mind, but aims to be generic interop as part of Personal Data Stores Interoperability project) and there is a lot of overlap between that spec draft and what DNSLink allows.

The gist is they want to support TXT pointers to the content using multiple protocols (DNSLink already allows that).
The differences are:

• providing "proxy" (we use subdomain+A record convention instead),
• "priority" (we don't have that)
• "url" (we use unix-like paths with namespace prefix instead).

The diff is mostly cosmetic, and my hope is they could reuse DNSLink, so we don't end up with the standard "standard" situation described in this XKCD:

DNSLink TXT record

dnslink=/<namespace>/<identifier>
dnslink=/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

Simply-domain TXT record

<proxy> <priority> <url>
www.resource.com 10 https://example.com/file.jpg
www.resource.com 15 https://mirror.example.org/file.jpg
www.resource.com 20 ipfs://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

<proxy> is solved in DNSLink by a convention of using of A record on the domain hosting _dnslink subdomain.
<priority> is discussed in #34, let's focus this issue about allowing URIs.
<url> is.. a subset of URIs, and there is a good argument they are more user and developer friendly than our content/addr paths (especially, given that multiaddr can't represent all HTTP URL attributes yet).

Proposal: allow URIs (URLs, URNs)

#16 made the spec a bit more strict, it removed mention of URLs being allowed (things that don't start with / are simply ignored).

After getting feedback from @sjoerdvangroning I feel that was a mistake that will alienate developers who are not entrenched in IPFS ecosystem, sabotaging DNSLink potential to be trully cross-protocol and cross-ecosystem.

DNSLink should support values being URIs (https://en.wikipedia.org/wiki/Uniform_Resource_Identifier).

👉 What notation should be used?

My proposal is to extend spec somehow.

(A) Assume URI unless the value starts with /

One way, is to state that everything that does not start with / should be interpreted as URI:

dnslink=/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
dnslink=hyper://hypercore-id+version-tag
dnslink=https://example.com/file.jpg

The main argument for this approach is that it looks natural when used outside IPFS ecosystem.
The entire world talks URIs, and this makes them first-class citizens.

My vote is to go this route.

(B) Create explicit /uri/ namespace

An alternative is to document /uri/ namespace, and allow things like:

dnslink=/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
dnslink=/uri/hyper://hypercore-id+version-tag
dnslink=/uri/https://example.com/file.jpg

It makes it clear that other namespaces than URI are allowed, but looks wasteful and redundant when used outside IPFS ecosystem.
My vote would be against this, as it feels unnecessary.

Backward-compatibility notes

This needs some thought around backward-compatibility and DX/UX:"

• Values that do not start with / are already ignored, so no change in existing software is necessary, no matter which way we go.
• IPFS specific question: to remove surprises, should IPFS software support ipfs:// and ipns:// in addition to paths? We most likely need to support both, open question is what we should default in the docs.
• What to do in reference libraries? How to map namespace and identifier when URI is detected?
  • If we go with /uri/ prefix, no change is needed
  • If we have no prefix, and identify URIs by the lack of '/ in the front, need to solve mapping somehow:
    • (A) namespace would be uri and the identifier be opaque string with the original value
    • (B) set namespace to the URI scheme, and turn the reminder after : into a weird slice (//example.com)?

Value added

Things like this would be possible:

• data mirrors on IPFS path (backward-compatible)
• data mirrors on IPFS URI (improved UX for people copying and pasting)
• URLs pointing multiple HTTP servers (with optional SRI), other p2p systems such as hyper:// (without having to convert them to awkward paths)
• Magnet URIs could be used as-is
• URNs with BitTorrent Info Hash, additional hashes that could act as additional content routing hints / integrity checks

dnslink=/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
dnslink=ipfs://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
dnslink=https://example.com/file.zip#sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC
dnslink=https://mirror.example.org/file.zip#sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC
dnslink=urn:btih:c12fe1c06bba254a9dc9f519b335aa7c1367a88a
dnslink=urn:sha1:FINYVGHENTHSMNDSQQYDNLPONVBZTICF
dnslink=urn:md5:b1946ac92492d2347c6235b4d2611184
dnslink=hyper://hypercore-id+version-tag

[lidel]

@sjoerdvangroning @martinheidegger @thibmeu I want to add URI support, and suggest we go with (A).
lmk if you have any concerns / flags / questions around that approach 🙏

[martinheidegger]

As far as the dnslink=ipfs:// vs. dnslink=/url/ipfs:// question is concerned: I do not see any benefit of the /url/ prefix. All scenarios I think about just make it equally complicated and the ux of dnslink=ipfs:// is just better imo.

The biggest question to me is if ipfs://<key> should be assumed equal to /ipfs/<key> by the protocol.

Equal, automatic-mapping → /ipfs/<key> == ipfs://<key>

In this implementation it is possible to assume ipfs://<key> to be equal to /ipfs/<key> but not the other way around as the current <key> is not necessarily a valid URI sans protocol.

However, since the // is part of the URI authority and not the protocol, there are (as in your example) URIs without a // after the protocol. Should urn:<etc> be mapped to /urn/<etc>?

The easiest way I could see this implemented is that the spec treats <namespace>:[//]<key> as equivalent to /<namespace>/<key>.

Having an "URI → /namespace/key" mapping would make the APIs simpler. No need to change the current APIs and I kinda like that.

Note that this would mean that there is no guarantee that ipfs:// is actually an RFC 3987 URI.

Different, no-mapping → links + uris

Without a mapping it is loosing quite a bit of appeal to me.

Every API would need to be changed to support classic dnslinks and/or URIs and each namespace/implementation would need to to be changed to use the newly returned values.

const { links, uris } = await resolve('dnslink.dev')

This would open the doors to implementation inconsistencies down the line... not sure I like that. On the positive side, this could actually allow dnslink=<uri> to be an RFC 3987 URI and even return an URL javascript object.

[sjoerdvangroning]

I think option A is great and it would be great if dnslink would add this to the spec. This will increase acceptance, in short term for example for ipfs, hypercore users, but also enable shorter names for humans when hosting your data on very long URL/path (that is where we started from).

The future of DNSlink
When we can make DNSlink more widely adopted, it will become a very useful feature for future protocols. Ipfs and hyper could be faster adopted with the adoption of a namespace in the proxy. To be honest, with a number of browsers already supporting ipfs://, once Chrome supports it, the usage of DNSlink will drop fast within the ipfs community.

[lidel]

Seems that we agree (A) is the path forward.

Realistically, we won't be switching IPFS DNSLinks to URIs any time soon, but we should have an answer how to handle paths and URIs coexistence to avoid diverging implementations.

I feel "pragmatism > aesthetics" applies here. Interop is the priority.

The mapping proposed by @martinheidegger (<namespace>:[//]<identifier> → /<namespace>/<identifier> feels like a better option:

• Intuitive convention, URI scheme becomes namespace, and we get meaningful identifiers
• Works with all URIs (URLs, URNs)
• Could be implemented in existing libraries (JS, GO), creating "rule fo thumb" that removes ambiguity how to handle path with namespaces vs URIs
  • If someone wants the original URI, they can always access it via raw value:

    let { txtEntries } = await resolve('uri-dnslink.example.com')
    txtEntries === [{ value: 'foo://bar', ttl: 60 }]

If there are no strong concerns around this, should we open PRs?
I can bring this up during IPFS Implementers Sync on 15th before merging.

---

@sjoerdvangroning digression on the usage of path-based DNSLink:

To be honest, with a number of browsers already supporting ipfs://, once Chrome supports it, the usage of DNSlink will drop fast within the ipfs community.

Does ipfs:// adoption mean path-based DNSLink usage will be dropping? Maybe, but it won't happen any time soon. We have a vast ecosystem of software and services built on top of dnslink=/ipfs/cid records which must always work:

• it is used in browsers for human-readable names. Brave supports it. When you open ipns://en.wikipedia-on-ipfs.org in Brave, it will resolve DNSLink and load content using local gateway
• public gateways support it, allowing for loading content from multiple locations:
  • https://cf-ipfs.com/ipns/en.wikipedia-on-ipfs.org/
  • https://dweb.link/ipns/en.wikipedia-on-ipfs.org/
• services like Fleek and Cloudflare provide hosting and TLS cert for DNSLik websites backed by IPFS
• it provides interop layer for resolving names using alternatives to ICANN DNS
  • one can use it with things like OpenNic, ENS, HNS etc.

What I would like us to accomplish here is to allow URIs in addition to paths, so non-IPFS use cases can use URI-based addressing if they prefer.

[sjoerdvangroning]

@dnslink usage:
My mistake in thinking dnslink would be obsolete. Did not fully realise that next to the different protocol (ipfs instead of http), you also have a different identifier and the identifier is not directly supported or easy readable. Can't easily map an A-record to a CID.

Adapters
I am not a coder, what would be awesome if it is possible to plug in (new) protocols. Maybe call them adapters (?) (comparable to flysystem). Flysystem could use an ipfs adapter btw. So you would need an adapter for ftp, http, hyper, magnet etc. Adapter needs to be able to fetch content from an identifier and translate content so it can be served again via another protocol.

[sjoerdvangroning]

As an update, we finished the first project/poc and created SimplyDomain that works for https. This means that we are now able to create easy names for humans so they can host content on a much longer path.

Website: https://www.simplydomain.nl/

host www.simplydomain.nl
www.simplydomain.nl is an alias for proxy.simplydomain.nl.
proxy.simplydomain.nl has address 167.71.67.112

host -t TXT _dnslink.www.simplydomain.nl
_dnslink.www.simplydomain.nl descriptive text "dnslink=https://simplydomain.dev.muze.nl/"

We are now looking to a follow up project where we (finally) will support ipfs. If there is anybody that would like to help, give us a heads up as we could use some ipfs knowledge.

Also looking into additional added behavior:

• caching
• redirect (instead of proxy)
• wildcard support for SPA

Any more ideas, today is a good day for new ideas. Just throw it and we will see later what sticks.