Driving work on a preload spec

[guybedford]

I would like to reopen the discussion about preload metadata as mentioned in https://github.com/WICG/import-maps#supplying-out-of-band-metadata-for-each-module and previously discussed in #21.

From the comment in the README, the allusion is towards a general manifest here, as purely an illustration, something like:

<script type="preload-manifest">
{
  "https://site.com/path/to/module.js": {
    "integrity": "...",
    "preloads": [{
      "href": "./x.js",
      "rel": "modulepreload"
    }]
  },
  "https://site.com/path/to/x.js": {
    "integrity": "...",
    "credentials": "include",
    // eg dynamic import, alternatively some kind of priority system
    "lazyPreloads": [{
      "href": "import:dep",
      "rel": "modulepreload"
    }]
  }
}
</script>

Note that the "integrity" and other preload information does not associate with the dependency entry itself, but rather associates with the top-level entry. This is to avoid repeated information (and possibly conflicting information), in line with such a manifest being able to be a single source of truth.

In trying to specify something along these lines I'm hitting some conceptual issues, and would value some feedback on how such a thing might work:

1. Strictly speaking, having purely a URL as the key in the manifest is not enough information for a general manifest. For example, fetch('https://site.com/path/to/module.js') should probably not trigger the preloading. That is, the preloading information needs to apply alongside the type of interpretation, currently captured by the "rel" or "as" attributes. Since the same resource may have multiple representations / loading profiles depending on how it is loaded. It should probably be something more like:

<script type="preload-manifest">
[
  {
    "url": "https://site.com/path/to/module.js",
    "rel": "modulepreload",
    "integrity": "...",
    "preloads": [...]
  },
]
</script>

This way each entry can participate in the preload system corresponding to its correct interpretation directly. At this point the manifest starts to look pretty convoluted though. Is there a solution to this problem?

2. Dependencies like import 'dep' can only use an href attribute to reference their dependency that is resolved through an import map when we assume support for import:dep URLs. These import: URLs are effectively a prerequisite for such a manifest. Are we sure this syntax will continue to make progress?

3. No one is currently working on a specification for this feature that I know of, and that is probably the biggest issue of all. I have had discussions with various browser implementors and spec authors over the past year and I found there to be little interest in this work at a general level. How can we bring together the interested parties to start this spec work? In terms of trying to scope the work - would this work be seen as an extension of the preload sections of the HTML specification? Or perhaps something that integrates into the fetch specification? Or even as its own specification entirely? I'd be happy to be involved in this process further, but there seems to be quite some uphill here since there are no browsers feeling direct pain points today that are solved by this feature. Just generating such interest is a job unto itself!

I'm opening this issue here because preload is very much related to modules and the design space of this specification, and even if this work is another spec process, getting feedback and some driving directions from this group seems the appropriate way to start along this path.

Thanks for suggestions in advance!

[MylesBorins]

Apologies in advance if I am missing anything, but I am curious if there Is there any overlap between the preload spec and existing work being done in IETF for web bundles

https://wicg.github.io/webpackage/draft-yasskin-wpack-bundled-exchanges.html (a part of web package https://github.com/WICG/webpackage)

[guybedford]

@MylesBorins I tend to think these optimization problems can be tackled by both a preloading technique and web bundles. Tying web bundles to being the only solution space seems somewhat restrictive to me. I'd like to avoid going down this rabbit hole much further if you don't mind, but "overlap" is just the word to consider here - web bundles are great for base application delivery, but aren't so great at dealing with possible duplicate content delivered when lazy loading different plugins / sections for an app that have shared depedencies, unless the server itself is performing active content negotiation, ruling out static server use cases.

[jkrems]

[...] unless the server itself is performing active content negotiation, ruling out static server use cases.

Even with a smart server, the server can only prune already loaded (or loading!) resources from the package if it actually knows with confidence what to prune. Cache digest may be able to help but it seems pretty risky to put all eggs into that basket. There are failure cases that seem pretty fundamental and are hard to rule out. It seems unlikely that such an approach could ever match a smart client that knows the transitive set of resources and can actually make an informed call on which ones to fetch.

Web package could still be used to batch those resource fetches, of course. Either through a service worker or a potential future spec for official batching. But it doesn't replace the need to determine the dependency set (or a reliable exclusion set) on the client.

[sebinsua]

@guybedford I have a question.

I was told that using import-maps makes "preloading (both link rel and the preload scanner) [...] impossible since it can't take this information into account".

Is that one of the key reasons for this spec?

(I also commented a bit futher in the now-closed Conditional package maps issue. I was originally discussing using feature detection to load the correct syntax packages when I was told that this would make preloading impossible.)

[guybedford]

@sebinsua yes exactly, the depcache proposal is designed to allow integration with the browser HTML preload mechanisms for all module graphs under import maps.

[guybedford]

SystemJS is going ahead with the "depcache" import map field proposal for preload hinting to the engine here, because this is a needed feature for our users.

I have moved the specification work to https://github.com/guybedford/import-maps-extensions#depcache to have a place to bring future discussions on this feature together given this repo is not pursuing it currently.

This way SystemJS can continue to solve its own problems while still trying to create specification bridges for discussion, and I'm always open to feedback further here.

[denghongcai]

Alibaba use a format called 'seed' for a long time, seems solve the problem. http://g.alicdn.com/code/npm/@ali/pcom-dynamic/2.0.6/seed.json . any inspiration?

[domenic]

Let's keep this discussion in https://github.com/guybedford/import-maps-extensions.