Manifest processing should not be a function of document URL

[mgiuca]

The steps for processing a manifest are currently defined as a function of both manifest URL and document URL:

The steps for processing a manifest are given by the following algorithm. The algorithm takes a string text as an argument, which represents a manifest, and a URL manifest URL, which represents the location of the manifest, and a URL document URL.

This means that the entire identity of the app is uniquely determined by the tuple (text, manifest URL, document URL). Though practically, since text can be derived from manifest URL, it means just the pair (manifest URL, document URL).

The problem is document URL. The app can be interpreted differently depending on which document URL the manifest was referenced by. In reality, the document URL is only used in two places:

1. As the default value for "start_url".
2. To check that "scope" is same-origin as document URL.

I maintain that the processing of a manifest should not be a function of the document URL. This means the app will potentially behave differently depending on which page the user was on when they installed the app, and we cannot meaningfully index or install an app from outside of a document (as Microsoft is currently doing).

We should:

1. Remove the default value for "start_url", and require "start_url".
2. Move the check that "scope" is same-origin as document URL outside of the steps for processing the manifest (so if that check fails, it means that the manifest is not valid for that page, as opposed to being an invalid manifest).

This means that the semantics of the manifest becomes a pure function of the manifest URL itself (and of course the manifest text), and thus it will behave the same no matter where you install it from.

Making the above change 1 will technically be a breaking change, but since sites can't rely on the default "start_url" (since it's dynamic) this should not be able to break any expectations.

[ithinkihaveacat]

I think it would be useful to determine how common "versioned" manifests are before making this change, since this will cause problems for existing sites.

One example of a site that will be affected is weather.com, which has manifests at "versioned" URLs such as https://weather.com/weather/assets/manifest.507fcb498f4e29acfeed7596fe002857.json. If the manifest URL uniquely identifies a PWA then a "new" PWA will be created every time weather's manifest is deployed or moved.

More generally, I'm concerned that the realization that different manifest URLs correspond to different PWAs is not obvious. Also, no manifest validator/linter will be able to identify the problem with complete accuracy, and even heuristic based approaches are awkward (see e.g. GoogleChrome/lighthouse#2570).

The failure mode is rather unfortunate too: sites unveiling a new PWA at an (incorrectly) versioned manifest URL will only discover the problem when the manifest is updated and deployed, a situation QA processes are likely to miss.

[mgiuca]

There are two problems. The versioned manifest is not the problem that we're dealing with here.

• Problem 1: Sites change their manifest URL every time they update. This means they effectively are publishing a new app every time they update.
• Problem 2: Sites do not specify start_url in their manifest. This means effectively you get a different app depending on which page you were on when you installed the site.

One example of a site that will be affected is weather.com, which has manifests at "versioned" URLs such as https://weather.com/weather/assets/manifest.507fcb498f4e29acfeed7596fe002857.json. If the manifest URL uniquely identifies a PWA then a "new" PWA will be created every time weather's manifest is deployed or moved.

This is already a problem (Problem 1), and my proposed change doesn't make it any worse. Right now, an app is uniquely identified by the tuple (manifest URL, document URL), so if the manifest URL changes, it's a new app. My proposal is to remove document URL from that tuple. It will still be a problem that if the manifest URL changes, it's a new app.

I don't want to talk too much about Problem 1 here. I agree it's a problem but we're talking about it elsewhere (GoogleChrome/lighthouse#2570, GoogleChromeLabs/gulliver#323).

For Problem 2: I noticed weather.com also doesn't have a start_url, which means according to the manifest spec, it should create a unique app for each page of the site. However it looks like Chrome, at least, considers the manifest invalid if it doesn't have a start_url:

"Site cannot be installed: the start URL in manifest is not valid"

(For my future reference, the Chromium code which does this is in installable_manager.cc where it sets START_URL_NOT_VALID).

Given that this is the case, it seems it would not be breaking any existing expectations if we made start_url mandatory in the manifest spec.

[kenchris]

I am OK with this change. I don't think this should be any issue for the Windows support either @boyofgreen

[mgiuca]

I must imagine (without checking) that Windows support also ignores both of the rules relating to document URL, since they wouldn't have a canonical document URL for their Microsoft Store PWAs. So this change would only be making the spec more in line with Microsoft's (and Chrome's) implementation. @boyofgreen

[marcoscaceres]

I'd be in favor of this change. @mgiuca, could you draft it up?

[mgiuca]

Working on it!

[ithinkihaveacat]

@mgiuca Ah, I didn't realize weather.com was no longer triggering any sort of A2HS prompt; that definitely worked in the past (they were showcased in the PWA tent at Google I/O last year) but either Chrome's behavior has changed, or start_url has been removed from their manifest such that it no longer triggers.

In any case, I agree with this change, and I take your point that this issue mostly separate to the question of how PWAs should be uniquely identified (#586).

[mgiuca]

I've put up a branch (but I can't make a PR until #670 lands; we should continue discussion on #669):
mgiuca/manifest@start-url-required...no-document-url

This one is also a breaking change and may require some discussion. The break is that before, if the start URL did not match the document URL, the start URL (and likely scope) would be set to their default values, and manifest succeeds. Now, the entire manifest fails to load (in the "obtaining a manifest" steps).

This is necessary in order to make the same-origin-as-document check not intrinsic to processing the manifest (allowing the manifest to be processed independent of the document), but rather make it just part of the process of loading a manifest from an HTML page. We can't really just alter the start_url and scope after the manifest is processed; then we're back to the manifest values being dependent on the document URL.

I wonder what Chrome for Android does in this case (since the WebAPKs are built from the manifest, independent of the document URL, they clearly don't alter them to a default start_url if the origin doesn't match --- I wonder if they fail or if they simply allow a cross-origin PWA installation).

[mgiuca]

This (the dependence on the document) goes up a bit higher than I previously thought. I'm trying to update the spec to basically allow a web app to be installed directly through its manifest URL (not from within a document context at all), which is how Microsoft Store PWAs work, as well as Google's WebAPKs, and any future "install app directly from a store page" type of installation. Currently, this isn't really allowed.

In addition to the "steps for processing a manifest" (fixed above):

• The "steps to install the web application" uses:
  • The Window, to fire the appinstalled event (this isn't needed if there is no document).
  • Optionally using the Document to get metadata if there is no manifest (this is irrelevant when installing an app with a known manifest).
  • Registering a Service Worker with the Document (this is the hardest thing to fix, but it can surely be done against the start URL or scope instead?).
• The "steps for obtaining a manifest" uses:
  • The Document, for obtaining the manifest URL (this isn't needed if we're installing directly from a manifest URL).
  • The Document's origin (terminate if it is an "opaque origin") (this isn't needed if there is no document).
  • The Document, to check whether it is same-origin as the start_url (this isn't needed if there is no document).

I think the above algorithms can be rewritten in the following way:

1. The "steps for obtaining a manifest" should be split into two: one for obtaining a manifest URL from a document, and one for obtaining a manifest from a manifest URL.
2. The "steps to install the web application" should similarly have two versions: one that is applied within a top-level browsing context, and one that is given a manifest URL as input.

These direct-install user agents can thus use the non-document versions of these algorithms.

[benfrancis]

I don't know if that is entirely true

I agree that the current manifest specification assumes installing a web app from a document but that was a design choice, it didn't have to be specified that way. That design choice unfortunately means that some implementations (e.g. app stores, mobile device management, kiosks) which already use web app manifests today can never be compliant with the specification.

should the Working Group concern itself with anything but what web browsers do with the manifest?

That's a good question. The web app manifest specification differs from other web specifications in that by its nature it can not be limited to just browsers, because it can not be implemented entirely inside a browser engine. It requires interactions between a browser engine, browser chrome, the underlying OS and in some cases external services like app stores and sync services.

My understanding of Progressive Web Apps is that they are all about web apps escaping web browsers. Restricting the scope of the specification to the context of a web browser would seem to fail to meet that primary use case.

I therefore agree in principle with the statement "manifest processing should not be a function of document URL", though in practice untangling the two at this stage would be challenging. Proposals like the proposed approach to unique IDs would further complicate this.

[aarongustafson]

This may be a naive question, but would it be possible to make the document optional so that the parser takes it into consideration if it's available but doesn't if it doesn't exist? Would that be too complicated?

[aarongustafson]

Also… in store/catalog contexts, I would think a start_url would be required, else the manifest would not be tied to a URL at all, right? We could make that a requirement in App Info if folx think it’s worth it.

[marcoscaceres]

This may be a naive question, but would it be possible to make the document optional so that the parser takes it into consideration if it's available but doesn't if it doesn't exist? Would that be too complicated?

The parsing aspect is not complicated, no. It's all the stuff that happens around it that gets complicated. That's why we presume this is coming form a document: because link relationships are from Document (and security/environment) -> linked thing (manifest), but never treated as independent.

We could make that a requirement in App Info if folx think it’s worth it.

Yes. I might make sense to do that. The stores can enforce that policy, while advising developers to also treat a manifest as an totally independent resource (not as a link relationship of a document).

[marcoscaceres]

@benfrancis wrote:

My understanding of Progressive Web Apps is that they are all about web apps escaping web browsers. Restricting the scope of the specification to the context of a web browser would seem to fail to meet that primary use case.

Note that this effort predates that write up, and I don't think there was every wholesale endorsement of PWAs as outlined in that blog post by the standards community (or browser vendors at large). As aspirational a guide as that blog post has been to the community, there are a lot of dissenting opinions, and in some cases, outright rejections, of what is proposes.

Without digressing too much, from the very beginning, the way we designed the "progressive" nature of this particular specification was that developers could add as few or as many manifest members as they wanted in their manifest, without any strict requirement on what they must include.

I therefore agree in principle with the statement "manifest processing should not be a function of document URL", though in practice untangling the two at this stage would be challenging. Proposals like the proposed approach to unique IDs would further complicate this.

Agree. And at this point in this nearly decade long journey, I'd really just like to get the unified base set of functionality standardized. It is, admittedly, a somewhat impoverished subset of the functionality web developers need - but I'm optimistic that we can continue to evolve this specification, along with implementations, in unison.

But I really just want to get the core of it "done" - so we can then incrementally standardize little enhancements to make everyone's lives better.

[garygreen]

@mgiuca

Installation without a document, e.g., from an app store. There is no document URL.
The spec currently doesn't actually allow the second scenario at all, so it doesn't reflect reality.

The reality is a document is required to install an app. Sure, you can INDEX an app (it's name, icons, etc) but to me that is not INSTALLING an app.

The installation of an app requires a document. It often requires a service worker, fetching of js, css, caching pages, numerous other stages during the install lifecyle of the service worker.

To really separate the document from the installation process - wouldn't the manifest file need to link to all the assets needed to install the app? i.e. is this issue trying to help address packaging of an app, and once packaged not really "needing" a document any longer? It seems to me the fact remains a document is needed to practically package and install an app. You could "index" it, but what does that really give you? You've indexed a bunch of icons, name, but not really done anything practical.

A document is an integral part of the manifest to describe where an app sits, how it can be accessed and installed? Maybe I'm misunderstanding the intention here though.

[aarongustafson]

The reality is a document is required to install an app. Sure, you can INDEX an app (it's name, icons, etc) but to me that is not INSTALLING an app.

The installation of an app requires a document.

I was thinking about this very thing. For digital catalogs, the start_url is really required because otherwise there’s no "there" there; you wouldn’t even have a domain to work from.

So, to go back to @mgiuca’s initial recommendation (and this PR), I’m all for making start_url required in Manifest (with a sensible upgrade plan in place, as Matt’s outlined in the PR). Moreover, I feel like the start_url (even the inferred one) should step in as the "document" in order to facilitate installation (see @marcoscaceres’ comment about CORS, etc.).

[marcoscaceres]

So, to go back to @mgiuca’s initial recommendation (and this PR), I’m all for making start_url required in Manifest (with a sensible upgrade plan in place, as Matt’s outlined in the PR).

I personally don't agree with making it required, for the reasons I already stated. But I'd be totally ok with making start_url and scope RECOMMENDED or including them as a "best practice".

[marcoscaceres]

I'm revising this again, and, as the spec level, I just don't think it's possible to separate them right now.

That's not to say that auxiliary crawlers, app stores, and other software can't treat the manifest as independent (they can and should, and that's fine); but at a spec level (and for the purpose of conforming "user agents") the manifest is inherently tied to its client window and document for the reasons I mentioned above.

Let's continue to stick to "browser land" for the purpose of conformance, as otherwise the scope of the problem becomes too large.

We can continue to revise this after we at least standardize how the three existing engines handle web manifests.

[mgiuca]

I want to re-open this as it's come up internally (at Google). I wrote a doc explaining the rationale, the proposed change, and potential issues. It's quite messy at the moment though including a bunch of unrelated changes but I wanted to restart this conversation.

https://docs.google.com/document/d/1NUsoTWkWrPFFNnxR00r1NF-MXzq-_rjl2ldAhZm-TEw/edit

I will follow up with a less messy proposal after a bit of informal discussion. (I'm also not 100% convinced that we should do this any more after @philloooo pointed out that from a security perspective, we don't necessarily want manifests to be able to "appify" a site on another origin without that site linking back to the manifest. But if we decide not to do this on those grounds, we can properly document that as I don't think this objection has been raised before.)

[benfrancis]

I'm pleased to see this reopened because this is still my biggest gripe with the Manifest specification. Not being able to install a web app directly from a manifest URL rules out lots of really interesting use cases (e.g. app stores, digital signage, mobile device management) where implementers have no choice but to either go against the specification or jump through strange hoops like generating fake documents or fetching and parsing an HTML document just to find a manifest URL.

To be honest I had thought that the id feature landing was the final nail in the coffin for separating manifest processing from document URL. As I understand it, the document URL is used in the default value for id if no start_url is provided.

I hope that you can untangle this in Chromium, because being backwards compatible with Chromium desktop historically using start_url as an identifier (and in turn document URL being the default value for start_url) has resulted in a huge amount of (IMO) unnecessary complexity in the specification.

If this can't be achieved, I wonder if it's worth considering a version 2.0 of Manifest which breaks backwards compatibility and allows the installation of web apps directly from a manifest URL (e.g. by using manifest URL as a default identifier rather than start_url, and possibly even requiring manifest URLs to be same-origin with start_url). This could drastically simplify the installation and update mechanisms and open up a lot of additional use cases.

[mgiuca]

This was discussed in a call yesterday with @marcoscaceres and @dmurph. The main idea of being able to process a manifest without a document is generally agreed as something we'd like to do, with good use cases behind it. But it's a little messy.

There was disagreement on what to do if there's no start_url. @marcoscaceres still likes the idea of it defaulting to the document URL, if there is one. I would prefer if we could move away from considering those apps without a start_url to be apps at all.

I think a reasonable compromise is to let the existing behaviour stand, but require start_url for new functionality. Concretely, let's continue to allow apps without a start_url to be installed from a document, but disallow them from being installed using the new "non-document-based" mechanism. In other words, you either need a document URL or a start_url, which means we don't need to answer the question "what if you don't have either a start_url or a document URL?".

Before we get to that, we need to refactor the algorithm so it takes URLs instead of HTML objects, so that it's at least theoretically callable without a document URL. That part is uncontroversial so I will get started on it now.

@benfrancis :

As I understand it, the document URL is used in the default value for id if no start_url is provided.

That's right. I think that, given the "start_url defaults to document URL", this is the right call: manifests that are missing a start_url essentially generate a different app for every page that you could install it from, therefore each of those apps should have a different ID.

I hope that you can untangle this in Chromium, because being backwards compatible with Chromium desktop historically using start_url as an identifier (and in turn document URL being the default value for start_url) has resulted in a huge amount of (IMO) unnecessary complexity in the specification.

I agree that it's created a bunch of unnecessary complexity. If I was designing it from scratch, I'd have made scope mandatory (or default to '/' on the same origin as the manifest URL), and make both start_url and id default to scope, and have nothing depend on document URL other than verifying that it's same-origin as scope. But for historical reasons, everything is based on start_url and that's OK. For developers, it's simple if you just supply an explicit start_url, scope and id - no need to worry what the defaults are if you explicitly supply everything.

If this can't be achieved, I wonder if it's worth considering a version 2.0 of Manifest which breaks backwards compatibility and allows the installation of web apps directly from a manifest URL

I don't think it's worth breaking backwards compat over this. It's a minor historical quirk that we can work around.

[mgiuca]

Another point: the most compelling reason we identified yesterday (I believe @dmurph raised) is privacy-sensitive: when performing these "automated" installs without user supervision (e.g., installing from sync, admin policy or a default app installed by the user agent), there is a privacy issue in sending a request to the server of every app being installed.

If we can install from a manifest without loading a whole document, it allows us to have the manifest body - and its icons - be the thing distributed (e.g. over sync) rather than its URL, and install it offline without hitting the server. (Unfortunately, this won't allow us to have a service worker cached, without Web Bundles, but at least we can have the app icon registered with the system.)

[mgiuca]

Spun off the first step in this process to a separate issue: #1068 (this doesn't remove the document URL, but it means we won't have to load the document before triggering install). We can continue to discuss making the document URL optional on this issue.

[marcoscaceres]

I think we addressed this in 22a0b1e

Closing, but @mgiuca reopen otherwise.

[benfrancis]

@marcoscaceres @mgiuca I note that the commit message in 22a0b1e describes that change as "pre-work" for this issue.

Since the algorithm for processing a manifest still takes document URL as an input, should this issue not still be open?

I'm also curious what the next step would be for separating these out, since the current specification is very much dependent on a document URL.

Perhaps a version 2.0?