[4.0] Proper cache invalidation of the static assets [.js/.css]

[dgrammatiko]

Pull Request for Issue # .

The problem

For some years now Joomla has a mediaversion PHP function which essentially creates a hash from the Joomla version. This was/is essential the weird part at the end of all the js/css files (?8567748356) and the reason for its existence was/is to invalidate the browser cache (the filename is the same but somehow we need to tell the browser to re fetch the content because there's a new Joomla version so the file might have been changed). But the process is extremely inefficient because FOR EVERY JOOMLA UPDATE WE INVALIDATE ALL THE STATIC ASSETS whether there was a change or not!

The solution

Since the introduction of Web Assets from @Fedik we have a registry where we can declaratively set some attributes. Amongst those attributes is also the version and this has been already mentioned by me a year ago when I was building the first truly SPA/PWA template: dgrammatiko/sloth@f315111 Nobody picked the code to do a PR so here I am doing it myself

Summary of Changes

• Add a script that will add a version attribute in every asset in the joomla.assets.json (only in the media folder)
• Move some code to utils (DRY)
• Fix the bootstrap ES5 (the commonjs plugin was missing from the rollup config)

Testing Instructions

• This PR needs CLI commands - NPM
• Apply this PR by git or by downloading the package
• Run npm install or npm ci
• Copy the file media/system/joomla.asset.json to media/system/default-joomla.asset.json
• Run the command npm run versioning
• Compare the 2 files and confirm that a version attribute is added in every style/script asset
• Check that the actual hash in the file is used in Joomla

Actual result BEFORE applying this Pull Request

    {
      "name": "core",
      "type": "script",
      "class": "CoreAssetItem",
      "uri": "system/core.min.js"
    },

Expected result AFTER applying this Pull Request

    {
      "name": "core",
      "type": "script",
      "class": "CoreAssetItem",
      "uri": "system/core.min.js",
      "version": "8f1a05a7e6e6fad83d22a4a5b67bafbb04ddc875"
    },

Documentation Changes Required

This basically as the gzip command affects only the release leader or anyone that will create a Joomla release but I guess this needs to be added in the steps for creating a release

@wilsonge

TODO

• Create the appropriate Joomla.assets.json files for all the modules
• Create the appropriate Joomla.assets.json files for all the plugins

[Stuartemk]

@dgrammatiko
Maybe for ...

Create the appropriate Joomla.assets.json files for all the menus

Create the appropriate Joomla.assets.json files for all the templates

Create the appropriate Joomla.assets.json files for all the componentes

Create the appropriate Joomla.assets.json files for all the categories

Create the appropriate Joomla.assets.json files for all the category

Create the appropriate Joomla.assets.json files for all the article

Create the appropriate Joomla.assets.json files for all the articles

Create the appropriate Joomla.assets.json files for all the languages

Create the appropriate Joomla.assets.json files for all the hashtags

Create the appropriate Joomla.assets.json files for all the contents

Create the appropriate Joomla.assets.json files for all the users

Create the appropriate Joomla.assets.json files for all the fields

Create the appropriate Joomla.assets.json files for all the contacts

Create the appropriate Joomla.assets.json files for all the banners

Create the appropriate Joomla.assets.json files for all the weblinks

Create the appropriate Joomla.assets.json files for all the extensions ... Etc.

[Fedik]

hm, by default version is auto, that means it should attach a random hash already

[dgrammatiko]

hm, by default version is auto, that means it should attach a random hash already

Yeah, it uses the PHP mediaVersion method but this method is tightly coupled to the Joomla version so all the assets will be invalidated in each Joomla update. With this approach the hash is generated from the actual content of the file, thus it will only change if the content of the file changed, win.

@Fedik this doesn't need some special code on the PHP side, it works fine already

[dgrammatiko]

@Stuartemk this PR covers ALL the core Joomla components and the system assets. It doesn't cover Modules and Plugins as there are not asset.json there yet. Also, the templates will be covered once we transition them to the Child mode, I'm working on it

[Fedik]

Create the appropriate Joomla.assets.json files for all the ...

it does not required, if the plugin or module have only 1 css/js, it does not make much sense, I guess

[dgrammatiko]

it does not required, if the plugin or module have only 1 css/js, it does not make much sense, I gues

Well, I disagree, the point is that if you have a joomla.asset.json then you can have a version that is derived from the actual content. In short, you have way better cache invalidation than invalidating everything on each Joomla release. That's the whole point of this PR, improve the static assets cache invalidation

[brianteeman]

Can you convince me please on the benefits of this PR over the current method. Is it really such a bad thing to revalidate assets on each release.

[dgrammatiko]

Is it really such a bad thing to revalidate assets on each release.

Think a bit about the global audience of Joomla here.
So, for a returning user (let's say the site is a news site) that is on a cheap mobile phone with limited storage and on a data plan in a country that data is super expensive (Africa, etc) invalidating cache has 2 negative effects:

• unnecessarily consuming storage on a low storage device
• unnecessarily consume more bandwidth

There is even a case for the privileged 1st world users: roaming
Anyways, if you check in J4 the jQuery or any other vendor script you'll see that they have their respected npm version after the question mark. This PR is doing the same, without the need to publish each and every asset to npm 😉

[brianteeman]

unnecessarily consuming storage on a low storage device

I suspect that thinking the browser cache will maintain assets for such long periods of time is unlikely - but I have nothing more than a feeling to back that up - for the exact reason you state about consuming storage

Plus thats only an argument for applying this to frontend assets and not all assets.

unnecessarily consume more bandwidth

Wouldnt using the cdn version of an asset be even more efficient. So you only ever cache one version of a jquery?

Having said all of that - this PR works and does as it says

[brianteeman]

I have tested this item ✅ successfully on 04ab752

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32485.}

[dgrammatiko]

I suspect that thinking the browser cache will maintain assets for such long periods of time is unlikely

Actually, the browser is quite dumb in this case and will try to hold on the file as long as the server tells it. So, if you have ExpiresByType text/x-javascript "access plus 1 year" it will cache it for 1 year, in the mean time you might get 15 new versions of Joomla so you have 16 copies of the same file but only one will ever be used.

Wouldnt using the cdn version of an asset be even more efficient. So you only ever cache one version of a jquery?

That's not the case since spectre and meltdown. Browsers now have double keyed caches per domain. If you have a script src to a cdn for jquery in your site and also joomla.org has the same script tag these two sites will never share the same cached file due to potential leaks cross-domain (security is hard). If you want to read more this is a good resource: whatwg/fetch#904 (comment)

[brianteeman]

That's not the case since spectre and meltdown

Oh I didnt know that - thanks for the link and info

[alikon]

I have tested this item ✅ successfully on 04ab752

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32485.}

[alikon]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32485.}

[wilsonge]

Can we not run this in the build.php file like we do the gzip command? seems like a relatively safe thing to do. I mean it's slight overkill as in theory it wouldn't be required but shouldn't cause any issues.

[dgrammatiko]

Can we not run this in the build.php file like we do the gzip command?

Sure. So basically there gonna be 3 node scripts that will run before the build.php (in this order):

• collect all the files folders for the static assets that need to be deleted with the new version (this is not written yet, but I wrote some ideas here: [4.0] separate data/logic #32715)
• compress the js/css
• version the entries on the Joomla.asset.json files

[rdeutz]

Can we not run this in the build.php file like we do the gzip command?

@dgrammatiko do you want to add this here or do another PR, if you will do another I am going to merge this one

[dgrammatiko]

@dgrammatiko do you want to add this here or do another PR

No, there are a couple more PRs pending for the internal tracking of the static assets but those will be separate PRs