[4.0][RFC] Fix CSP and apply deferred attribute to all scripts (with exceptions)

[dgrammatiko]

Pull Request for Issue # .

Summary of Changes

• Apply nonce to all inline scripts and styles
• Introduce the type="module" nomodule for scripts (eg load ES2015+ for new browsers or legacy js for old browsers)
• Defer all scripts (unless an attribute data-joomla-no-defer exists or the script has an attribute async, or an attribute type=module which is deferred by default)
• To defer the inline scripts the contents are base64 encoded and assigned to the src attribute, keep reading...
• This is fully backwards compatible!!!

The CSP problem

First and mostly lets thank @zero-24 for implementing the endpoint and the headers part for CSP and @wilsonge adding the nonce attribute for the inline scripts/styles. That said the CSP, as is right now` is totally broken even when someone will use the Joomla API and here's the proof:

• add these lines into atum/cassiopeia index.php:

$this->addCustomTag('<script>
if (window.jQuery) {
	console.log(\'hello\')
}
</script>');
$this->addCustomTag('<style>
:root {
    --atum-sidebar-bg: red;
}
</style>');

• The expectation is that since I'm using the API the style/script would have a nonce attribute but they don't and thus the CSP is BROKEN! To take this one step forward try to insert the following code into any layout (component/module/JLayout)

<style>
:root {
    --atum-sidebar-bg: red;
}
</style>
<script>
if (window.jQuery) {
	console.log(\'hello\')
}
</script>

Once again the CSP is broken as there's no nonce attribute

• The proposed fix: regex sales/scripts in the component/modules renderers and apply the missing attribute nonce.

The script defer

Some background (very brief) information on the scripts loading (read more on https://flaviocopes.com/javascript-async-defer/ )

• Joomla 3.x has all the scripts (appended using the API) in the head of the document resulting into blocking the parser:
  

• Joomla 4.0 moved the code to the body end (breaking B/C and possibly any inline script that exists into any component/module content). Although it doesn't pause the parser it introduces another problem, scripts starts downloading late and maybe behind fetches for images and other not critical assets
  

• The proposed solution: defer all scripts but keep them in the head (or wherever the dev injected them)
  

The last one is not the first time tried for Joomla 4.0 (eg: #22460 ) but that PR had side effects, notably the inline scripts inside any content area. So the solution is a two fold here, since fixing the CSP requires adding the nonce attribute we can add also a defer attribute. Unfortunately this will not work for inline scripts since the browsers behaviour (unless type=module) is to parse and execute immediately. We will bend things a bit here by appending the content of the inline script to the src attribute and to do that we will use the very well known/supported way of encoding the data: $attribs['src'] = 'data:text/javascript;base64,' . base64_encode($content);. That's it. Scripts get to load the new ES2015+ code for newer browsers, have a fallback for IE11 or other legacy, and the inline scripts are properly deferred.

Things that need a decision

• The name of the attribute that will bail out from this behaviour (atm data-joomla-no-defer
• Shall the templates have the ability to skip the parsing of the components/modules for CSP/deferred scripts? atm I left a snippet there of what that could look like:

if ($template->params->getBool('joomla_skip_assets_processing_modules', true))
{
   $module->content = $this->fixAssets($content);
}

Testing Instructions

Follow the description above

Actual result BEFORE applying this Pull Request

Lighthouse really bad results

Expected result AFTER applying this Pull Request

A bit better Lighthouse results, but there's a lot more work to get to the green fields...

@zero-24 since I know that you're in charge of the Lighthouse insights integration could you please ask Paul Irish to comment on this proposal?

Documentation Changes Required

Probably

[brianteeman]

The name of the attribute that will bail out from this behaviour (atm data-joomla-no-defer

Seems sensible suggestion assuming that joomla is needed as some sort of unique identifier.

[brianteeman]

but jquery

[dgrammatiko]

but jquery

As I said this is 100% backwards compatible (jQuery or even Mootools would be happy. I won't if I see them, but we all know I'm a special case 😃)

Edit: By the way, although this is far for an actual PR the code is fully functional, try it if you don't believe me

[dgrammatiko]

@Fedik can you provide some feedback here? Can we expand the schema to have a module entry so we don't have to use expensive IO checks?

[Fedik]

Can we expand the schema to have a module entry so we don't have to use expensive IO checks?

hm, what exactly do you mean?

About PR. Sorry, it no go.
First because of the output parsing. Use such parsing to fix someone's crap, it is to expensive, if it not work it should not work.
And I have added defer already where it possible in assets, left only jquery and core.js.

The chance to make all defer is lost. It was needed to make a year ago, when it was suggested. The extensions developers would follow.
Now it to late.

[dgrammatiko]

First because of the output parsing. Use such parsing to fix someone's crap, it is to expensive, if it not work it should not work.

I don't like this as well but it is needed not only for defer but to make sure that CSP will work as intended. If we cannot guarantee that CSP won't be broken for styles/scripts inside the body tag of a component/module then we should probably roll it back because he number of extensions that adding inline scripts/styles without using the API is too big to be ignored.

The chance to make all defer is lost. It was needed to make a year ago, when it was suggested

But this RFC goes a bit further it also introduces type=module/nomodule which is the norm these days. Sooner or later this will have to be applied (the number of js scripts that aren't a module these days is shrinking dramatically, only legacy/abandoned scripts are available only as iife)

[Fedik]

is needed not only for defer but to make sure that CSP will work as intended

That why I said, if it not work then it should not work. If someone add script with addCustomTag or other non standard way, then CSP should throw an error and skip that script.
If the script added in the editor, then probably better to make some check while content.prepare.

But this RFC goes a bit further it also introduces type=module/nomodule which is the norm these days

You already can use module/nomodule, it just a "type" attribute of the script tag.

[dgrammatiko]

You already can use module/nomodule, it just a "type" attribute of the script tag.

Ok, I think I didn't explain well my intend. Files with .es6.js should have a type=module and also the same script (the plain .js) should also appended with a nomodule attribute.

If the script added in the editor, then probably better to make some check while content.prepare.

That will require some kind of db migration for anyone updating from J3. But yes I would also prefer to spend the extra time on the form save rather on each page cycle.

[Fedik]

Files with .es6.js should have a type=module and also the same script

I think we should not do such assumptions, developers should be free to use any filename for any kind of script. It is up to them use or not use es6 es10 etc.
I do not think such "magic" will help, it may have opposite effect and confuse people even more ;)

That will require some kind of db migration for anyone updating from J3.

No. It also will be parsing (as you made here), but only for the editor content, similar to loadmodule and emailcloack

joomla-cms/plugins/content/loadmodule/loadmodule.php

Line 36 in 55f78da

public function onContentPrepare($context, &$article, &$params, $page = 0)

.

[dgrammatiko]

No. It also will be parsing (as you made here), but only for the editor content, similar to loadmodule and emailcloack

Sorry I fail to see what's different using the plugin hook. I mean essentially this line

joomla-cms/plugins/content/loadmodule/loadmodule.php

Line 81 in 55f78da

$output = $this->_load($position, $style);

is effectively getting the output from the module renderer (or I got this totally wrong)

I think we should not do such assumptions

It's not an assumption. The javascript group intentionally named the modern js as .es6.js so we are not assuming anything. If any 3rd PD needs to follow another path they could use another naming convention. This whole idea to attach type=module to any es6.js file is all about serving those scripts, right now they are just pure overhead. Also if we are about to use type=module then we need to reassure that all other scripts are deferred or things would just break bad. Thus the whole idea of moving the inline scripts to src=base64, etc.
I mean if we're not about to use them why did we spent all that time to create them in the first place?
Anyways I think you're against this

[Fedik]

I mean essentially this line

Yea, that line just renders the module for the given position.
I just meant whole onContentPrepare event, where you can run parser:

if (strpos($article->text, '<script') !== false)
{
	// Run CSP fix for $article->text;
}

The javascript group intentionally named the modern js as .es6.js so we are not assuming anything

That is a group assumption ;)
You know it, I know it, but people outside of the group may not understand.

I mean if we're not about to use them why did we spent all that time to create them in the first place?

That a good question ;)
Well, I thought they will be used "as is", but then all stuck somewhere.

But that time not lost for nothing, now we know how to write es6 code and stuff :)

[Fedik]

A note for whom may concern, about module/nomodule, it can be used like that:

...
{
  "name": "foo",
  "type": "script",
  "uri": "com_example/foo.module.js",
  "attributes": {
    "type": "module",
  },
  "dependencies": [
    "foo.nomodule"
  ]
},
{
  "name": "foo.nomodule",
  "type": "script",
  "uri": "com_example/foo.regular.js",
  "attributes": {
    "nomodule": true,
  }
},
...

In the view/layout :

$wa->useScript('foo');

In the result this will render both module and nomodule scripts.

[dgrammatiko]

@Fedik I think you're right, that would make sense a year ago when we were discussing the defer issue, now it's too late for such changes.

@wilsone please make sure that #31485 (comment) is documented in the assets manager documentation and also communicate somehow that styles and scripts that aren't introduced using the Joomla API will break CSP. The B/C break due to moving the scripts at the end of the page is also something that needs documentation...