[4.0] Updated files and folders deletion on updates and remove 3.9.x update SQL scripts

[richard67]

Pull Request for Issue # .

Summary of Changes

Updates the list of files and folders to be deleted to the changes since 4.0 Beta 5 and last changes in staging and 3.10-dev.

Add also files and folders having been forgotten with the release of 4.0 beta 2.

Changes comments for sections so they tell start and end point ("from" and "to") of the comparison.

Moved some Joomla 3.10 deletions form later sections to the first section and removed some duplicates.

Old J3 sample data images will not be deleted on update because they might be used in content, see discussion here #32744 .

The lists have been pre-generated with use of #25559 .

Order of folders within a release section changed to descending so it fits to what the tool produces because this makes sure the folders are delete from child to parent.

The /libraries/vendor/paragonie/sodium_compat/src/Core/Xsalsa20.php file is removed from this list of files to be deleted. This file has been renamed once from Xsalsa20.php to XSalsa20.php on Joomla 3 and is still present on Joomla 4 with the correct name. Having it in the list with the old name will cause it to be deleted on OS and files systems with case-insensitive file names. When using the script modified by PR #25559 , that file correctly doesn't appear on the list of files to be deleted.

In addition, this PR deletes from sources those new 3.9.x SQL update scripts which meanwhile have been merged up to 4.0-dev but should not be there.

Testing Instructions

1. Code review by an experienced contributor or maintainer who knows the recent developments in J4:
   Expand the changes in file script.php on GitHub and check that what has been added fits to the recent changes, like change of the build process for javascript or the change from Bootstrap 5.
   => Ping @dgrammatiko and @wilsonge .
2. Update test 3.10: Update from 3.10-dev or latest 3.10 nightly to the update package or custom update URL made by drone for this PR and compare the result on the file system with a new installation of current 4.0-dev or latest 4.0 nightly.
3. Verify that after the update from test 3 no file has been deleted which has been added to staging since 3.9.25 was released but which haven't been merged up to 4.0-dev yet.
4. Update tests 4.0 Beta: Update a 4.0 Beta 7 or an earlier 4.0 Beta to the update package or custom update URL made by drone for this PR and compare the result on the file system with a new installation of current 4.0-dev or latest 4.0 nightly.

Actual result BEFORE applying this Pull Request

After an update from 3.10 or a previous 4.0 Beta to current 4.0-dev or latest 4.0 nightly, there are lots of obsolete files and folder left compared to a new installation of the same current 4.0-dev or latest 4.0 nightly, lots more than the exceptions mentioned below in section "Expected result AFTER applying this Pull Request".

E.g. this gazillion of .es6. files and their minified and gzipped versions being obsolete after the change of the build process, or all that old BS 4 stuff which should not be there.

On the other hand there are 2 "3.9.x" update SQL scripts for each database type after an update from 3.10, which should be deleted but they aren't.

Expected result AFTER applying this Pull Request

After an update from 3.10 or a previous 4.0 Beta to the update package or custom update URL made by drone for this PR, there are only following old files and folders left compared to a new installation of current 4.0-dev.

1. Following folders including all sub folders and files are not deleted:

• /administrator/components/com_search
• /components/com_search
• /installation
• /modules/mod_search
• /plugins/fields/repeatable
• /plugins/search

2. Following files in other folders are not deleted:

• /administrator/language/en-GB/en-GB.com_search.ini
• /administrator/language/en-GB/en-GB.com_search.sys.ini
• /administrator/language/en-GB/en-GB.plg_fields_repeatable.ini
• /administrator/language/en-GB/en-GB.plg_fields_repeatable.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_categories.ini
• /administrator/language/en-GB/en-GB.plg_search_categories.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_contacts.ini
• /administrator/language/en-GB/en-GB.plg_search_contacts.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_content.ini
• /administrator/language/en-GB/en-GB.plg_search_content.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_newsfeeds.ini
• /administrator/language/en-GB/en-GB.plg_search_newsfeeds.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_tags.ini
• /administrator/language/en-GB/en-GB.plg_search_tags.sys.ini
• /administrator/language/en-GB/en-GB.plg_search_weblinks.ini
• /administrator/language/en-GB/en-GB.plg_search_weblinks.sys.ini
• /administrator/language/en-GB/en-GB.plg_editors-xtd_weblink.ini
• /administrator/language/en-GB/en-GB.plg_editors-xtd_weblink.sys.ini
• /administrator/language/en-GB/en-GB.plg_system_weblinks.ini
• /administrator/language/en-GB/en-GB.plg_system_weblinks.sys.ini
• /language/en-GB/en-GB.com_search.ini
• /language/en-GB/en-GB.mod_search.ini
• /language/en-GB/en-GB.mod_search.sys.ini

There might be other files present after an update from 3.10 which are not mentioned in the list above. In this case there are files which have been recently added to the staging branch and merged up into 3.10-dev but not have been merged up into 4.10-dev yet.

There are no "3.9.x" update SQL scripts anymore after an update from 3.10.

Documentation Changes Required

None.

[dgrammatiko]

I have diffs up to beta 7 (https://github.com/dgrammatiko/JfixMediaFolder/blob/main/diff_beta_6_7.json) and those are inline with your results. For the diff with the tip of the head I need to alter the script a bit (will do it tomorrow)

Can we also get a decision on the data separation? This script.php is becoming unreadable...

[richard67]

Can we also get a decision on the data separation? This script.php is becoming unreadable...

@dgrammatiko I'm not the guy to decide anything, that's George, I think.

I am in favour of data separation, but I'm not sure if we will have it ready before the next Beta.

So I thought I have to make a quick fix to be sure the next Beta will be ok.

[richard67]

@dgrammatiko I did use George's script in the following way to make this PR:

• I've used full installation packages for the comparisons.
• I've compared last nightly build of 4.0-dev with the one of 3.10-dev to get the changes from 3 to 4. I've noticed that comparing 3.10-dev with 4.0 Beta 1 like I thought once that it could be done would give false results, because new things added in 3.x and later being merged up into 4.0-dev will be added to the lists for removal when being compared with a 4.0 Beta 1.
• The result of the comparison from 3.10-dev to 4.0-dev I have added to the first section "// Joomla 4 Beta 1" in script.php.
• From this section I have removed duplicated which would be in the array because later being removed again with a comparison between 4.0 Beta x and 4.0 Beta (x+1). This is not really necessary, they only cause a bit longer run time, but they would become more and more if not removed. I have decided to remove duplicates in the 1st section "// Joomla 4 Beta 1", except of 3.x update SQL scripts, those I handle there and not in later sections.
• The later sections "// Joomla 4 Beta 1" and so on are easy. My results are as far as I could see the same as you have, but I haven't checked every detail yet.
• If not all has been merged up from 3.10-dev to 4.0-dev, it might be necessary to manually remove files or folders from the list, or to limit the comparison between last 4.0 Beta and the 4.0-dev branch to the last commit which the 3.10-dev and the 4.0-dev branches have in common, or something like that.

[dgrammatiko]

On a quick check, you have 4515 entries (5011 -496) for the 3x to 4.0-beta1 and I have 4833 files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and 1167 folders (you have 1171) https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json (the folders are off because the script is echoing all the nested folders) so I guess we are on the same boat here. A small detail my script is diffing the tip of 3.10 against the tip of 4.0-dev which is not ideal in this case

@wilsonge can you decide on the data separation here? It would make the checks here so much easier if we have to compare xmls or jsons... #32715 is already using XML files so the only question is if we want multiple files or the build tools could append to one existing file instead

[richard67]

On a quick check, you have 4515 entries (5011 -496) for the 3x to 4.0-beta1 and I have ...

@dgrammatiko What I was trying to explain above is that we shall not compare 3.x to 4.0-beta1, because they are 2 branches and things which are merged up go into the last 4.0 comparison, too. It is better if we compare 3.x with the end point of the 4.0 comparisons, e.g. current 4.0-dev or last nightly. For this reason I could have changed the comments "// Joomla 4.0 Beta 1" in script.php to "// From 3.x to current 4.0" or so, but I was not sure if this might confuse people.

[richard67]

@dgrammatiko You should also check wilsonge#60 and wilsonge#61 . We need that in your scripts, too:

• Update exclusion list by sample data images
• Handle list of files to be renamed (which is not the same as moving. Rename is just changing the base name, and here only regarding casing. Moving we don't need to handle.)

[dgrammatiko]

What I was trying to explain above is that we shall not compare 3.x to 4.0-beta1

Ok, so we are doing the same diffing then. If you have the raw results can you put them on a gist, it would be easier to check against my scripts, although this shouldn't be a code validation rather a consistency check (what needs to stay or not) which tbh I don't have the complete oversight on many parts

Handle list of files to be renamed

We can add another section in the xmls, eg:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<files>
  <filename>components/com_config/src/Model/CmsModel.php</filename>
  <folder>components/com_config/src/Model</folder >
  <rename from="media/vendor/skipto/js/skipTo.js" to="media/vendor/skipto/js/skipto.js"></rename>
</files>

https://github.com/dgrammatiko/JfixMediaFolder/blob/main/diff_beta_1_2.xml

[richard67]

@dgrammatiko Get my raw files plus my modified PHP script and my shell script as Zip from here: http://test5.richard-fath.de/j4-update-script-deleted-files.zip . Let me know if you got it so I can remove it from there.

[richard67]

@dgrammatiko I have a new version of the PHP and the shell script and my raw files: http://test5.richard-fath.de/j4-update-script-deleted-files_2021-04-19.zip . My latest changes on this PR here are based on that. When comparing J4 versions, files and folders will only be added to the lists if they are not already present in the lists from the 3.10 to current 4.0 comparison. This means I could use the raw files to make up the sections in script.php without making any manual modifications (except of removing stuff from the lists which is on the way being merged up from 3.10 to 4.0 soon).

[dgrammatiko]

@richard67 so the idea is to keep a registry (some ref files) in the repo? If so maybe it worth keeping the data in a .json file instead of .txt. After a quick check it seems fine to me. Maybe we should finally have a decision on the data/logic separation to finalise this whole part, so @wilsonge it's up to you I guess...

[richard67]

@richard67 so the idea is to keep a registry (some ref files) in the repo?

Not necessarily, but it could make sense.

[richard67]

@dgrammatiko @wilsonge I've just updated this PR to the latest upmerges so it is up to date and ready for the next beta or RC release. You can find the raw files, my modified PHP script and the shell script which I've used in that zip: http://test5.richard-fath.de/j4-update-script-deleted-files_2021-05-03.zip . Just check the shell script and you will see how it worked. The modified PHP script contains a change for an optional new parameter to check the files from the 3.10-to-4 comparison in addition and if something is in that file already, it will not be added to any of the 4.0-x-to-4.0-y files.

P.S.: I did not have to make any modifications on the raw files. I could copy their content as is to the corresponding sections in scrip.php. This worked because after the recent upmerge, the the 4.0-dev branch is fully up to date with the 3.10-dev branch.

[wilsonge]

Thankyou!

[richard67]

Thank you, too.