[3.9] purge update cache only on user request

[HLeithner]

Pull Request for Issue #23663 .

Summary of Changes

• Removed auto purge from model::findUpdates

Now we only purge if the user clicks the purge button in the backend.

Testing Instructions

• Check for updates on the dashboard.
• Check for updates in the installer component.
• Test this pr with version 3.9.3 to see if 3.9.4 is detected
• Test this 3rd party components
• Simulate updates by manipulating xml updates files

Cache should update after 1 hour.

Expected result

Doesn't check for updates in cache and updates after 1 hour

Actual result

Always checks for updates.

Documentation Changes Required

no

[mbabker]

This PR is conceptually wrong IMO:

• An end user should always be able to purge the update cache
• This changes the find updates controller logic to ignore the update caching mechanism, which means that all update servers will always be pinged when clicking the find button

[mbabker]

What is the issue you are actually trying to address? Because if I'm reviewing this diff correctly, what you're doing is essentially this:

• Removing purge button
• Making update.find task auto-purge every time it is executed (which is the toolbar button on the com_installer update view and the link shown in the alert the quickicon pops up on the control panel)

This PR does nothing to change the behavior of the control panel or the quickicon's AJAX call and greatly changes the behavior of the com_installer update view's actions.

[brianteeman]

Before removing the button you should read why it was originally put back into joomla by @nikosdion #3775 and the discussions below

[nikosdion]

Quick recap for those not reading my old PR's thread. If you try to fetch the update information of all extensions, every single time, you will end up with Joomla hanging since it will try to fetch too many things at once, taking too much time because you either have too many extensions or because the extension's update server does not respond / your site cannot connect to it. That's the reason each Update Site has its own "last checked" time. If you really want to force a full check you can use the Purge button. If something gets stuck and you only get halfway through fetching updates, go back and click that button again; only the Update Sites NOT already queried will be queried. So in the worst case scenario you have to repeat the process a few times to get over a problematic update server.

So, with this PR you always have to fetch all Update Sites in a single page load every time. This is wrong. It will end up not fetching any updates for a very large number of sites with numerous extensions. Every time you try and fail to fetch updates it will remove existing update information. This means that Joomla will be unable to fetch update information for Joomla itself (since the Update Sites mechanism is common for extensions and the core) which is a massive security issue. Ouch!!!

On top of that it is fundamentally wrong having Joomla ping the update server on every single update check request. Nevermind the abysmal performance or white pages. The bandwidth for serving those XML files is expensive. For example, I currently pay 150 Euros per month to serve the tens of millions requests to Akeeba Backup Core's update XML file. Your change increases that by something like 10x. I'd have to stop providing automatic updates outright or I'd have to provide my own (caching) updater like I was doing back in 2007 to 2012. Either way, that'd make it much harder for my users to know if there's an update available and install it. I wouldn't be the only developer with that issue; eventually everyone with a popular extension would have to discontinue updates through the core or charge a hefty per site fee for providing update information to recuperate the expenses. No, using something like a public GitHub repo is not a serious alternative as it has a bandwidth cap which is far lower than what a semi-popular extension requires. There's no good free alternative and the paid ones are too expensive. Do you think I like paying money if I can find something for free? LOL! But I digress. In the best case it takes us back 8 years, in the worst case we nullify the reason of having extension updates in the core to the detriment of our users, the very people we are supposed to serve.

I understand the reason why you think your solution is "smart": you are frustrated with the Joomla and extension updates not showing up in Joomla's backend the very instant they are published by the developer. You know what? It's not just Joomla itself and its extensions. You do not see updates immediately upon their release for Windows, macOS, iOS, Android or even Linux. Not for their applications either. There is a lag enforced by caching and other mechanisms which are meant to reduce both the overall traffic and the traffic spikes in the update infrastructure. So what Joomla is doing is not dumb; it's smart, it's common, it's best practice. Don't worry, it took me a couple of years after I started provided updates to realise why. In fact, it took a cock up in my updater back in 2011 --Joomla's extensions updater didn't support paid extensions back then-- which disabled caching and ended up accidentally DDoS'ing my server and costing me an arm and a leg in the 24 hours it took for everyone to update to the new version.

I very strongly downvote this PR. It's bad for security, it's bad for user experience and it's bad for developers of extensions that are basically anywhere in the top 100 extensions in JED.

[brianteeman]

This is why institutional memory is so important AND why I have repeatedly said for years every time we look at changing something we should look to see why it was introduced in the first place

[nikosdion]

100% agree, Brian. Git has a great feature called git-blame which lets people find the relevant commit which can lead back to the issue / PR that discussed the feature or change to a feature. This preserves institutional memory even if all people involved in a feature have left the project. That was the whole point of moving to GitHub, right? :)

[brianteeman]

it was by using git blame that I found the pr.

[HLeithner]

I'm an idiot I missed to commit the most important part of the commit, because since 2013 we always purge the update cache, so never ever in the last 6 years any request to any update server was cached.

[HLeithner]

@nikosdion I think with this PR I can save you 100 USD per month ;-)

[HLeithner]

I have no problem revert the purge button, but in fact if I press the "refresh" button in the backend my user experience is very bad because it doesn't refresh anything because it doesn't purge first so I have always press 2 buttons.

The cpanel uses the ajax task and this now uses the cache.

[brianteeman]

too tired from travelling to search right now but there was a very specific reason that we have two buttons

[mbabker]

The way the UI is designed right now, hitting the find button should in theory only trigger a refresh for expired cached update entries. Clicking purge then find will force all sites to refresh. Keeping separate tasks, and therefore buttons, is the correct way to go about things.

[HLeithner]

The way the UI is designed right now, hitting the find button should in theory only trigger a refresh for expired cached update entries. Clicking purge then find will force all sites to refresh. Keeping separate tasks, and therefore buttons, is the correct way to go about things.

ok that makes sense I reverted the purge button removal, only problem on this is that I think that all timeouts always have the same timestamp

[mbabker]

Not much you can do about that without a backend cron job continually refreshing things so the entire cache doesn't expire at once.

[HLeithner]

the cachetime decision should be part of the component update specification, so the component author can decide how often someone should check for updates.

[mbabker]

No, it shouldn't. At the server level, developers can (and should) implement HTTP oriented caching but they should not be specifying something in either their extension or update server manifest stating an update should only be checked at X frequency by Joomla.

[HLeithner]

Then the timeout saved per extension is useless because its always the same.

Edit: edit maybe except an update fails because of a timeout.

[mbabker]

That's a fault of nobody's on a site for several days and the entire update cache has expired (because updates aren't being queries with frontend requests unless you've got a plugin doing that or running a cron job). That's not a system design bug. That's almost like saying turning on Joomla's caching for the default 15 minutes results in uncached pages because a page only gets visited once per hour.

[HLeithner]

Anyway I don't care about the caching timeout I only want that it works, so whats your opinion to the PR?

[mbabker]

It's fine now.

[brianteeman]

could you please update the title and descrption

[infograf768]

restarted drone

[HLeithner]

@nikosdion any chance to get a test from you?

[nikosdion]

Sure. I will do it later today or tomorrow. I am in the middle of something else at the moment :)

[alikon]

I have tested this item ✅ successfully on 8aa7a6a

with some dirty sql hack

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24217.}

[ggppdk]

I have tested this item ✅ successfully on 8aa7a6a

Applied patch

I visit Control panel

• 1st visit on control panel updates timestamps of all enabled update sites, a long delay for me here 39 seconds, (hhmm i have a lot of extensions installed for testing)
• 2nd visit none update sites is queried, caching works as desired (described above)

Same thing (none update sites is queried) when i click find updates, unless i purge the cache, in which case "find updates" queries all

I manually expired some caching of some extensions (my caching setting in installer is 6 hours) i set timestamps to 7 hours in the past

UPDATE `PREFIX_update_sites`  
SET `last_check_timestamp` = (UNIX_TIMESTAMP() - (7 * 3600))  WHERE `update_site_id` > 150

Result on visiting control panel or on clicking find updates, only the updates site with expired timestamps are queried and have their timestamps updated
Reloading page of control panel or going to updates page and clicking the "Find updates" does not query the update sites

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24217.}

[Quy]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24217.}

[nikosdion]

Successful test. The code change is very straightforward, too.