[4.4] Add database configuration option for "sql_big_selects" session variable on MySQL or MariaDB databases

[richard67]

Pull Request for Issue #41156 .

See also issues #17580 and #39479 , discussion #42198 and support forum thread https://forum.joomla.org/viewtopic.php?f=811&t=995872 .

Requires joomla-framework/database#292 .

Summary of Changes

This pull request (PR) adds a new option to the "Database" section of Global Configuration which allows to set the "sql_big_selects" session variable to on or off or leave it untouched, which is the default.

So when people run into the 1104 The SELECT would examine more than MAX_JOIN_SIZE rows ... error with their database server, they can fix that by switching the new option to "Yes".

When the default value of the option is used, which is also the case after a CMS update to a new version which includes this PR here, then nothing is done, so there is no unnecessary SQL statement issues with each connection in that case.

The issue has been reported more frequently in recent times, so I think we should fix it as a bug fix in 4.4-dev and merge it up later.

I have made the additional feature PR #42559 for the 5.1-dev branch to show the values of the "sql_big_selects" and "max_join_size" variables in the System Information on MySQL or MariaDB databases.

In case if the framework PR #joomla-framework/database#292 is not accepted for some reason, I have created PR #42558 as an alternative to this one here, which does the same but doesn't require a change in the database framework. That alternative is not nice, but it works the same as this one here.

Testing Instructions

This PR is only relevant for MySQL or MariaDB databases, not for PostgreSQL.

When having tested with success, please leave a comment about that in the framework PR joomla-framework/database#292 because a successful test for this PR here will also mean a successful test of that PR.

For some test steps which change the global "sql_big_select" variable it needs to have administrator privileges for the database. E.g. on a local MySQL you have to connect with user "root" when using e.g. phpMyAdmin. That means that these test steps can't be executed in a shared hosting environment.

1. Add the following line of code to the "index.php" file of your site template below the use statements in the PHP code at the top, e.g. if Cassiopeia is used file "templates/cassiopeia/index.php" at line 20, to show the current value of the "sql_big_select" session variable:

echo 'TEST: ' . Factory::getDbo()->setQuery('SELECT @@sql_big_selects;')->loadResult() . '<br>';

2. Go to the site (frontend) and note the current value of the "sql_big_select" session variable shown at the top after the text "TEST: ".
3. Check the global and session variables "sql_big_selects" in a client like e.g. phpMyAdmin:

SELECT @@GLOBAL.sql_big_selects, @@SESSION.sql_big_selects;

4. Check that the values from step 3 are the same as the value from step 2.
5. Apply the changes from this PR and the framework PR [2.0-dev] New option to set sql_big_selects on MySQL/MariaDB joomla-framework/database#292 .
   You can find zip packages for new installation and update and a custom update URL for packages which include the changes from both PRs here: https://test5.richard-fath.de/pr-42557/ .
6. In the administrator, go to Global Configuration, tab "server", section "Database" and check the options at the end of that section.
   Result: There is a new list select option "Allow large queries", and the first value "No change (server controlled)" is selected, which is the default,
7. On site, check again the current value of the "sql_big_selects" session variable as described in step 2. If the page is still open, reload the page to get the current value.
   Result: The value of the "sql_big_selects" session variable has not changed, it is still the same as in step 2.
8. In the administrator in Global Configuration, change the value of the new "Allow large queries" option so that the value of the "sql_big_selects" session variable should change. I.e. if the value recorded in step 2 was "0" select "Yes" and if the value from step 2 was "1" select "No", and save the changes.
9. On site, check again the current value of the "sql_big_selects" session variable as described in step 2. If the page is still open, reload the page to get the current value.
   Result: The value of the "sql_big_selects" session variable has changed to the desired value (0 for "No" and 1 for "Yes").
10. This step requires administrator privileges for the database, e.g. user "root".
    In a client like e.g. phpMyAdmin, change the value of the "sql_big_selects" global variable so that the session variable should change. E.g. if the current value of the session variable check in step 9 was 0, change the global variable to 1, and vice versa.
    Example for changing to 1:

SET GLOBAL sql_big_selects=1;

11. On site, check again the current value of the "sql_big_selects" session variable as described in step 2. If the page is still open, reload the page to get the current value.
    Result: The value of the "sql_big_selects" session variable has not changed, it is still the same as selected in step 9.
12. In the administrator in Global Configuration, change the value of the new "Allow large queries" option back to the default "No change (server controlled)".
13. On site, check again the current value of the "sql_big_selects" session variable as described in step 2. If the page is still open, reload the page to get the current value.
    Result: Now the value of the "sql_big_selects" session variable has changed compared to steps 9 and 11. It is equal to the value set in step10 for the global variable.
14. In the administrator in Global Configuration, change the value of the new "Allow large queries" option so that the value of the "sql_big_selects" session variable should change. I.e. if the value recorded in step 13 was "0" select "Yes" and if the value from step 13 was "1" select "No", and save the changes.
15. If you want, continue to play around with the global variable as described in step 10 and the different values of the new Global Configuration option.
    Result: When the new option has its default value "No change (server controlled)", the "sql_big_selects" session variable has the value as set by the server with the global variable.
    If the new option has value "No", the "sql_big_selects" session variable is set to 0.
    If the new option has value "Yes", the "sql_big_selects" session variable is set to 1.
16. Clean up the changes to the global variable from step 10:
    Either just restart the database sever, or set the global variable back to the value which it had in step 3.

Actual result BEFORE applying this Pull Request

There is no way to change the value of the "sql_big_select" session variable.

Expected result AFTER applying this Pull Request

A new list select option "Allow large queries" is available at the end of the "Database" section in the "Server" tab of Global Configuration ($dbsqlbigselects in configuration.php).

The option is only shown when the selected database type is "MySQLi" or "MySQL (PDO)". When "PostgreSQL (PDO)" is selected, the new option is hidden.

The value of that option is the default value "No change (server controlled)" after a new installation or an update.

When the new option has that default value, the value of the "sql_big_selects" session variable is not changed by the CMS, and no additional SQL statement is performed when connecting to the database.

When the new option has value "No", the value of the "sql_big_selects" session variable is set to 0 when connecting to the database.

When the new option has value "Yes", the value of the "sql_big_selects" session variable is set to 1 when connecting to the database. This will prevent the SQL error 1104 The SELECT would examine more than MAX_JOIN_SIZE rows ....

Link to documentations

Please select:

• Documentation link for docs.joomla.org: https://docs.joomla.org/Help4.x:Site_Global_Configuration https://docs.joomla.org/Help4.x:Site_Global_Configuration_Server

• No documentation changes for docs.joomla.org needed

• Pull Request link for manual.joomla.org:

• No documentation changes for manual.joomla.org needed

A description of the new "Allow large queries" parameter should be added to the end of sub-section "database" of the "Server" section on https://docs.joomla.org/Help4.x:Site_Global_Configuration / https://help.joomla.org/proxy?keyref=Help44:Site_Global_Configuration&lang=en .

Suggested text:

• Allow large queries
  -- "No change (server controlled)" (default)
  -- "No"
  -- "Yes"
  Use "Yes" if you get the SQL error "1104 The SELECT would examine more than MAX_JOIN_SIZE rows" on your site.

The screenshot on https://docs.joomla.org/Help4.x:Site_Global_Configuration_Server needs to be updated so it shows the new option.

[wilsonge]

I think rather than just blasting the server for all queries we need to think about a more targeted approach. Judging by the page and the fact this happens more frequently it would be my guess that this is some sort of query from the extension compatibility information? We don't want the database to limit access to the rest of the site when the Update Joomla page is shown because the db is throttled with the other query (and that's a genuine thing - a while back on the main downloads site loading the ARS dashboard gave this issue because of the number of downloads).

We a) need to investigate what query is failing when we load that page and optimise it and b) if we really really need this (and I'd hope we don't for anything in core) then we need to consider allowing this on a page specific way I think rather than globally hiding the issue for all future queries.

[richard67]

@wilsonge Even if we fix the query for the CMS core there might still be a 3rd party extension causing the same issue, so this PR here would still make sense for that case and for the core it would be a workaround until we have identified the critical query.

[richard67]

@wilsonge P.S. Unfortunately I wasn’t able to find the critical query up to now. I could not reproduce the issue with the small max_join_size and sql_big_queries disabled like it was reported in the support forum. It seems to need certain starting conditions. Possibly a bunch of 3rd party extensions or whatever.

[richard67]

@wilsonge P.P.S.: The "bloating" thing in your comment I don't understand. This PR only allows to enforce a settings which is the default anyway in most cases, and for most environments nothing changes at all, and no additional SQL statement is sent.

[richard67]

@wilsonge Meanwhile we got a stack trace in issue #41156 , and it's the big query for getting the IDs of all core extensions. But I don't get the issue on MySQL, so either it happens only with MariaDB 10.6, or the "extension" index is missing for some reason on the site with the issue so the query results in a full table scan.

[richard67]

@wilsonge I could not reproduce the issue with MySQL 8, but I can now with a MariaDB 10.4, so it seems that MySQL and MariaDB behave differently. I will see of I can fix that query.

[richard67]

@wilsonge I think I have a fix for that query. PR will come later today.

[richard67]

@wilsonge PR is #42576 . It is draft because I need to complete the testing instructions. As soon as ready for tests I will post here and close this issue. But interested readers may already have a look on it. It's still worth a question if this PR here could make sense for the case that the error happens also with other queries, e.g. from 3rd party extensions.

[richard67]

PR #42576 is ready for testing. However this PR here still might be useful when that SQL error happens with a 3rd party extension so people have a workaround until there is a fix for that extension.

[richard67]

Meanwhile I think that the fix provided here will not really be necessary anymore when PR #42576 will be merged.

Closing in favour of #42576 .

[wilsonge]

Sorry with holidays and stuff in December didn't get time to look at this - but I'm much happier with fixing the query as proposed (yes I know it needs a follow up) than this PR. So major +1 on the way forward (closing this and merging that one + the fix)