Skip to content

Commit 5d2dab7

Browse files
sgiehlmattabtsteurPeterUpfolddiosmosis
authored
Merge branch 3.x-dev into 4.x-dev (matomo-org#15543)
* Updates search engine and social definitions (matomo-org#15384) * updates device detector to latest release (matomo-org#15388) * updates device detector to latest release * updates tests * translation update (matomo-org#15389) * Fix Could not get the lock for ID, when creating a site (matomo-org#15401) * Lock key start * do not empty key lock Co-authored-by: Thomas Steur <[email protected]> * 3.13.1 * submodule updates * Use correct name in update available message (matomo-org#15423) * Fix removing user capabilities (matomo-org#15422) * Order of implode() args, avoid E_NOTICE in PHP7.4 (matomo-org#15428) * Fixes possible php warning in visitor log (matomo-org#15442) * silence is_executable call (matomo-org#15446) * Make sure geolocation admin experience is consistent if user is not using GeoIp2 plugin. (matomo-org#15447) * Fix referrers test. (matomo-org#15448) * Ensure to close visitor popover correctly (matomo-org#15443) * Fixes possible warning (matomo-org#15453) * Forward instance_id from local config when reseting config during tests. (matomo-org#15445) * Add event that allows plugins to disable archiving for certain periods/sites if they want. (matomo-org#15457) * Add event that allows plugins to disable archiving for certain periods/sites if they want. * apply review feedback * Fix possible warning for columns without index (matomo-org#15467) * Day range archiving issue (matomo-org#15462) * Improve lock ID check for max length (matomo-org#15407) Better patch for matomo-org#15401 which was merged last minute... This way it always works even when someone calls `acquireLock` directly instead of `execute` Pushing this for now into 3.x-dev but can also put it into 4.x-dev directly but then there might be merge conflicts when merging 3.x-dev into 4.x-dev * Use SameSite none for session token when embedded into iframe (matomo-org#15439) * Make sure tracking works in IE9 and lower (matomo-org#15480) * Mention Joomla install FAQ (matomo-org#15481) * Make sparklines work when mbstring extension is not installed (matomo-org#15489) 1) Too few arguments to function mb_strtolower(), 1 passed in matomo/vendor/davaxi/sparkline/src/Sparkline/StyleTrait.php on line 129 and exactly 2 expected 2) mb_strlen is not defined * update screenshots (matomo-org#15488) * 3.13.2-rc1 * Use safemode when running CLI commands (matomo-org#15472) * update icons submodule (matomo-org#15490) * update icons submodule * update UI tests * Fix possible undefined index notice (matomo-org#15502) * Use latest davaxi/sparkline release (matomo-org#15464) * translation update * submodule updates * Fix deprecation notice (matomo-org#15530) see matomo-org#15467 (comment) * 3.13.2-rc2 * update cache component (matomo-org#15536) * fixes copy dashboard to user for more than 100 users (matomo-org#15538) cherry picking matomo-org#15424 to fix matomo-org#15420 in 3.x-dev * Add missing return statement. (matomo-org#15539) * 3.13.2 * update tests * update tests Co-authored-by: Matthieu Aubry <[email protected]> Co-authored-by: Thomas Steur <[email protected]> Co-authored-by: Peter Upfold <[email protected]> Co-authored-by: diosmosis <[email protected]> Co-authored-by: Lukas Winkler <[email protected]>
1 parent ab1e700 commit 5d2dab7

File tree

68 files changed

+480
-133
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+480
-133
lines changed

‎composer.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@
3737
"matomo/ini": "~2.0",
3838
"matomo/matomo-php-tracker": "~2.0",
3939
"matomo/network": "~2.0",
40-
"matomo/referrer-spam-blacklist": "~1.0",
41-
"matomo/searchengine-and-social-list": "~1.0",
40+
"matomo/referrer-spam-blacklist": "~3.0",
41+
"matomo/searchengine-and-social-list": "~3.0",
4242
"monolog/monolog": "~1.11",
4343
"mustangostang/spyc": "~0.6.0",
4444
"pear/pear_exception": "~1.0.0",

‎composer.lock

+11-11
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎console

+5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
#!/usr/bin/env php
22
<?php
3+
4+
use Piwik\FrontController;
5+
36
if (!defined('PIWIK_DOCUMENT_ROOT')) {
47
define('PIWIK_DOCUMENT_ROOT', dirname(__FILE__) == '/' ? '' : dirname(__FILE__));
58
}
@@ -23,5 +26,7 @@ if (!defined('PIWIK_ENABLE_ERROR_HANDLER') || PIWIK_ENABLE_ERROR_HANDLER) {
2326
Piwik\ExceptionHandler::setUp();
2427
}
2528

29+
FrontController::setUpSafeMode();
30+
2631
$console = new Piwik\Console();
2732
$console->run();

‎core/Archive.php

+3-2
Original file line numberDiff line numberDiff line change
@@ -649,10 +649,11 @@ private function cacheArchiveIdsAfterLaunching($archiveGroups, $plugins)
649649
foreach ($this->params->getIdSites() as $idSite) {
650650
$site = new Site($idSite);
651651

652-
if ($period->getLabel() === 'day'
652+
if ($period->getLabel() === 'day'
653653
&& !$this->params->getSegment()->isEmpty()
654654
&& Common::getRequestVar('skipArchiveSegmentToday', 0, 'int')
655-
&& $period->getDateStart()->toString() == Date::factory('now', $site->getTimezone())->toString()) {
655+
&& $period->getDateStart()->toString() == Date::factory('now', $site->getTimezone())->toString()
656+
) {
656657

657658
Log::debug("Skipping archive %s for %s as segment today is disabled", $period->getLabel(), $period->getPrettyString());
658659
continue;

‎core/ArchiveProcessor/PluginsArchiver.php

+4-2
Original file line numberDiff line numberDiff line change
@@ -160,13 +160,15 @@ public function callAggregateAllPlugins($visits, $visitsConverted, $forceArchivi
160160
try {
161161
self::$currentPluginBeingArchived = $pluginName;
162162

163+
$period = $this->params->getPeriod()->getLabel();
164+
163165
$timer = new Timer();
164166
if ($this->shouldAggregateFromRawData) {
165-
Log::debug("PluginsArchiver::%s: Archiving day reports for plugin '%s'.", __FUNCTION__, $pluginName);
167+
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' from raw data.", __FUNCTION__, $pluginName);
166168

167169
$archiver->callAggregateDayReport();
168170
} else {
169-
Log::debug("PluginsArchiver::%s: Archiving period reports for plugin '%s'.", __FUNCTION__, $pluginName);
171+
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' using reports for smaller periods.", __FUNCTION__, $pluginName);
170172

171173
$archiver->callAggregateMultipleReports();
172174
}

‎core/ArchiveProcessor/Rules.php

+16-4
Original file line numberDiff line numberDiff line change
@@ -226,9 +226,21 @@ public static function isArchivingDisabledFor(array $idSites, Segment $segment,
226226
return !$isArchivingEnabled;
227227
}
228228

229-
public static function isRequestAuthorizedToArchive()
229+
public static function isRequestAuthorizedToArchive(Parameters $params = null)
230230
{
231-
return Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();
231+
$isRequestAuthorizedToArchive = Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();
232+
233+
if (!empty($params)) {
234+
/**
235+
* @ignore
236+
*
237+
* @params bool &$isRequestAuthorizedToArchive
238+
* @params Parameters $params
239+
*/
240+
Piwik::postEvent('Archiving.isRequestAuthorizedToArchive', [&$isRequestAuthorizedToArchive, $params]);
241+
}
242+
243+
return $isRequestAuthorizedToArchive;
232244
}
233245

234246
public static function isBrowserTriggerEnabled()
@@ -293,11 +305,11 @@ public static function isSegmentPreProcessed(array $idSites, Segment $segment)
293305
*
294306
* @return string[]
295307
*/
296-
public static function getSelectableDoneFlagValues($includeInvalidated = true)
308+
public static function getSelectableDoneFlagValues($includeInvalidated = true, Parameters $params = null)
297309
{
298310
$possibleValues = array(ArchiveWriter::DONE_OK, ArchiveWriter::DONE_OK_TEMPORARY);
299311

300-
if (!Rules::isRequestAuthorizedToArchive()
312+
if (!Rules::isRequestAuthorizedToArchive($params)
301313
&& $includeInvalidated
302314
) {
303315
//If request is not authorized to archive then fetch also invalidated archives

‎core/Archiver/Request.php

+27-2
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ class Request
3131
*/
3232
public function __construct($url)
3333
{
34-
$this->url = $url;
34+
$this->setUrl($url);
3535
}
3636

3737
public function before($callable)
@@ -68,9 +68,34 @@ public function setUrl($url)
6868
}
6969

7070
public function changeDate($newDate)
71+
{
72+
$this->changeParam('date', $newDate);
73+
}
74+
75+
public function makeSureDateIsNotSingleDayRange()
76+
{
77+
// TODO: revisit in matomo 4
78+
// period=range&date=last1/period=range&date=previous1 can cause problems during archiving due to Parameters::isDayArchive()
79+
if (preg_match('/[&?]period=range/', $this->url)) {
80+
if (preg_match('/[&?]date=last1/', $this->url)) {
81+
$this->changeParam('period', 'day');
82+
$this->changeParam('date', 'today');
83+
} else if (preg_match('/[&?]date=previous1/', $this->url)) {
84+
$this->changeParam('period', 'day');
85+
$this->changeParam('date', 'yesterday');
86+
} else if (preg_match('/[&?]date=([^,]+),([^,&]+)/', $this->url, $matches)
87+
&& $matches[1] == $matches[2]
88+
) {
89+
$this->changeParam('period', 'day');
90+
$this->changeParam('date', $matches[1]);
91+
}
92+
}
93+
}
94+
95+
public function changeParam($name, $newValue)
7196
{
7297
$url = $this->getUrl();
73-
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $url);
98+
$url = preg_replace('/([&?])' . preg_quote($name) . '=[^&]*/', '$1' . $name . '=' . $newValue, $url);
7499
$this->setUrl($url);
75100
}
76101
}

‎core/CliMulti/CliPhp.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ private function isValidPhpType($path)
8686

8787
private function getPhpCommandIfValid($path)
8888
{
89-
if (!empty($path) && is_executable($path)) {
89+
if (!empty($path) && @is_executable($path)) {
9090
if (0 === strpos($path, PHP_BINDIR) && $this->isValidPhpType($path)) {
9191
return $path;
9292
}

‎core/Concurrency/Lock.php

+7-7
Original file line numberDiff line numberDiff line change
@@ -50,13 +50,6 @@ public function getAllAcquiredLockKeys()
5050

5151
public function execute($id, $callback)
5252
{
53-
if (Common::mb_strlen($id) > self::MAX_KEY_LEN) {
54-
// Lock key might be too long for DB column, so we hash it but leave the start of the original as well
55-
// to make it more readable
56-
$md5Len = 32;
57-
$id = Common::mb_substr($id, 0, self::MAX_KEY_LEN - $md5Len - 1) . md5($id);
58-
}
59-
6053
$i = 0;
6154
while (!$this->acquireLock($id)) {
6255
$i++;
@@ -76,6 +69,13 @@ public function acquireLock($id, $ttlInSeconds = 60)
7669
{
7770
$this->lockKey = $this->lockKeyStart . $id;
7871

72+
if (Common::mb_strlen($this->lockKey) > self::MAX_KEY_LEN) {
73+
// Lock key might be too long for DB column, so we hash it but leave the start of the original as well
74+
// to make it more readable
75+
$md5Len = 32;
76+
$this->lockKey = Common::mb_substr($id, 0, self::MAX_KEY_LEN - $md5Len - 1) . md5($id);
77+
}
78+
7979
$lockValue = substr(Common::generateUniqId(), 0, 12);
8080
$locked = $this->backend->setIfNotExists($this->lockKey, $lockValue, $ttlInSeconds);
8181

‎core/Console.php

+15
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,21 @@ public function renderException($e, $output)
7777
}
7878

7979
public function doRun(InputInterface $input, OutputInterface $output)
80+
{
81+
try {
82+
return $this->doRunImpl($input, $output);
83+
} catch (\Exception $ex) {
84+
try {
85+
FrontController::generateSafeModeOutputFromException($ex);
86+
} catch (\Exception $ex) {
87+
// ignore, we re-throw the original exception, not a wrapped one
88+
}
89+
90+
throw $ex;
91+
}
92+
}
93+
94+
private function doRunImpl(InputInterface $input, OutputInterface $output)
8095
{
8196
if ($input->hasParameterOption('--xhprof')) {
8297
Profiler::setupProfilerXHProf(true, true);

‎core/CronArchive.php

+19-2
Original file line numberDiff line numberDiff line change
@@ -1104,7 +1104,16 @@ private function archiveReportsFor($idSite, $period, $date, $archiveSegments, Ti
11041104
return Request::ABORT;
11051105
}
11061106

1107+
$urlBefore = $request->getUrl();
11071108
$request->changeDate($newDate);
1109+
$request->makeSureDateIsNotSingleDayRange();
1110+
1111+
// check again if we are already archiving the URL since we just changed it
1112+
if ($request->getUrl() !== $urlBefore
1113+
&& $self->isAlreadyArchivingSegment($request->getUrl(), $idSite, $period, $segment)
1114+
) {
1115+
return Request::ABORT;
1116+
}
11081117

11091118
$this->logArchiveWebsite($idSite, $period, $newDate);
11101119
});
@@ -1972,9 +1981,17 @@ private function getUrlsWithSegment($idSite, $period, $date)
19721981
return Request::ABORT;
19731982
}
19741983

1975-
$url = $request->getUrl();
1976-
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $url);
1984+
$urlBefore = $request->getUrl();
1985+
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $urlBefore);
19771986
$request->setUrl($url);
1987+
$request->makeSureDateIsNotSingleDayRange();
1988+
1989+
// check again if we are already archiving the URL since we just changed it
1990+
if ($request->getUrl() !== $urlBefore
1991+
&& $self->isAlreadyArchivingSegment($request->getUrl(), $idSite, $period, $segment)
1992+
) {
1993+
return Request::ABORT;
1994+
}
19781995

19791996
$processedSegmentCount++;
19801997
$logger->info(sprintf(

‎core/DataAccess/ArchiveSelector.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ public static function getArchiveIdAndVisits(ArchiveProcessor\Parameters $params
7171
$plugins = array("VisitsSummary", $requestedPlugin);
7272

7373
$doneFlags = Rules::getDoneFlags($plugins, $segment);
74-
$doneFlagValues = Rules::getSelectableDoneFlagValues($includeInvalidated);
74+
$doneFlagValues = Rules::getSelectableDoneFlagValues($includeInvalidated, $params);
7575

7676
$results = self::getModel()->getArchiveIdAndVisits($numericTable, $idSite, $period, $dateStartIso, $dateEndIso, $minDatetimeIsoArchiveProcessedUTC, $doneFlags, $doneFlagValues);
7777

‎core/FrontController.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ private static function generateSafeModeOutputFromError($lastError)
106106
* @param Exception $e
107107
* @return string
108108
*/
109-
private static function generateSafeModeOutputFromException($e)
109+
public static function generateSafeModeOutputFromException($e)
110110
{
111111
StaticContainer::get(LoggerInterface::class)->error('Uncaught exception: {exception}', [
112112
'exception' => $e,

‎core/Plugin/Report.php

+4-4
Original file line numberDiff line numberDiff line change
@@ -496,8 +496,8 @@ protected function getMetricsDocumentation()
496496
} elseif ($metric instanceof Metric) {
497497
$name = $metric->getName();
498498
$metricDocs = $metric->getDocumentation();
499-
if (empty($metricDocs)) {
500-
$metricDocs = @$translations[$name];
499+
if (empty($metricDocs) && !empty($translations[$name])) {
500+
$metricDocs = $translations[$name];
501501
}
502502

503503
if (!empty($metricDocs)) {
@@ -513,8 +513,8 @@ protected function getMetricsDocumentation()
513513
} elseif ($processedMetric instanceof Metric) {
514514
$name = $processedMetric->getName();
515515
$metricDocs = $processedMetric->getDocumentation();
516-
if (empty($metricDocs)) {
517-
$metricDocs = @$translations[$name];
516+
if (empty($metricDocs) && !empty($translations[$name])) {
517+
$metricDocs = $translations[$name];
518518
}
519519

520520
if (!empty($metricDocs)) {

‎core/Session.php

+11
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,17 @@ public static function isSessionStarted()
171171
return self::$sessionStarted;
172172
}
173173

174+
public static function getSameSiteCookieValue()
175+
{
176+
$config = Config::getInstance();
177+
$general = $config->General;
178+
if (!empty($general['enable_framed_pages']) && ProxyHttp::isHttps()) {
179+
return 'None';
180+
}
181+
182+
return 'Lax';
183+
}
184+
174185
/**
175186
* Write cookie header. Similar to the native setcookie() function but also supports
176187
* the SameSite cookie property.

‎core/Session/SessionAuth.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ private function updateSessionExpireTime(SessionFingerprint $sessionFingerprint)
195195
$sessionParams['domain'],
196196
$sessionParams['secure'],
197197
$sessionParams['httponly'],
198-
'lax'
198+
Session::getSameSiteCookieValue()
199199
);
200200

201201
// ...and we also update the expiration time stored server side so we can prevent expired sessions from being reused

‎core/Tracker/Model.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -346,7 +346,7 @@ public function updateAction($idLinkVa, $valuesToUpdate)
346346

347347
list($updateParts, $sqlBind) = $this->fieldsToQuery($valuesToUpdate);
348348

349-
$parts = implode($updateParts, ', ');
349+
$parts = implode(', ', $updateParts);
350350
$table = Common::prefixTable('log_link_visit_action');
351351

352352
$sqlQuery = "UPDATE $table SET $parts WHERE idlink_va = ?";

‎core/Tracker/Settings.php

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ public function getConfigId(Request $request, $ipAddress)
4040
$deviceDetector = StaticContainer::get(DeviceDetectorFactory::class)->makeInstance($userAgent);
4141
$aBrowserInfo = $deviceDetector->getClient();
4242

43-
if ($aBrowserInfo['type'] != 'browser') {
43+
if (empty($aBrowserInfo['type']) || 'browser' !== $aBrowserInfo['type']) {
4444
// for now only track browsers
4545
unset($aBrowserInfo);
4646
}

0 commit comments

Comments
 (0)