Commit 0d5663c
committed
Tokenizer/PHP: fix tokenization of the
As per: #3326 (comment)
> After `PHP::tokenize()`, the `DEFAULT` is still tokenized as `T_DEFAULT`. This causes the `Tokenizer::recurseScopeMap()` to assign it as the `scope_opener` to the `;` semi-colon at the end of the constant declaration, with the class close curly brace being set as the `scope_closer`.
> In the `PHP::processAdditional()` method, the `DEFAULT` is subsequently retokenized to `T_STRING` as it is preceded by a `const` keyword, but that is too late.
>
> The `scope_opener` being set on the semi-colon is what is causing the errors to be displayed for the above code sample.
The commit fixes this by:
1. Abstracting the list of `T_STRING` contexts out to a class property.
2. Using the list from the property in all places in the `Tokenizer\PHP` class where keyword tokens are (potentially) being re-tokenized to `T_STRING`, including in the `T_DEFAULT` tokenization code which was added to address the PHP 8.0 `match` expressions.
Note: the issue was not introduced by `match` related code, however, that code being there does make it relatively easy now to fix this particular case.
While this doesn't address 3336 yes, it is a step towards addressing it and will sort out one of the most common causes for bugs.default keyword1 parent 65ab395 commit 0d5663c
1 file changed
+27
-44
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
454 | 454 | | |
455 | 455 | | |
456 | 456 | | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
457 | 480 | | |
458 | 481 | | |
459 | 482 | | |
| |||
1332 | 1355 | | |
1333 | 1356 | | |
1334 | 1357 | | |
1335 | | - | |
1336 | | - | |
1337 | | - | |
1338 | | - | |
1339 | | - | |
1340 | | - | |
1341 | | - | |
1342 | | - | |
1343 | | - | |
1344 | | - | |
| 1358 | + | |
1345 | 1359 | | |
1346 | 1360 | | |
1347 | 1361 | | |
| |||
1389 | 1403 | | |
1390 | 1404 | | |
1391 | 1405 | | |
1392 | | - | |
1393 | | - | |
1394 | | - | |
1395 | | - | |
1396 | | - | |
1397 | | - | |
1398 | | - | |
1399 | | - | |
| 1406 | + | |
1400 | 1407 | | |
1401 | 1408 | | |
1402 | 1409 | | |
| |||
1894 | 1901 | | |
1895 | 1902 | | |
1896 | 1903 | | |
1897 | | - | |
1898 | | - | |
1899 | | - | |
1900 | | - | |
1901 | | - | |
1902 | | - | |
1903 | | - | |
1904 | | - | |
1905 | | - | |
1906 | | - | |
1907 | | - | |
1908 | | - | |
1909 | | - | |
1910 | | - | |
1911 | | - | |
1912 | | - | |
1913 | | - | |
1914 | | - | |
1915 | | - | |
| 1904 | + | |
1916 | 1905 | | |
1917 | 1906 | | |
1918 | 1907 | | |
| |||
2784 | 2773 | | |
2785 | 2774 | | |
2786 | 2775 | | |
2787 | | - | |
2788 | | - | |
2789 | | - | |
2790 | | - | |
2791 | | - | |
2792 | | - | |
2793 | | - | |
| 2776 | + | |
2794 | 2777 | | |
2795 | 2778 | | |
2796 | 2779 | | |
| |||
0 commit comments