Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hhvm-repo-mode cannot stat some files from MediaWiki 1.25.1 #5834

Closed
alex-mashin opened this issue Aug 1, 2015 · 10 comments
Closed

hhvm-repo-mode cannot stat some files from MediaWiki 1.25.1 #5834

alex-mashin opened this issue Aug 1, 2015 · 10 comments

Comments

@alex-mashin
Copy link

hhvm-repo-mode cannot stat some files from MediaWiki 1.25.1 and some of its important extensions: namely, PagedTiffHandler, Score, SemanticMediaWiki, SemanticResultFormats, SyntaxHighlight_GeSHi, TimedMediaHandler and Validator.

As a result, MediaWiki cannot be run in RepoAuthoritative mode.

List of files that cannot be stated below.

Unable to stat file (mediawiki installation path)/docs/Maintenance.php
Unable to stat file (mediawiki installation path)/docs/php-memcached/MemCachedClient.inc.php
Unable to stat file (mediawiki installation path)/extensions/PagedTiffHandler/tests/PHPUnit/Framework.php
Unable to stat file (mediawiki installation path)/extensions/Score/File/Ogg.php
Unable to stat file (mediawiki installation path)/extensions/SemanticMediaWiki/vendor/autoload.php
Unable to stat file (mediawiki installation path)/extensions/SemanticResultFormats/vendor/autoload.php
Unable to stat file (mediawiki installation path)/extensions/SyntaxHighlight_GeSHi/geshi/contrib/geshi.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/ApiBase.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/File/Ogg.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/PEAR.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/PEAR/Exception.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Bitstream.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Flac.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Speex.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Theora.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Vorbis.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/File/Ogg/Opus.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/Ogg/File/Ogg/Media.php
Unable to stat file (mediawiki installation path)/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg/File/Ogg/File/Ogg/Bitstream.php
Unable to stat file (mediawiki installation path)/extensions/Validator/vendor/autoload.php
Unable to stat file (mediawiki installation path)/includes/mail/Mail/mime.php
Unable to stat file (mediawiki installation path)/includes/mail/Mail.php
Unable to stat file (mediawiki installation path)/maintenance/PHPUnit/Autoload.php
Unable to stat file (mediawiki installation path)/vendor/data-values/common/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/data-values/data-values/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/data-values/interfaces/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/data-values/validators/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/jakub-onderka/php-parallel-lint/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/leafo/lessphp/docs/lessc.inc.php
Unable to stat file (mediawiki installation path)/vendor/liuggio/statsd-php-client/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/mediawiki/mediawiki-codesniffer/PHP/CodeSniffer/CLI.php
Unable to stat file (mediawiki installation path)/vendor/param-processor/param-processor/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/Build/PHPExcel.phar
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/Examples/PHPExcel/IOFactory.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/Examples/PHPExcel/Calculation.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/Calculation/testDataFileIterator.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/Calculation/custom/complexAssert.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/testDataFileIterator.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/Cell/testDataFileIterator.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/Shared/testDataFileIterator.php
Unable to stat file (mediawiki installation path)/vendor/phpoffice/phpexcel/unitTests/Classes/PHPExcel/Style/testDataFileIterator.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-code-coverage/scripts/PHP/CodeCoverage/Autoload.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-token-stream/tests/_fixture/test1.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-token-stream/tests/_fixture/test2.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-token-stream/tests/_fixture/test3.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-token-stream/tests/_fixture/test4.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/php-token-stream/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/phpunit/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/sebastian/comparator/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/serialization/serialization/vendor/autoload.php
Unable to stat file (mediawiki installation path)/vendor/squizlabs/php_codesniffer/CodeSniffer/PHP/CodeSniffer.php
Unable to stat file (mediawiki installation path)/vendor/squizlabs/php_codesniffer/CodeSniffer/PHP/CodeSniffer/Reporting.php
Unable to stat file (mediawiki installation path)/vendor/squizlabs/php_codesniffer/scripts/PHP/CodeSniffer/CLI.php
Unable to stat file (mediawiki installation path)/vendor/zordius/lightncandy/tests/src/lightncandy.php
Unable to stat file (mediawiki installation path)/vendor/zordius/lightncandy/tests/tests/helpers_for_test.php
Unable to stat file (mediawiki installation path)/vendor/phpunit/phpunit-mock-objects/vendor/autoload.php
@sagsai
Copy link

sagsai commented Aug 2, 2015

I am also seeing a similar sort of message for MediaWiki v1.25.1, although for a small set of file names.
However I noticed two things:

  1. None of the files actually exist.
  2. I didn't check thoroughly all the access points but the wiki seemed to be working

The error message in my case:

Unable to stat file /var/www/wiki/wg/includes/mail/Mail/mime.php
Unable to stat file /var/www/wiki/wg/docs/Maintenance.php
Unable to stat file /var/www/wiki/wg/includes/mail/Mail.php
Unable to stat file /var/www/wiki/wg/docs/php-memcached/MemCachedClient.inc.php
Unable to stat file /var/www/wiki/wg/vendor/leafo/lessphp/docs/lessc.inc.php
Unable to stat file /var/www/wiki/wg/maintenance/PHPUnit/Autoload.php
Unable to stat file /var/www/wiki/wg/vendor/zordius/lightncandy/tests/src/lightncandy.php
Unable to stat file /var/www/wiki/wg/vendor/liuggio/statsd-php-client/vendor/autoload.php
Unable to stat file /var/www/wiki/wg/vendor/zordius/lightncandy/tests/tests/helpers_for_test.php
Unable to stat file /var/www/wiki/wg/extensions/SyntaxHighlight_GeSHi/geshi/contrib/geshi.php

Forgive me for asking my own questions on someone else's thread, but I did not want to open an issue for the same problem.
Why is hhvm trying to stat non-existent files?
Is it somehow affecting the generated hhbc file?

Appreciate any inputs.

@Orvid
Copy link
Contributor

Orvid commented Aug 2, 2015

I believe it stats them because some code path somewhere in the source code has included them, it may even be a code path that is guarded by a constant that would normally prevent its inclusion, however, because that code path exists, HHVM attempts to include it in the generated hhbc file.

Unless there is a safety mechanism in HHVM that kicks in to disable some optimizations, then, as long as these files aren't being included in an active code path, they shouldn't effect the performance of the generated hhbc file, provided the server is being run in repo-auth mode.

Edit:
As the original issue claims that HHVM is unable to generate the repo because of this, I will assume that such a safety mechanism is indeed in place. A simple, temporary hacky solution would be to put empty files in the places it's looking for. As long as they aren't being included in an active code path, what I claimed above should still be the case.

@jwatzman
Copy link
Contributor

jwatzman commented Aug 3, 2015

Is this a stock MediaWiki installation? Does anything in your source code reference those files? I think what @Orvid says is basically correct.

@paulbiss
Copy link
Contributor

paulbiss commented Aug 3, 2015

Also worth noting that Unable to stat file shouldn't generally prevent the repo from building.

@sagsai
Copy link

sagsai commented Aug 4, 2015

As mentioned in my previous comment, none of the files actually exist.
I tried to find the filename references in MediaWiki's source code and I think the reasons can be broadly put in the following 3 categories:

  1. require/require_once etc. statements appearing in some static text/documentation/example files. For example docs/maintenance.txt, docs/php-memcached/Documentation & vendor/leafo/lessphp/docs/docs.md have such lines. Can be prevented by editing /usr/bin/hhvm-repo-mode.
  2. Conditional require in some core MediaWiki php files, e.g. includes/mail/UserMailer.php, maintenance/checkLess.php. Not really hhvm's issue and eliminating these errors requires Mediawiki's core code modification.
  3. require statements in some test/example php files in some 3rd party libs e.g. vendor/zordius/lightncandy/tests/*.php, vendor/liuggio/statsd-php-client/tests/bootstrap.php & extensions/SyntaxHighlight_GeSHi/geshi/contrib/aliased.php. I think these directories can be safely removed and the extension SyntaxHighlight_GeSHi should not be used at all with repo auth mode on, as it uses create_function.

I think HHVM should not read/parse the static files as I am inclined to believe, even without any knowledge about how hhvm works, that static files are not included in the repo. Not only this will eliminate some of the error messages, will also probably require less memory and time to to build the repo.

Replacing the line find "$1" -type f > "$FILE_LIST" in /usr/bin/hhvm-repo-mode with find -L "$1" -type f -name "*.php" -o -name "*.php5" > "$FILE_LIST" can do it.

Or maybe it can take the list of file types which are to be handled by hhvm, as an argument, in the absence of which a default list is used and build the find command.

Also -L really should be there to handle symlinks.

@sagsai
Copy link

sagsai commented Aug 4, 2015

@paulbiss the repo is getting built. From initial testing it seems that things are ok. Since the references are from static/test files I think the repo file should be all right. But will appreciate if hhvm-repo-mode is improved in later versions.
@jwatzman please see my above comment about references.

@alex-mashin
Copy link
Author

Is this a stock MediaWiki installation?
In my case, the stable MW 1.25.1.

Also worth noting that Unable to stat file shouldn't generally prevent the repo from building.
However, this error (or some other error in hphp) somehow prevents MediaWiki from working with the following MediaWiki error:

\nFatal error: MediaWiki requires the <a href="https://github.com/php-fig/log">PSR-3 logging library</a> to be present. This library is not embedded directly in MediaWiki's git repository and must be installed separately by the end user.\n\nPlease see <a href="https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries">mediawiki.org</a> for help on installing the required components. in /var/www/traditio.ru/w/includes/debug/logger/LoggerFactory.php on line 107

It's interesting that this library is not among those that couln't be statted.

Of course, the library is present, readable and MediaWiki works in non-RepoAuthoritative mode.

the extension SyntaxHighlight_GeSHi should not be used at all, as it uses create_function
This is an important extension, bundled with MediaWiki and used by WikiMedia projects. Although it was rewritten for MW 1.26.

@sagsai
Copy link

sagsai commented Aug 4, 2015

@alex-mashin I also got the fatal error, but that also is not hhvm's fault.
Changing the line interface_exists( '\Psr\Log\LoggerInterface' ) to interface_exists( 'Psr\Log\LoggerInterface' ) in the file includes/debug/logger/LoggerFactory.php and running the hhvm-repo-mode command again solves the problem.

It's interesting that this library is not among those that couln't be statted.

I think If you try to find each file for which stating failed, you would find none of them exists and yet they are referenced in some of the text/php files, as I found out.

Of course, the library is present, readable and MediaWiki works in non-RepoAuthoritative mode.

Of course the library is present, but extensions/SyntaxHighlight_GeSHi/geshi/contrib/geshi.php is not present, yet it has been require_onced in the in the extensions/SyntaxHighlight_GeSHi/geshi/contrib/aliased.php file. This file is actually an example file as mentioned at the top of it, so can be removed safely.

This is an important extension, bundled with MediaWiki and used by WikiMedia projects. Although it was rewritten for MW 1.26.

It is, but the code in SyntaxHighlight_GeSHi bundled with MW 1.25.1 stable release uses create_function which can/will case memory leak/poor performance with repo auth mode on #4250 . Nevertheless please feel free to ignore my suggestion.

@jwatzman
Copy link
Contributor

jwatzman commented Aug 4, 2015

Replacing the line find "$1" -type f > "$FILE_LIST" in /usr/bin/hhvm-repo-mode with find -L "$1" -type f -name "*.php" -o -name "*.php5" > "$FILE_LIST" can do it.

Yep, something like that, see #5833. It sounds like if we're smarter about which files to include, that will solve the bits of this that are actually solvable (i.e., not conditional includes for files that in fact don't exist). Going to close this in favor of #5833.

@jwatzman jwatzman closed this as completed Aug 4, 2015
@alex-mashin
Copy link
Author

Changing the line interface_exists( '\Psr\Log\LoggerInterface' ) to interface_exists( 'Psr\Log\LoggerInterface' ) in the file includes/debug/logger/LoggerFactory.php and running the hhvm-repo-mode command again solves the problem.

Does it mean that HHVM in repo authoritative mode doesn't understand absolute namespaces? Or it's a bug in MediaWiki which plain HHVM or PHP somehow bypass?

Of course the library is present, but extensions/SyntaxHighlight_GeSHi/geshi/contrib/geshi.php is not present, yet it has been require_onced in the in the extensions/SyntaxHighlight_GeSHi/geshi/contrib/aliased.php file.

The file is present in extensions/SyntaxHighlight_GeSHi/geshi.

It is successfully included in non-RepoAuthoritative mode because somewhere the path to it is registered as include path. Perhaps, I should file a bug with SyntaxHighlight GeSHi.

reedy added a commit to wikimedia/mediawiki that referenced this issue Nov 30, 2015
facebook/hhvm#5834
facebook/hhvm#5833

Change-Id: I138ffa5df874c5660897dc7feab36adef9f32aea
reedy added a commit to wikimedia/mediawiki that referenced this issue Dec 2, 2015
facebook/hhvm#5834
facebook/hhvm#5833

Change-Id: I138ffa5df874c5660897dc7feab36adef9f32aea
reedy added a commit to wikimedia/mediawiki that referenced this issue Dec 2, 2015
facebook/hhvm#5834
facebook/hhvm#5833

Change-Id: I138ffa5df874c5660897dc7feab36adef9f32aea
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants