Initialisation process of the app takes too long and is too memory and CPU hungry as it tries to map all media files found in the system

[dbeckmann]

Gallery+ App does not scale

Similar to the original version - as in this issue, the gallery app still does not scale. There are over 100,000 images on the server, and very quickly after opening the main app page, the 512MB of memory allowed to PHP are exhausted. The same "searchByMimes" is still used to initialize the gallery as in the previous version, and as is noted in the comments in service/infoservice.php, it "can lead to performance issues", or it may fail entirely.

[oparoz]

Solutions exploration

Full map

searchByMimes is used to get a list of all images stored in the user's account in order to build a full "map" of the gallery , allowing to quickly navigate between folders.

Pros

• Going in and out of folders is smooth and fast

Cons

• Nothing happens until that map is built. This does not work at all for instances holding thousands of pictures.
• It's a bandwidth killer, especially on mobile
• The browser holds the full map in memory

Current folder

getDirectoryListing is used to retrieve only the paths of pictures in the current folder + 1-4 images per sub-folder

Pros

• Loads the gallery app quickly

Cons

• Going in and out of folders introduces a delay because a new list has to be fetched every time
• Might still not be optimal for people who store thousands of folders and images in one folder
• Empty folders have to be drilled down so that media files sitting in lower folders can be reached

Notes

• Batch processing those operations is currently not (easily) doable due to the limitations of core. See shared cache search and searchByMime don't scale well core#10077
• Preloading image lists for the sub and parent folders could speed up navigation, but would also fill the memory as the whole map would be slowly rebuilt
• Session caching of album maps is an option, but event hooks should be implemented first so that the map can be updated without having to reload the list from the server

[oparoz]

@jancborchardt - Any thoughts on how loading only the current folder affects usability?

[oparoz]

@dbeckmann - Try this branch and let me know how it works for you
https://github.com/interfasys/galleryplus/tree/performance-fixes

# git clone -b performance-fixes https://github.com/interfasys/galleryplus.git

[timmmmmey]

The performance-fixes branch works like a charm for me !
([error] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 79 bytes) in /var/www/owncloud/lib/private/files/node/folder.php on line 270)
is gone!

[oparoz]

Thanks for the feedback :)
It's still a bit rough and more changes are coming, especially to fix #27

[timmmmmey]

Oh that was too quick XD
After some time (264 Folder) it stopped. The only error message in log/apache is
File does not exist: /var/www/owncloud/core/vendor/jquery/dist, referer: https://xxxxxxxx/owncloud/index.php/apps/galleryplus/

But 264 Folders are better than none :)

[oparoz]

OK, so you have more than 250 sub-folders in your view? :-O

Check out data/ownloud.log to see if there is anything suspicious. File locking error, preview error, some exception, file not found, etc.

Use the Firefox debugger (better than Chrome's) and look for

• EventSource issue. I've seen it randomly fail.
• Any other errors.

EDIT: If you have more than 250 folders in your view, you've probably fallen victim to #27

[timmmmmey]

Only 635 Folders... Never user Picasa to organize your images :)
Okay something is wrong here:
ls -alh /media/owncloud/owncloud.log -rw-r----- 1 www-data www-data 133M Feb 18 16:09 /media/owncloud/owncloud.log
Okay 133MB Logfile... Lots of
INFO: Read lock requested for \/media\/owncloud\ Fetching the preview","level":0,"time":"2015-02-18T15:17:28+00:00","method":"GET",

Lets decrease the owncloud debug level to error, warnings,e.g. and clean the logfile
{"reqId":"xxx","remoteAddr":"xxx","app":"PHP","message":"Trying to get property of non-object at \/var\/www\/owncloud\/lib\/private\/log\/owncloud.php#126","level":3,"time":"2015-02-18T15:18:13+00:00"} {"reqId":"xxx","remoteAddr":"xxx","app":"PHP","message":"Trying to get property of non-object at \/var\/www\/owncloud\/lib\/private\/log\/owncloud.php#126","level":3,"time":"2015-02-18T15:18:15+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:31+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagepng() expects parameter 1 to be resource, boolean given at \/var\/www\/owncloud\/lib\/private\/image.php#292","level":3,"time":"2015-02-18T15:23:31+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":"2015-02-18T15:23:31+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:31+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecopyresampled() expects parameter 2 to be resource, boolean given at \/var\/www\/owncloud\/apps\/galleryplus\/preview\/preview.php#228","level":3,"time":"2015-02-18T15:23:31+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagepng(): Unable to open '\/media\/owncloud\/XXX_OWNCLOUD_USERNAME_XXX\/' for writing: Is a directory at \/var\/www\/owncloud\/lib\/private\/image.php#249","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagepng() expects parameter 1 to be resource, boolean given at \/var\/www\/owncloud\/lib\/private\/image.php#292","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecopyresampled() expects parameter 2 to be resource, boolean given at \/var\/www\/owncloud\/apps\/galleryplus\/preview\/preview.php#228","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":"2015-02-18T15:23:32+00:00"} {"reqId":"...","remoteAddr":"xxx","app":"PHP","message":"imagepng(): Unable to open '\/media\/owncloud\/XXX_OWNCLOUD_USERNAME_XXX\/' for writing: Is a directory at \/var\/www\/owncloud\/lib\/private\/image.php#249","level":3,"time":"2015-02-18T15:23:32+00:00"}

Firefox and Chrome Debugger was okay. The only error message was
'Content Security Policy: Die Einstellungen der Seite haben das Laden einer Ressource auf self blockiert ("script-src https://xxxxx.com 'unsafe-eval'").' Which is something like: Blocked loading a ressource from self, reason: settings for page in english.

[oparoz]

Thanks.
While working on performance fixes, I've noticed that my custom cache updater wasn't working properly, so this may be why you're seeing write errors, but in my tests, it didn't stop the app from loading.
You can make sure by commenting out this line:
https://github.com/interfasys/galleryplus/blob/performance-fixes/controller/servicecontroller.php#L268

I've noticed that your data folder seems to be in a different location that the web server root and that's a configuration that I haven't accounted for.

[timmmmmey]

Okay I changed the file but no change.
I havesome new errors:
[{"reqId":"","remoteAddr":"","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":""}, {"reqId":"","remoteAddr":"","app":"PHP","message":"imagepng() expects parameter 1 to be resource, boolean given at \/var\/www\/owncloud\/lib\/private\/image.php#292","level":3,"time":""}, {"reqId":"","remoteAddr":"","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":""} ,{"reqId":"","remoteAddr":"","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":""},

I also reconed that the gallery is always printing the whole line. So I thought about a problem with next line of folders. I moved away some, but there was no change. Do I have to clear some cache for Gallery+? The only obvious difference between the folder previous is that there is the first folder which contains some NEF (Nikon Raw) images

[timmmmmey]

Update: I found the malicious folder. If i move it away, it works fine. But there is no obvious reason why gallery+ fails with the folder. The "normal" OC8 fileviewer is able to create thumbnails.
Is there any command which I should test to find the malicious file inside the folder?

[timmmmmey]

Update:
Okay I found the malicious file. And I put some debugcode into /var/www/owncloud/lib/private/image.php
Line 543 following

default:
    // this is mostly file created from encrypted file                                
    $im=imagecreatefromstring(\OC\Files\Filesystem::file_get_contents(\OC\Files\Filesystem::getLocalPath($imagePath)));
    if ($im !== false) {
            $this->resource = $im;
             $iType = IMAGETYPE_PNG;
           $this->logger->debug('OC_Image->loadFromFile, Default', array('app' => 'core'));
            break;
    }
    else{
            $e = new Exception();
             $this->logger->error($e->getTraceAsString(), array('app' => 'core'));
    }
}
if ($this->valid()) {
....

Results in following stacktrace

 #0 /var/www/owncloud/lib/private/preview/image.php(43): OC_Image->loadFromFile('/media/owncloud...') 

#1 /var/www/owncloud/lib/private/preview.php(765): OC\Preview\Image->getThumbnail('//Photos/2000.0...', 2048, 2048, false, Object(OC\Files\View)) 

#2 /var/www/owncloud/lib/private/preview.php(516): OC\Preview->generatePreview(146349) 

#3 /var/www/owncloud/apps/galleryplus/preview/preview.php(196): OC\Preview->getPreview() 

#4 /var/www/owncloud/apps/galleryplus/preview/preview.php(128): OCA\GalleryPlus\Preview\Preview->getPreviewFromCore(false)

 #5 /var/www/owncloud/apps/galleryplus/service/previewservice.php(142): OCA\GalleryPlus\Preview\Preview->preparePreview(200, 200, false) 

#6 /var/www/owncloud/apps/galleryplus/controller/servicecontroller.php(307): OCA\GalleryPlus\Service\PreviewService->createPreview('Photos/2000.02....', 200, 200, false, true) 

#7 /var/www/owncloud/apps/galleryplus/controller/servicecontroller.php(262): OCA\GalleryPlus\Controller\ServiceController->getPreview('Photos/2000.02....', 200, 200, false, false, true)

#8 /var/www/owncloud/apps/galleryplus/controller/servicecontroller.php(192): OCA\GalleryPlus\Controller\ServiceController->getThumbnail('Photos/2000.02....', true, true) 

#9 [internal function]: OCA\GalleryPlus\Controller\ServiceController->getThumbnails('Photos/1. Mai/D...', true, true) 

#10 /var/www/owncloud/lib/private/appframework/http/dispatcher.php(158): call_user_func_array(Array, Array) 

#11 /var/www/owncloud/lib/private/appframework/http/dispatcher.php(86): OC\AppFramework\Http\Dispatcher->executeController(Object(OCA\GalleryPlus\Controller\ServiceController), 'getThumbnails')

 #12 /var/www/owncloud/lib/private/appframework/app.php(97): OC\AppFramework\Http\Dispatcher->dispatch(Object(OCA\GalleryPlus\Controller\ServiceController), 'getThumbnails')

 #13 /var/www/owncloud/lib/private/appframework/routing/routeactionhandler.php(44): OC\AppFramework\App::main('ServiceControll...', 'getThumbnails', Object(OC\AppFramework\DependencyInjection\DIContainer), Array)

 #14 [internal function]: OC\AppFramework\routing\RouteActionHandler->__invoke(Array)

 #15 /var/www/owncloud/lib/private/route/router.php(250): call_user_func(Object(OC\AppFramework\routing\RouteActionHandler), Array)

 #16 /var/www/owncloud/lib/base.php(763): OC\Route\Router->match('/apps/gallerypl...') 

#17 /var/www/owncloud/index.php(36): OC::handleRequest()

 #18 {main}

My php skills are limited, so I'm not sure if it is an OC8 error or a Gallery+ problem.

[oparoz]

Thanks for the logs :)
Commenting that line did nothing as it wasn't even updating the thumbnail in the version you have, my bad :S.
I'm starting to suspect a corrupt thumbnail cache. It's happened to me while working on that branch and I'm not sure why and how to fix it...yet.

• What is the media type of the file?

Check the folder data/<userId>/thumbnails/<fileId>

• Is there a file ending with -max.png?
• Do the file size look suspicious? Like a file being only a few bytes long.

To see it's that's the problem, you can move or delete that folder (<fileId>) and see if things change after a refresh.

[timmmmmey]

Okay, how do I get the fileId ? But I think this is not the problem anymore.

I definitely found the files which are creating the problems. If I delete them, everything works find.
The type is a corrupt binary file. Its just rubbish inside, probably created by my old external harddisk.
So my bug is a corrupt file :)

But the main problem seem to be, that we are not checking whether the file is a valid image resource (somewhere in the above stack trace).

[oparoz]

It will be easier to get the fileId with the next version, but in the mean time, you can find it in the HTML, on the Files app side.

And I agree. Ideally, the app should be able to survive such failure, but the problem is that most of these problems are in core and if it doesn't raise an exception, it's difficult to catch without some crazy workarounds to intercept PHP errors. I did look into this at some point...

Best thing to do is to report those issues in core.
The methods found in your log should be able to handle failures
https://github.com/owncloud/core/blob/master/lib/private/image.php#L292
https://github.com/owncloud/core/blob/master/lib/private/image.php#L545

[{"reqId":"","remoteAddr":"","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":""}, {"reqId":"","remoteAddr":"","app":"PHP","message":"imagepng() expects parameter 1 to be resource, boolean given at \/var\/www\/owncloud\/lib\/private\/image.php#292","level":3,"time":""},

{"reqId":"","remoteAddr":"","app":"PHP","message":"imagecreatefromstring(): Data is not in a recognized format at \/var\/www\/owncloud\/lib\/private\/image.php#545","level":3,"time":""} ,{"reqId":"","remoteAddr":"","app":"core","message":"OC_Image->data. Error getting image data.","level":3,"time":""},

[jospoortvliet]

@timmmmmey @oparoz did the issues get reported to core? I can reproduce (after installing this branch my logs are full of errors :D ) and that suggests to me these should be fixed before gallery+ can be a viable successor to the current gallery app - and I like it to be as it looks prettier and we all like pretty things 🎱

[oparoz]

@jospoortvliet - Not reported yet I think. I agree these should be fixed, but since I don't think they're blocking the execution, they will have a low priority, like this one: owncloud/core#14036

The next release of the app should be able to handle these issues more gracefully, but it's difficult to test all possible use cases.

[jospoortvliet]

Well, this release (8.1) would focus on performance and stability so this kind of stuff sounds like good candidates for some love and attention...

[oparoz]

@dbeckmann - Check out master if you can or wait for 2.0.6. It should scale, but you may still not get the app to load from the root if you have deep folders, because of this bug #27.
@timmmmmey - master should contain enough fixes to avoid broken files, short of testing them individually (which would slow down the initial data collection)