NGiNX try_files support

[Zae]

Because after #7 glide stores images with the query params as md5 in the filename there is not easy way to use try_files to let nginx find the cached image itself and speed up the whole process a lot.

Is there an easy way to change how getCachePath generates the filenames so I can make them so that nginx can find them?

[reinink]

Hmm, shoot. Will think about how best to handle this.

[evilive3000]

+1. Are there some news?

[ddedic]

+1

[reinink]

I'm most likely going to add the ability to either group cached images into folders, or keep them all in one large folder. This will be part of the 1.0 release.

[joshrhykerd]

+1 for grouping cached images into folders. One large folder means all cached images will need deleted at the same time. Individually grouped cached folders would allow for deleting specific images leaving other cached images alone.

[reinink]

Okay, this feature has been added (ec5c79a). You can now enable and disable grouping into folders.

// Set using the factory
$server = ServerFactory::create([
    'group_cache_in_folders' => false
]);

// Set using setter method
$server->setGroupCacheInFolders(false);

[Zae]

I'm still not sure how ec5c79a will help with this. The biggest problem preventing try_files is still that there is no easy way for nginx to calculcate an md5 hash of the parameters used, so it can't translate image.jpg?resize to the actual file.

[reinink]

Yeah, and I'm not sure there is anything I can do to help there. What's your recommendation?

From what I understood with try_files, is it actually still hits PHP, but PHP returns a very simple header telling nginx where the actual real file is. But this isn't something I've actually done, so I cannot explain exactly how to do it.

[reinink]

Wait, sorry, it's not try_files, it's X-accel.

You can use Nginx's X-accel to still output the image using Nginx. This is done by basically passing the file path to Nginx using headers in your PHP script. Nginx then picks up those headers and takes care of the rest.

I believe Apache has similar functionality, called X-Sendfile.

[Zae]

No try_files is like apache's -f and -d flags. It will check all the files listed in the try_files array and if the file is found on the filesystem it will respond using this file.

Usually this is configured like this

try_files $uri $uri/ /index.php?$query_string;

This way nginx will first check if the actual file exists on the filesystem, then check if there is a directory with the same name and serve the index from this directory and lastly serve the page using index.php with the path as a php query string.

Only using the last try it will use php to serve the file, however if the actual file could be found this will speed up the process of serving files immensely because php is quite slow in comparison to nginx.

What you were thinking about was indeed the X-Accel header, this way you can let nginx serve the file from the filesystem instead of letting php fetch the file and transfer it to nginx from RAM.

I'll try to think about how we can encode the params in the filename in a way nginx can understand.

[reinink]

Yes, sorry, I was incorrect on the try_files directive.

At the end of the day, having Nginx (or Apache) pick up the work after the first manipulation has been done probably isn't very feasible with Glide, since it works by hitting your PHP server. If you require that sort of fine tune performance tweaking, this library may not be best suited for your needs.

[argb]

Most of time we need to get benefits from nginx to serve image files, like @Zae said, php is quite slow in comparison to nginx, if i have to fetch images use PHP, most of time,I have to give up the library. So I think it's very import to support this feature.

[cdowdy]

sorry to pull up a zombie/closed thread. But I've been thinking about this and when @reinink mentioned x-accel-redirect for nginx and x-sendfile for apache it got me to dig in and actually check it out and see if this could actually be implemented.

All my "testing" was done in a symfony/silex based app (bolt cms) using nginx 1.10.2 and PHP 7.0.4

So here it goes. I'll also tag the other issue(s) with try_files in them so those others can see :) #106

For this to work you'll need to save files with their extensions

// your service provider if you use it/howerver its registered in your "app" 
// this is using the symfony streamed response 
  return ServerFactory::create([
    'response' => new SymfonyResponseFactory($app['request']),
    'source' => $Filesystem, 
    'cache' => $Filesystem,
    'cache_path_prefix' => '.cache',
    'cache_with_file_extensions' => true,
    'base_url' => '/img/',
  ]); 

// or the example from glide's docs using symfony  

use League\Glide\ServerFactory;
use League\Glide\Responses\SymfonyResponseFactory;

$server = ServerFactory::create([
    'response' => new SymfonyResponseFactory()
    'source' => $Filesystem, 
    'cache' => $Filesystem,
    'cache_path_prefix' => '.cache',
    'cache_with_file_extensions' => true,
    'base_url' => '/img/',

]);

// or laravel  
use League\Glide\ServerFactory;
use League\Glide\Responses\LaravelResponseFactory;

$server = ServerFactory::create([
    'response' => new LaravelResponseFactory(app('request')),
    'source' => $Filesystem, 
    'cache' => $Filesystem,
    'cache_path_prefix' => '.cache',
    'cache_with_file_extensions' => true,
    'base_url' => '/img/',
]);

Now I'm not using laravel when I tested this out so your mileage may vary :)

in glide-symfony package we need to add a few headers.

• We need to add X-Accel-Redirect & for Apache X-Sendfile

// in league/glide-symfony/src/Responses/SymfonyResponseFactory.php
// @link https://github.com/thephpleague/glide-symfony/blob/master/src/Responses/SymfonyResponseFactory.php#L32-L57

public function create(FilesystemInterface $cache, $path)
    {
        $stream = $cache->readStream($path);

        $response = new StreamedResponse();
        $response->headers->set('Content-Type', $cache->getMimetype($path));
        $response->headers->set('Content-Length', $cache->getSize($path));
        $response->headers->set('X-Accel-Redirect', $path );
        $response->headers->set('X-Sendfile', $path );
       // rest of it below

Nginx Setup

this is unchanged from @Zae 's mentioned try_files. So similar to:

location ~* /img/(.*)$ {
  try_files $uri $uri/ /index.php?$query_string;
}

and if you use this you'll get the Redirected file (in nginx) with a few caveats.

1. The cache path is whatever is set in this examples cache_path_prefix so it'll be .cache/filename.jpg/hashed-filename.jpg. If your cache is a sub directory of your source directory (/files/.cache/) then you'll have the wrong path for the redirect. We need to get the full cache path somehow

Example being used

Here just to show it's "working" I've created an image that would be different from the image being requested. It's simply an image with a blue background with x-accel/x-sendfile in it.

Normal Glide usage doing a "hard refresh"

With X-Accel-Redirect Added to The response and again doing a "hard refresh"

---

The image served with x-accel/x-sendfile need to be in the cache. I haven't really dove into the performance aspects of this and if it's significantly better than what is currently provided by glide.

Also there might be alternative nginx/apache configs that will be faster than using try_files I'm not really sure since I haven't explored it besides just testing this out.

Nginx suggests to use an alternative location block syntax found here https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/ and their sendfile page https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/ . There is also additional headers to set. Those may impact performance, again I'm not sure since I haven't really toyed enough with it nor have I measured anything.

Hopefully someone better versed in profiling php applications can take a look at this and get some actual concrete numbers. Plus I edited a composer package to test this out so it'll need to be approved and added to any response factories which takes a lot more testing. More than I'm currently capable of.

Try it out yourselves! Hopefully this gets someone on a path to knocking this out!

[Zae]

@cdowdy this is great, thanks for the research, will be trying this out real soon!

[cdowdy]

@Zae cool, thanks!

If glide saved images with the query string inside a filename directory, that is if say the image query string would be appended to the filename before the extension img/filename.jpg/filename?W=200.jpg or the URL would be /img/filename{with or without extension)/w=200&h=200/filename.jpg then we could use the actual try_files directive since a request to /img/filename.jpg/filename?w=200.jpg or /img/filename/modifications/file.jpg could find the actual image if it exists before hitting PHP, but off the top of my head this could get ugly quickly with a a lot of modifications plus watermarks etc. You could hash the second examples modifications part to maybe shorten it. Which could also work as the signature?

You'd also need to have the actual img folder be a publicly accessible directory. The cache would also need to be named img while still being able to get your source images from either a /files/ or /source/` directory or wherever your images are stored.

The query string before the extension could also act as a version strategy instead of hashed names just more human readable

But all that is a major reconstruction of glide and would break existing uses

Using x-accel you still have to hit PHP since the response needs to send the header. If I get time I'll try to get some user timings through JavaScript and see if there is a difference in image loading

[Zae]

@cdowdy Yes, still needing to hit php is a shame, but I wonder what happens if we put fastcgi_cache in front of that, will nginx duplicate the entire file in it's cache or just the x-accel header and fetch the actual file from it's original location.

[reinink]

Hey all! Some awesome ideas in here. Funny enough I've already been planning Glide 2.0, which I hope will make serving images via the web server much easier. Please see an outline of my ideas here: #170

[argb]

I think the most easy way is keep the image accessing path same as the laravel route, which means defining a laravel route which looks like
Route::get('uploads/avatars/{path}/{name}', 'ImageController@image');
the the real url maybe http://xxx/public/image/w100-h100-filter/w100-h100-filter_iamanimage.jpg

if the file exists, of course nginx will response it back directly, but if file doesn't exist, the route will be hit and work, then we can do some work in the route method to generate the image file.
how to generate depend on the path parameter or file name which taking some information, we can analyze the parameters to get the necessary informations.

The principle same as Cache and Database, we first access the Cache such as redis, if no data, we get from Database such as mysql and then fill it to Cache.

I think this is the most natural and easy to implementing way.