-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speeding up generating compressed files #148
Comments
Also worth noting is that if I switch from |
Breakdown of
So this is all on Brotli, and not due to the filesystem walking/reading parts or gzip (albeit the standalone compressor example here was just for 35 files; but even for a 10,000 file directory Brotli compression times would dwarf anything else even if the filesystem walking happened to be inefficient). |
@evansd I have a multi-threading solution locally that uses Would you be open to me adding a dependency for Python 2 only on the futures package (which is a backport of Python 3 |
@edmorley Thanks a lot for this Ed, and for the other work you've been doing on whitenoise recently. Sorry I haven't responded sooner; things have been a bit busy lately. Yes, I'd be open to adding a dependency on futures. In general I like the fact that whitenoise is dependency-free, but backports of the Python 3 stdlib are a different case and I don't think it's a problem to add those. |
Really, the compression level should be configurable. https://github.com/evansd/whitenoise/blob/master/whitenoise/compress.py#L84 |
I've taken a stab at processing files in parallel in #484. |
In a project I work on, we use both
CompressedStaticFilesMixin
and the standalone compressor (python -m whitenoise.compress <DIR>
) during Heroku deployments.At the moment these steps are a considerable percentage (30-40%) of our deployment times.
For example using Python 2.7.13, Django 1.11.5, WhiteNoise master, Brotli 0.6.0, a Heroku-16 one-off performance-m dyno (2 cores, 2.5GB RAM, Ubuntu 16.04) with the static files directory cleared (to emulate deployment, since state intentionally isn't carried over):
As a baseline, using the stock Django
ManifestStaticFilesStorage
results in:For the above, the 202 files output from
ManifestStaticFilesStorage
have a combined file-size of 15MB.Moving onto the standalone compressor (which we use on the output of a webpack build, for the SPA part of the project):
Ideas off the top of my head to speed this up:
concurrent.futures
or similar to take advantage of all coresscantree()
implementation might be faster than compress.py'sos.walk()
plus later statsWHITENOISE_KEEP_ONLY_HASHED_FILES
and Try to work around leftover intermediate ManifestStaticFilesStorage files #147)CompressedStaticFilesMixin
and the CLI version, to double check that most of the time is indeed being spent in the compiled gzip/brotli code and not somewhere unexpected.The text was updated successfully, but these errors were encountered: