diff --git a/README.rst b/README.rst index f2c02c2b..c304c005 100644 --- a/README.rst +++ b/README.rst @@ -31,7 +31,7 @@ useful on Heroku, OpenShift and other PaaS providers.) It's designed to work nicely with a CDN for high-traffic sites so you don't have to sacrifice performance to benefit from simplicity. -WhiteNoise works with any WSGI-compatible app but has some special auto-configuration +WhiteNoise works with any ASGI or WSGI compatible app but has some special auto-configuration features for Django. WhiteNoise takes care of best-practices for you, for instance: diff --git a/docs/asgi.rst b/docs/asgi.rst new file mode 100644 index 00000000..2398da61 --- /dev/null +++ b/docs/asgi.rst @@ -0,0 +1,76 @@ +Using WhiteNoise with any ASGI application +========================================== + +.. note:: These instructions apply to any ASGI application. However, for Django + applications you would be better off using the :doc:`WhiteNoiseMiddleware + ` class which makes integration easier. + +To enable WhiteNoise you need to wrap your existing ASGI application in a +WhiteNoise instance and tell it where to find your static files. For example: + +.. code-block:: python + + from whitenoise import AsgiWhiteNoise + + from my_project import MyASGIApp + + application = MyASGIApp() + application = AsgiWhiteNoise(application, root="/path/to/static/files") + application.add_files("/path/to/more/static/files", prefix="more-files/") + +On initialization, WhiteNoise walks over all the files in the directories that have +been added (descending into sub-directories) and builds a list of available static files. +Any requests which match a static file get served by WhiteNoise, all others are passed +through to the original ASGI application. + + +.. tip:: ``AsgiWhiteNoise`` inherits all interfaces from WSGI ``WhiteNoise`` but adds + support for ASGI applications. See the :doc:`WSGI WhiteNoise documentation ` for + more details. + + +AsgiWhiteNoise API +------------------ + +``AsgiWhiteNoise`` inherits its interface from WSGI ``WhiteNoise``, however, ``application`` +must be an ASGI application. + +See the section on WSGI ``WhiteNoise`` :ref:`interface ` for details. + + +Compression Support +-------------------- + +See the section on WSGI ``WhiteNoise`` :ref:`compression support ` for details. + + +Caching Headers +--------------- + +See the section on WSGI ``WhiteNoise`` :ref:`caching headers ` for details. + + +Index Files +----------- + +See the section on WSGI ``WhiteNoise`` :ref:`index files ` for details. + + +Using a Content Distribution Network +------------------------------------ + +See the instructions for :ref:`using a CDN with Django ` . The same principles +apply here although obviously the exact method for generating the URLs for your static +files will depend on the libraries you're using. + + +Redirecting to HTTPS +-------------------- + +See the section on WSGI ``WhiteNoise`` :ref:`redirecting to HTTPS ` for details. + + +Configuration attributes +------------------------ + +See the section on WSGI ``WhiteNoise`` :ref:`configuration attributes ` for details. diff --git a/docs/changelog.rst b/docs/changelog.rst index 65e0ef39..38538f1c 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -180,7 +180,7 @@ Other changes include: ``wsgi.py``. See the :ref:`documentation ` for more details. - (The :doc:`pure WSGI ` integration is still available for non-Django apps.) + (The :doc:`pure WSGI ` integration is still available for non-Django apps.) * The ``whitenoise.django.GzipManifestStaticFilesStorage`` alias has now been removed. Instead you should use the correct import path: diff --git a/docs/django.rst b/docs/django.rst index 3b6e6421..2e4222f3 100644 --- a/docs/django.rst +++ b/docs/django.rst @@ -2,7 +2,7 @@ Using WhiteNoise with Django ============================ .. note:: To use WhiteNoise with a non-Django application see the - :doc:`generic WSGI documentation `. + :doc:`generic WSGI documentation ` or the :doc:`generic ASGI documentation `. This guide walks you through setting up a Django project with WhiteNoise. In most cases it shouldn't take more than a couple of lines of configuration. diff --git a/docs/index.rst b/docs/index.rst index 1c52bbea..17f6559a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -79,8 +79,28 @@ WhiteNoise instance and tell it where to find your static files. For example: application = WhiteNoise(application, root="/path/to/static/files") application.add_files("/path/to/more/static/files", prefix="more-files/") -And that's it, you're ready to go. For more details see the :doc:`full -documentation `. +And that's it, you're ready to go. For more details see the :doc:`full WSGI +documentation `. + + +QuickStart for other ASGI apps +------------------------------ + +To enable WhiteNoise you need to wrap your existing ASGI application in a +WhiteNoise instance and tell it where to find your static files. For example: + +.. code-block:: python + + from whitenoise import AsgiWhiteNoise + + from my_project import MyASGIApp + + application = MyASGIApp() + application = AsgiWhiteNoise(application, root="/path/to/static/files") + application.add_files("/path/to/more/static/files", prefix="more-files/") + +And that's it, you're ready to go. For more details see the :doc:`full ASGI +documentation `. Using WhiteNoise with Flask @@ -94,7 +114,7 @@ the standard WSGI protocol it is easy to integrate with WhiteNoise (see the Compatibility ------------- -WhiteNoise works with any WSGI-compatible application and is tested on Python +WhiteNoise works with any ASGI or WSGI compatible application and is tested on Python **3.8** – **3.12**, on both Linux and Windows. Django WhiteNoiseMiddleware is tested with Django versions **3.2** --- **4.2** @@ -202,6 +222,7 @@ MIT Licensed self django - base + wsgi + asgi flask changelog diff --git a/docs/base.rst b/docs/wsgi.rst similarity index 99% rename from docs/base.rst rename to docs/wsgi.rst index b0b272ae..0c5e8bbc 100644 --- a/docs/base.rst +++ b/docs/wsgi.rst @@ -27,12 +27,14 @@ See the sections on :ref:`compression ` and :ref:`caching for further details. +.. _interface: + WhiteNoise API -------------- .. class:: WhiteNoise(application, root=None, prefix=None, \**kwargs) - :param callable application: Original WSGI application + :param Callable application: Original WSGI application :param str root: If set, passed to ``add_files`` method :param str prefix: If set, passed to ``add_files`` method :param \**kwargs: Sets :ref:`configuration attributes ` for this instance @@ -146,6 +148,8 @@ apply here although obviously the exact method for generating the URLs for your files will depend on the libraries you're using. +.. _https: + Redirecting to HTTPS -------------------- diff --git a/setup.cfg b/setup.cfg index 241dda4a..d1a64d6c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,7 +1,7 @@ [metadata] name = whitenoise version = 6.5.0 -description = Radically simplified static file serving for WSGI applications +description = Radically simplified static file serving for ASGI or WSGI applications long_description = file: README.rst long_description_content_type = text/x-rst url = https://github.com/evansd/whitenoise @@ -26,6 +26,7 @@ classifiers = Programming Language :: Python :: 3.11 Programming Language :: Python :: 3.12 Programming Language :: Python :: Implementation :: CPython + Topic :: Internet :: WWW/HTTP :: ASGI :: Middleware Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware Typing :: Typed keywords = Django @@ -35,6 +36,8 @@ project_urls = [options] packages = find: +install_requires = + aiofiles>=22.1.0 python_requires = >=3.8 include_package_data = True package_dir = diff --git a/src/whitenoise/__init__.py b/src/whitenoise/__init__.py index 42ffb9d3..7e7f50a4 100644 --- a/src/whitenoise/__init__.py +++ b/src/whitenoise/__init__.py @@ -1,5 +1,6 @@ from __future__ import annotations -from .base import WhiteNoise +from .asgi import AsgiWhiteNoise +from .wsgi import WhiteNoise -__all__ = ["WhiteNoise"] +__all__ = ["AsgiWhiteNoise", "WhiteNoise"] diff --git a/src/whitenoise/asgi.py b/src/whitenoise/asgi.py new file mode 100644 index 00000000..65d0e2a2 --- /dev/null +++ b/src/whitenoise/asgi.py @@ -0,0 +1,91 @@ +from __future__ import annotations + +import asyncio + +from asgiref.compatibility import guarantee_single_callable + +from .string_utils import decode_path_info +from whitenoise.base import BaseWhiteNoise + +# This is the same size as wsgiref.FileWrapper +BLOCK_SIZE = 8192 + + +class AsgiWhiteNoise(BaseWhiteNoise): + user_app = None + + async def __call__(self, scope, receive, send): + # Ensure ASGI v2 is converted to ASGI v3 + if not self.user_app: + self.user_app = guarantee_single_callable(self.application) + + # Determine if the request is for a static file + path = decode_path_info(scope["path"]) + static_file = None + if scope["type"] == "http": + if self.autorefresh and hasattr(asyncio, "to_thread"): + # Use a thread while searching disk for files on Python 3.9+ + static_file = await asyncio.to_thread(self.find_file, path) + elif self.autorefresh: + static_file = self.find_file(path) + else: + static_file = self.files.get(path) + + # Serve static file if it exists + if static_file: + await AsgiFileServer(static_file)(scope, receive, send) + return + + # Serve the user's ASGI application + await self.user_app(scope, receive, send) + + +class AsgiFileServer: + """Simple ASGI application that streams a StaticFile over HTTP in chunks.""" + + def __init__(self, static_file): + self.static_file = static_file + + async def __call__(self, scope, receive, send): + # Convert ASGI headers into WSGI headers. Allows us to reuse all of our WSGI + # header logic inside of aget_response(). + wsgi_headers = { + "HTTP_" + key.decode().upper().replace("-", "_"): value.decode() + for key, value in scope["headers"] + } + + # Get the WhiteNoise file response + response = await self.static_file.aget_response(scope["method"], wsgi_headers) + + # Start a new HTTP response for the file + await send( + { + "type": "http.response.start", + "status": response.status, + "headers": [ + # Convert headers back to ASGI spec + (key.lower().encode(), value.encode()) + for key, value in response.headers + ], + } + ) + + # Head responses have no body, so we terminate early + if response.file is None: + await send({"type": "http.response.body", "body": b""}) + return + + # Stream the file response body + async with response.file as async_file: + while True: + chunk = await async_file.read(BLOCK_SIZE) + more_body = bool(chunk) + await send( + { + "type": "http.response.body", + "body": chunk, + "more_body": more_body, + } + ) + if not more_body: + break diff --git a/src/whitenoise/base.py b/src/whitenoise/base.py index 6bee7a74..34e8d047 100644 --- a/src/whitenoise/base.py +++ b/src/whitenoise/base.py @@ -6,18 +6,16 @@ from posixpath import normpath from typing import Callable from wsgiref.headers import Headers -from wsgiref.util import FileWrapper from .media_types import MediaTypes from .responders import IsDirectoryError from .responders import MissingFileError from .responders import Redirect from .responders import StaticFile -from .string_utils import decode_path_info from .string_utils import ensure_leading_trailing_slash -class WhiteNoise: +class BaseWhiteNoise: # Ten years is what nginx sets a max age if you use 'expires max;' # so we'll follow its lead FOREVER = 10 * 365 * 24 * 60 * 60 @@ -71,27 +69,8 @@ def __init__( if root is not None: self.add_files(root, prefix) - def __call__(self, environ, start_response): - path = decode_path_info(environ.get("PATH_INFO", "")) - if self.autorefresh: - static_file = self.find_file(path) - else: - static_file = self.files.get(path) - if static_file is None: - return self.application(environ, start_response) - else: - return self.serve(static_file, environ, start_response) - - @staticmethod - def serve(static_file, environ, start_response): - response = static_file.get_response(environ["REQUEST_METHOD"], environ) - status_line = f"{response.status} {response.status.phrase}" - start_response(status_line, list(response.headers)) - if response.file is not None: - file_wrapper = environ.get("wsgi.file_wrapper", FileWrapper) - return file_wrapper(response.file) - else: - return [] + def __call__(self, *args, **kwargs): + raise NotImplementedError("Subclasses must implement `__call__`") def add_files(self, root, prefix=None): root = os.path.abspath(root) diff --git a/src/whitenoise/middleware.py b/src/whitenoise/middleware.py index 3f5a8091..def9daa9 100644 --- a/src/whitenoise/middleware.py +++ b/src/whitenoise/middleware.py @@ -1,17 +1,26 @@ from __future__ import annotations +import asyncio +import concurrent.futures +import contextlib import os from posixpath import basename +from typing import AsyncIterable from urllib.parse import urlparse +import django +from aiofiles.base import AiofilesContextManager +from asgiref.sync import iscoroutinefunction +from asgiref.sync import markcoroutinefunction from django.conf import settings from django.contrib.staticfiles import finders from django.contrib.staticfiles.storage import staticfiles_storage from django.http import FileResponse from django.urls import get_script_prefix -from .base import WhiteNoise +from .asgi import BLOCK_SIZE from .string_utils import ensure_leading_trailing_slash +from .wsgi import WhiteNoise __all__ = ["WhiteNoiseMiddleware"] @@ -28,14 +37,58 @@ def set_headers(self, *args, **kwargs): pass +class AsyncWhiteNoiseFileResponse(FileResponse): + """ + Wrapper for Django's FileResponse that has a few differences: + - Doesn't use Django response headers (headers are already generated by WhiteNoise). + - Only generates responses for async file handles. + - Provides Django an async iterator for more efficient file streaming. + - Opens the file handle within the iterator to avoid WSGI thread ownership issues. + """ + + def _set_streaming_content(self, value): + # Make sure the value is an async file handle + if not isinstance(value, AiofilesContextManager): + self.file_to_stream = None + return super()._set_streaming_content(value) + + iterator = AsyncFileIterator(value) + + # Django < 4.2 doesn't support async iterators within `streaming_content`, so we + # must convert to sync + if django.VERSION < (4, 2): + iterator = AsyncToSyncIterator(iterator) + + super()._set_streaming_content(iterator) + + def set_headers(self, filelike): + pass + + if django.VERSION >= (4, 2): + + def __iter__(self): + """The way that Django 4.2+ converts from async to sync is inefficient, so + we override it with a better implementation. Django uses this method for all + WSGI responses.""" + try: + return iter(self.streaming_content) + except TypeError: + return iter(AsyncToSyncIterator(self.streaming_content)) + + class WhiteNoiseMiddleware(WhiteNoise): """ Wrap WhiteNoise to allow it to function as Django middleware, rather - than WSGI middleware. + than ASGI/WSGI middleware. """ - def __init__(self, get_response=None, settings=settings): + async_capable = True + sync_capable = True + + def __init__(self, get_response, settings=settings): self.get_response = get_response + if iscoroutinefunction(get_response): + markcoroutinefunction(self) try: autorefresh: bool = settings.WHITENOISE_AUTOREFRESH @@ -115,6 +168,12 @@ def __init__(self, get_response=None, settings=settings): self.add_files_from_finders() def __call__(self, request): + if iscoroutinefunction(self.get_response): + return self.acall(request) + else: + return self.call(request) + + def call(self, request): if self.autorefresh: static_file = self.find_file(request.path_info) else: @@ -123,6 +182,18 @@ def __call__(self, request): return self.serve(static_file, request) return self.get_response(request) + async def acall(self, request): + if self.autorefresh and hasattr(asyncio, "to_thread"): + # Use a thread while searching disk for files on Python 3.9+ + static_file = await asyncio.to_thread(self.find_file, request.path_info) + elif self.autorefresh: + static_file = self.find_file(request.path_info) + else: + static_file = self.files.get(request.path_info) + if static_file is not None: + return await self.aserve(static_file, request) + return await self.get_response(request) + @staticmethod def serve(static_file, request): response = static_file.get_response(request.method, request.META) @@ -134,6 +205,17 @@ def serve(static_file, request): http_response[key] = value return http_response + @staticmethod + async def aserve(static_file, request): + response = await static_file.aget_response(request.method, request.META) + status = int(response.status) + http_response = AsyncWhiteNoiseFileResponse(response.file or (), status=status) + # Remove default content-type + del http_response["content-type"] + for key, value in response.headers: + http_response[key] = value + return http_response + def add_files_from_finders(self): files = {} for finder in finders.get_finders(): @@ -200,3 +282,51 @@ def get_static_url(self, name): return staticfiles_storage.url(name) except ValueError: return None + + +class AsyncFileIterator: + def __init__(self, file_context: AiofilesContextManager): + self.file_context = file_context + + async def __aiter__(self): + """Async iterator compatible with Django Middleware. Yields chunks of data from + the provided async file context manager.""" + async with self.file_context as async_file: + while True: + chunk = await async_file.read(BLOCK_SIZE) + if not chunk: + break + yield chunk + + +class AsyncToSyncIterator: + """Converts any async iterator to sync as efficiently as possible while retaining + full compatibility with any environment. + + Currently used to add aiofiles compatibility to Django WSGI and Django versions + that do not support __aiter__. + + This converter must create a temporary event loop in a thread for two reasons: + 1) Allows us to stream the iterator instead of buffering all contents in memory. + 2) Allows the iterator to be used in environments where an event loop may not exist, + or may be closed unexpectedly.""" + + def __init__(self, iterator: AsyncIterable): + self.iterator = iterator + + def __iter__(self): + # Create a dedicated event loop to run the async iterator on. + loop = asyncio.new_event_loop() + thread_executor = concurrent.futures.ThreadPoolExecutor( + max_workers=1, thread_name_prefix="WhiteNoise" + ) + + # Convert from async to sync by stepping through the async iterator and yielding + # the result of each step. + generator = self.iterator.__aiter__() + with contextlib.suppress(GeneratorExit, StopAsyncIteration): + while True: + yield thread_executor.submit( + loop.run_until_complete, generator.__anext__() + ).result() + loop.close() diff --git a/src/whitenoise/responders.py b/src/whitenoise/responders.py index 9501ea65..734f3cd9 100644 --- a/src/whitenoise/responders.py +++ b/src/whitenoise/responders.py @@ -12,6 +12,10 @@ from urllib.parse import quote from wsgiref.headers import Headers +import aiofiles +from aiofiles.base import AiofilesContextManager +from aiofiles.threadpool.binary import AsyncBufferedIOBase + class Response: __slots__ = ("status", "headers", "file") @@ -47,12 +51,16 @@ class SlicedFile(BufferedIOBase): been reached. """ - def __init__(self, fileobj, start, end): - fileobj.seek(start) + def __init__(self, fileobj: BufferedIOBase, start: int, end: int): self.fileobj = fileobj + self.seeked = False + self.start = start self.remaining = end - start + 1 def read(self, size=-1): + if not self.seeked: + self.fileobj.seek(self.start) + self.seeked = True if self.remaining <= 0: return b"" if size < 0: @@ -67,6 +75,45 @@ def close(self): self.fileobj.close() +class AsyncSlicedFile: + """ + Variant of `SlicedFile` that works as an async context manager for `aiofiles`. + + This class does not need a `close` or `__await__` method, since we always open + async file handle via context managers (`async with`). + """ + + def __init__(self, context_manager: AiofilesContextManager, start: int, end: int): + self.fileobj: AsyncBufferedIOBase # This is populated during `__aenter__` + self.seeked = False + self.start = start + self.remaining = end - start + 1 + self.context_manager = context_manager + + async def read(self, size=-1): + if not self.fileobj: # pragma: no cover + raise RuntimeError("Async file objects need to be open via `async with`.") + if not self.seeked: + await self.fileobj.seek(self.start) + self.seeked = True + if self.remaining <= 0: + return b"" + if size < 0: + size = self.remaining + else: + size = min(size, self.remaining) + data = await self.fileobj.read(size) + self.remaining -= len(data) + return data + + async def __aenter__(self): + self.fileobj = await self.context_manager + return self + + async def __aexit__(self, exc_type, exc, tb): + await self.fileobj.close() + + class StaticFile: def __init__(self, path, headers, encodings=None, stat_cache=None): files = self.get_file_stats(path, encodings, stat_cache) @@ -97,6 +144,33 @@ def get_response(self, method, request_headers): pass return Response(HTTPStatus.OK, headers, file_handle) + async def aget_response(self, method, request_headers): + """Variant of `get_response` that works with async HTTP requests. + To minimize code duplication, `request_headers` conforms to WSGI header spec.""" + if method not in ("GET", "HEAD"): + return NOT_ALLOWED_RESPONSE + if self.is_not_modified(request_headers): + return self.not_modified_response + path, headers = self.get_path_and_headers(request_headers) + if method != "HEAD": + # We do not await this async file handle to allow us the option of opening + # it in a thread later + file_handle = aiofiles.open(path, "rb") + else: + file_handle = None + range_header = request_headers.get("HTTP_RANGE") + if range_header: + try: + return await self.aget_range_response( + range_header, headers, file_handle + ) + except ValueError: + # If we can't interpret the Range request for any reason then + # just ignore it and return the standard response (this + # behaviour is allowed by the spec) + pass + return Response(HTTPStatus.OK, headers, file_handle) + def get_range_response(self, range_header, base_headers, file_handle): headers = [] for item in base_headers: @@ -113,6 +187,23 @@ def get_range_response(self, range_header, base_headers, file_handle): headers.append(("Content-Length", str(end - start + 1))) return Response(HTTPStatus.PARTIAL_CONTENT, headers, file_handle) + async def aget_range_response(self, range_header, base_headers, file_handle): + """Variant of `get_range_response` that works with async file objects.""" + headers = [] + for item in base_headers: + if item[0] == "Content-Length": + size = int(item[1]) + else: + headers.append(item) + start, end = self.get_byte_range(range_header, size) + if start >= end: + return await self.aget_range_not_satisfiable_response(file_handle, size) + if file_handle is not None: + file_handle = AsyncSlicedFile(file_handle, start, end) + headers.append(("Content-Range", f"bytes {start}-{end}/{size}")) + headers.append(("Content-Length", str(end - start + 1))) + return Response(HTTPStatus.PARTIAL_CONTENT, headers, file_handle) + def get_byte_range(self, range_header, size): start, end = self.parse_byte_range(range_header) if start < 0: @@ -151,6 +242,17 @@ def get_range_not_satisfiable_response(file_handle, size): None, ) + @staticmethod + async def aget_range_not_satisfiable_response(file_handle, size): + """Variant of `get_range_not_satisfiable_response` that works with + async file objects. Async file handles do not need to be closed, since they + are only opened via context managers while being dispatched.""" + return Response( + HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE, + [("Content-Range", f"bytes */{size}")], + None, + ) + @staticmethod def get_file_stats(path, encodings, stat_cache): # Primary file has an encoding of None @@ -241,6 +343,9 @@ def __init__(self, location, headers=None): def get_response(self, method, request_headers): return self.response + async def aget_response(self, method, request_headers): + return self.response + class NotARegularFileError(Exception): pass diff --git a/src/whitenoise/wsgi.py b/src/whitenoise/wsgi.py new file mode 100644 index 00000000..213eb99f --- /dev/null +++ b/src/whitenoise/wsgi.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +from wsgiref.util import FileWrapper + +from .base import BaseWhiteNoise +from .string_utils import decode_path_info + + +class WhiteNoise(BaseWhiteNoise): + def __call__(self, environ, start_response): + path = decode_path_info(environ.get("PATH_INFO", "")) + if self.autorefresh: + static_file = self.find_file(path) + else: + static_file = self.files.get(path) + if static_file is None: + return self.application(environ, start_response) + else: + return self.serve(static_file, environ, start_response) + + @staticmethod + def serve(static_file, environ, start_response): + response = static_file.get_response(environ["REQUEST_METHOD"], environ) + status_line = f"{response.status} {response.status.phrase}" + start_response(status_line, list(response.headers)) + if response.file is not None: + file_wrapper = environ.get("wsgi.file_wrapper", FileWrapper) + return file_wrapper(response.file) + else: + return [] diff --git a/tests/test_asgi.py b/tests/test_asgi.py new file mode 100644 index 00000000..f8a4b714 --- /dev/null +++ b/tests/test_asgi.py @@ -0,0 +1,122 @@ +from __future__ import annotations + +import asyncio +from pathlib import Path + +import pytest + +from .utils import AsgiReceiveEmulator +from .utils import AsgiScopeEmulator +from .utils import AsgiSendEmulator +from .utils import Files +from whitenoise.asgi import AsgiWhiteNoise + + +@pytest.fixture() +def test_files(): + return Files( + js=str(Path("static") / "app.js"), + ) + + +@pytest.fixture(params=[True, False]) +def application(request, test_files): + """Return an ASGI application that can serve the test files.""" + + async def asgi_app(scope, receive, send): + if scope["type"] != "http": + raise RuntimeError("Incorrect response type!") + + await send( + { + "type": "http.response.start", + "status": 404, + "headers": [[b"content-type", b"text/plain"]], + } + ) + await send({"type": "http.response.body", "body": b"Not Found"}) + + return AsgiWhiteNoise( + asgi_app, root=test_files.directory, autorefresh=request.param + ) + + +def test_get_js_static_file(application, test_files): + scope = AsgiScopeEmulator({"path": "/static/app.js"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.body == test_files.js_content + assert b"text/javascript" in send.headers[b"content-type"] + assert send.headers[b"content-length"] == str(len(test_files.js_content)).encode() + + +def test_user_app(application): + scope = AsgiScopeEmulator({"path": "/"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.body == b"Not Found" + assert b"text/plain" in send.headers[b"content-type"] + assert send.status == 404 + + +def test_ws_scope(application): + scope = AsgiScopeEmulator({"type": "websocket"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + with pytest.raises(RuntimeError): + asyncio.run(application(scope, receive, send)) + + +def test_head_request(application, test_files): + scope = AsgiScopeEmulator({"path": "/static/app.js", "method": "HEAD"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.body == b"" + assert b"text/javascript" in send.headers[b"content-type"] + assert send.headers[b"content-length"] == str(len(test_files.js_content)).encode() + assert len(send.message) == 2 + + +def test_small_block_size(application, test_files): + scope = AsgiScopeEmulator({"path": "/static/app.js"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + from whitenoise import asgi + + DEFAULT_BLOCK_SIZE = asgi.BLOCK_SIZE + asgi.BLOCK_SIZE = 10 + asyncio.run(application(scope, receive, send)) + assert send[1]["body"] == test_files.js_content[:10] + asgi.BLOCK_SIZE = DEFAULT_BLOCK_SIZE + + +def test_request_range_response(application, test_files): + scope = AsgiScopeEmulator( + {"path": "/static/app.js", "headers": [(b"range", b"bytes=0-13")]} + ) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.body == test_files.js_content[:14] + + +def test_out_of_range_error(application, test_files): + scope = AsgiScopeEmulator( + {"path": "/static/app.js", "headers": [(b"range", b"bytes=10000-11000")]} + ) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.status == 416 + assert send.headers[b"content-range"] == b"bytes */%d" % len(test_files.js_content) + + +def test_wrong_method_type(application, test_files): + scope = AsgiScopeEmulator({"path": "/static/app.js", "method": "PUT"}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(application(scope, receive, send)) + assert send.status == 405 diff --git a/tests/test_django_whitenoise.py b/tests/test_django_whitenoise.py index 2654424f..a9f92b32 100644 --- a/tests/test_django_whitenoise.py +++ b/tests/test_django_whitenoise.py @@ -1,21 +1,28 @@ from __future__ import annotations +import asyncio import shutil import tempfile from contextlib import closing from urllib.parse import urljoin from urllib.parse import urlparse +import brotli import pytest from django.conf import settings from django.contrib.staticfiles import finders from django.contrib.staticfiles import storage +from django.core.asgi import get_asgi_application from django.core.management import call_command from django.core.wsgi import get_wsgi_application from django.test.utils import override_settings from django.utils.functional import empty from .utils import AppServer +from .utils import AsgiAppServer +from .utils import AsgiReceiveEmulator +from .utils import AsgiScopeEmulator +from .utils import AsgiSendEmulator from .utils import Files from whitenoise.middleware import WhiteNoiseFileResponse from whitenoise.middleware import WhiteNoiseMiddleware @@ -62,6 +69,11 @@ def application(_collect_static): return get_wsgi_application() +@pytest.fixture() +def asgi_application(_collect_static): + return AsgiAppServer(get_asgi_application()) + + @pytest.fixture() def server(application): app_server = AppServer(application) @@ -84,6 +96,21 @@ def test_versioned_file_cached_forever(server, static_files, _collect_static): ) +def test_asgi_versioned_file_cached_forever_brotoli( + asgi_application, static_files, _collect_static +): + url = storage.staticfiles_storage.url(static_files.js_path) + scope = AsgiScopeEmulator({"path": url}) + receive = AsgiReceiveEmulator() + send = AsgiSendEmulator() + asyncio.run(asgi_application(scope, receive, send)) + assert brotli.decompress(send.body) == static_files.js_content + assert ( + send.headers.get(b"Cache-Control", b"").decode("utf-8") + == f"max-age={WhiteNoiseMiddleware.FOREVER}, public, immutable" + ) + + def test_unversioned_file_not_cached_forever(server, static_files, _collect_static): url = settings.STATIC_URL + static_files.js_path response = server.get(url) diff --git a/tests/utils.py b/tests/utils.py index 0db45d75..13b7179a 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -53,8 +53,21 @@ def close(self): self.server.server_close() +class AsgiAppServer: + def __init__(self, application): + self.application = application + + async def __call__(self, scope, receive, send): + if scope["type"] != "http": + raise RuntimeError("Incorrect response type!") + + # Remove the prefix from the path + scope["path"] = scope["path"].replace(f"/{AppServer.PREFIX}", "", 1) + await self.application(scope, receive, send) + + class Files: - def __init__(self, directory, **files): + def __init__(self, directory="", **files): self.directory = os.path.join(TEST_FILE_PATH, directory) for name, path in files.items(): url = f"/{AppServer.PREFIX}/{path}" @@ -63,3 +76,98 @@ def __init__(self, directory, **files): setattr(self, name + "_path", path) setattr(self, name + "_url", url) setattr(self, name + "_content", content) + + +class AsgiScopeEmulator(dict): + """Simulate a real scope. Individual scope values can be overridden by passing + a dictionary to the constructor.""" + + def __init__(self, scope_overrides: dict | None = None): + scope = { + "asgi": {"version": "3.0"}, + "client": ["127.0.0.1", 64521], + "headers": [ + (b"host", b"127.0.0.1:8000"), + (b"connection", b"keep-alive"), + ( + b"sec-ch-ua", + b'"Not/A)Brand";v="99", "Brave";v="115", "Chromium";v="115"', + ), + (b"sec-ch-ua-mobile", b"?0"), + (b"sec-ch-ua-platform", b'"Windows"'), + (b"dnt", b"1"), + (b"upgrade-insecure-requests", b"1"), + ( + b"user-agent", + b"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" + b" (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36", + ), + ( + b"accept", + b"text/html,application/xhtml+xml,application/xml;q=0.9,image/" + b"avif,image/webp,image/apng,*/*;q=0.8", + ), + (b"sec-gpc", b"1"), + (b"sec-fetch-site", b"none"), + (b"sec-fetch-mode", b"navigate"), + (b"sec-fetch-user", b"?1"), + (b"sec-fetch-dest", b"document"), + (b"accept-encoding", b"gzip, deflate, br"), + (b"accept-language", b"en-US,en;q=0.9"), + ], + "http_version": "1.1", + "method": "GET", + "path": "/", + "query_string": b"", + "raw_path": b"/", + "root_path": "", + "scheme": "http", + "server": ["127.0.0.1", 8000], + "type": "http", + } + + if scope_overrides: + scope.update(scope_overrides) + + super().__init__(scope) + + +class AsgiReceiveEmulator: + """Provides a list of events to be awaited by the ASGI application. This is designed + be emulate HTTP events.""" + + def __init__(self, *events): + self.events = [{"type": "http.connect"}] + list(events) + + async def __call__(self): + return self.events.pop(0) if self.events else {"type": "http.disconnect"} + + +class AsgiSendEmulator: + """Any events sent to this object will be stored in a list.""" + + def __init__(self): + self.message = [] + + async def __call__(self, event): + self.message.append(event) + + def __getitem__(self, index): + return self.message[index] + + @property + def body(self): + """Combine all HTTP body messages into a single bytestring.""" + return b"".join( + [message["body"] for message in self.message if message.get("body")] + ) + + @property + def headers(self): + """Return the headers from the first event.""" + return dict(self[0]["headers"]) + + @property + def status(self): + """Return the status from the first event.""" + return self[0]["status"]