Question: producing https addresses using url_for

[cglacet]

I'm looking for a way to produce static file routes with https prefixes instead of the default http. For example, say I have the following template:

<script src="{{ url_for('static', path='file.js') }}"></script>

I would like it to produce:

<script src="https://my-app.herokuapp.com/static/file.js"></script>

I don't find information about this on the documentation, so I was wondering if it was possible? I can't use the <script src="/static/file.js"></script> form because I want to use the generated html on a different domain.

In case it makes a difference: I'm using fastAPI and the app is running on heroku.

[devsetgo]

Did you ever get an answer on this? I am having a similar problem using jwilder/nginx-proxy as it redirects automatically to https, but the references are to http://domain.name instead of https://domain.name.

The difference is I am using Starlette and not FastAPI.

[tiangolo]

Note: If in a hurry, skip to the end.

The way this information is passed to the app is via the ASGI spec itself. The server (Uvicorn, Hypercorn, or any other) is the one that tells the app if it is running on HTTPS or HTTP.

If you provide the HTTPS certificates to the ASGI server (e.g. Uvicorn) it will know it is running on HTTPS and pass that information to the framework (Starlette, FastAPI, or anything else).

But it's very common to have a "TLS Proxy" on top of it. That would be something that has the HTTPS certificates, handles the connection, and passes the pure HTTP to the thing running behind (in this case, Uvicorn, running your Starlette/FastAPI app). Examples of programs that can run as TLS Proxies include Nginx, HAProxy, Traefik (I recommend Traefik 😉 ). The same would be done by Heroku or jwilder's Docker image (based on Nginx).

But these TLS Proxies (and actually many other layers and servers) create some HTTP headers to let the thing that runs after them know that they are handling HTTPS for them.

But by default, none of the intermediate parts (Nginx, Traefik, Uvicorn) receive and accept those HTTPS headers from outside, as that would be a security risk. But if you know that the specific part (e.g. Uvicorn) is behind a TLS Proxy, you can normally configure/override it to receive those HTTP headers about the HTTPS connection.

In Uvicorn, the command parameter is --proxy-headers.

[devsetgo]

@tiangolo - Thank you for the response and really like FastAPI by the way. This solves the problem. I also temporarily solved it by adding to my template until I make the changes in Nginx or switch to Traefik (been meaning to try it).

In Flask the URL would be served up relative , but in Starlette it is served up absolute. Would it not be better for it to be relative vs absolute?

Maybe it should just be an option in how static files are severed. Something like this.
app = Router(routes=[
Mount('/static', app=StaticFiles(directory='static', absolute=False), name="static"),
])

If I am correct in how I think Starlette is operating it would be good to update the static file documentation to include that it is absolute path and not relative. Plus how to get Uvicorn use --proxy-headers.

[tiangolo]

@devsetgo if you want to use relative URLs, I think you can just write them as is in your template, you then don't need to use url_for.

I think the main thing url_for does is add the corresponding http or https and the domain that was requested (that is not known in the code before).

But with relative URLs you don't need those, so I think you can just use the explicit relative URL.

[tomchristie]

Using --proxy-headers is the right way to resolve this.

However it might be nice to not have to worry about this, and for us to just serve up relative urls by default, since there's less potential for misconfiguration that way.

We could still provide an alternative for absolute URLs, but we could more throughly document how to ensure it's properly configured.

Also relevant is encode/uvicorn#369, since that'd mean that we're getting the configuration correct automatically more often.

[i4x]

@tiangolo why the FastAPI doesn't work correctly, even when --proxy-headers is passed to uvicorn directly? The request.url, when imported as described, doesn't return the right scheme. (Sorry for saying this on Starlette's side, but relative URLs are not enough. Starlette works great, and has no issue here.)

[tomchristie]

@i4x Could you make sure to update to the latest version of uvicorn? What platform is the service deployed to? Can you output request.headers and request.scope so we can see exactly what's being passed through?

[tomchristie]

Note that the latest version of uvicorn now matches gunicorn behaviour - by default it will accept any X-Forwarded-For and X-Forwarded-Scheme headers if the client IP is in --forwarded_allow_ips (Defaults to the $FORWARDED_ALLOW_IPS environment variable, or else "127.0.0.1")

[i4x]

The version tested in the parent was uvicorn==0.10.2, that's installed automatically, when I install fastapi==0.42.0. After update to uvicorn-0.10.3, the behavior didn't change. I'm using Caddy as proxy (and it works well with uvicorn==0.9.1), and what's in request.headers under uvicorn==0.10.2, is:

{"host":"mydomain.net","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cache-control":"max-age=0","cookie":"SESS99def2ca8b51de44904e6acf20ead392=8t5neatlj9ffn43933toggif16","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.254","x-forwarded-proto":"https","x-real-ip":"192.168.1.254"}

The request.scope under uvicorn==0.10.2, was:

{'type': 'http', 'http_version': '1.1', 'server': ('192.168.1.103', 1717), 'client': ('192.168.1.185', 42582), 'scheme': 'http', 'method': 'GET', 'root_path': '', 'path': '/items/1', 'raw_path': b'/items/1', 'query_string': b'', 'headers': [(b'host', b'mydomain.net'), (b'user-agent', b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept-language', b'en-US,en;q=0.9'), (b'cache-control', b'max-age=0'), (b'cookie', b'SESS99def2ca8b51de44904e6acf20ead392=8t5neatlj9ffn43933toggif16'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'upgrade-insecure-requests', b'1'), (b'x-forwarded-for', b'192.168.1.254'), (b'x-forwarded-proto', b'https'), (b'x-real-ip', b'192.168.1.254')], 'fastapi_astack': <contextlib.AsyncExitStack object at 0x7fbdd5f5b6d0>, 'app': <fastapi.applications.FastAPI object at 0x7fbdd8f7a410>, 'router': <fastapi.routing.APIRouter object at 0x7fbdd8f7a610>, 'endpoint': <function read_root at 0x7fbdd5f53cb0>, 'path_params': {'item_id': '1'}}

---

Downgrading the default uvicorn installed by FastAPI to uvicorn==0.9.1 resolved the issue. The request.headers under uvicorn==0.9.1 was:

{"host":"mydomain.net","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cache-control":"max-age=0","cookie":"SESS99def2ca8b51de44904e6acf20ead392=8t5neatlj9ffn43933toggif16","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.254","x-forwarded-proto":"https","x-real-ip":"192.168.1.254"}

The request.scope under uvicorn==0.9.1 was:

{'type': 'http', 'http_version': '1.1', 'server': ('192.168.1.103', 1717), 'client': ('192.168.1.254', 0), 'scheme': 'https', 'method': 'GET', 'root_path': '', 'path': '/items/1', 'raw_path': b'/items/1', 'query_string': b'', 'headers': [(b'host', b'mydomain.net'), (b'user-agent', b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept-language', b'en-US,en;q=0.9'), (b'cache-control', b'max-age=0'), (b'cookie', b'SESS99def2ca8b51de44904e6acf20ead392=8t5neatlj9ffn43933toggif16'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'upgrade-insecure-requests', b'1'), (b'x-forwarded-for', b'192.168.1.254'), (b'x-forwarded-proto', b'https'), (b'x-real-ip', b'192.168.1.254')], 'fastapi_astack': <contextlib.AsyncExitStack object at 0x7fa8cb9c3e90>, 'app': <fastapi.applications.FastAPI object at 0x7fa8cee52250>, 'router': <fastapi.routing.APIRouter object at 0x7fa8ce531550>, 'endpoint': <function read_root at 0x7fa8cb9c6320>, 'path_params': {'item_id': '1'}}

[i4x]

Note that the latest version of uvicorn now matches gunicorn behaviour - by default it will accept any X-Forwarded-For and X-Forwarded-Scheme headers if the client IP is in --forwarded_allow_ips (Defaults to the $FORWARDED_ALLOW_IPS environment variable, or else "127.0.0.1")

Wonderful. So, under uvicorn==0.10.2, in the systemd, adding Environment="FORWARDED_ALLOW_IPS=192.168.1.185" resolved the problem for me, and now the request.url has correct scheme. :) Thanks, @tomchristie

[tomchristie]

Wonderful! Thanks for the feedback. ✨

[hamx0r]

Here's a note for Nginx users: be sure to configure your "server" config section to actually forward the X-Forwarded-Proto and X-Forwarded-For headers so that uvicorn can get them. Here's an example:

 server {
        server_name example.com
        location / {
            proxy_redirect     off;
# These are the critical headers needed by uvicorn to honor HTTPS in url_for :
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
# These are just some other headers you may find useful
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-Host $server_name;

           ...
        }


    listen 443 ssl; 

Lastly, uvicorn uses X-Forwarded-Proto, not X-Forwarded-Scheme, though the Nginx variable is called $scheme

[nickleman]

@devsetgo if you want to use relative URLs, I think you can just write them as is in your template, you then don't need to use url_for.

I think the main thing url_for does is add the corresponding http or https and the domain that was requested (that is not known in the code before).

But with relative URLs you don't need those, so I think you can just use the explicit relative URL.

@tiangolo 'url_for' is supposed to be a layer between the routing and the templates/web pages. that's why you reference the function name in the call to url_for instead of the path. It would be nice if url_for worked the same way as it does in flask where it returns a relative link, unless there is something that drives it to an absolute, or at least an argument to return relative instead of absolute. Maybe it would have to be a filter instead so as not to interfere with any key,value pairs that someone would want to use. May the default would be relative with a filter to pass it through to force an absolute (and maybe even a specific protocol: http/https).

[elmcrest]

Note that the latest version of uvicorn now matches gunicorn behaviour - by default it will accept any X-Forwarded-For and X-Forwarded-Scheme headers if the client IP is in --forwarded_allow_ips (Defaults to the $FORWARDED_ALLOW_IPS environment variable, or else "127.0.0.1")

Wonderful. So, under uvicorn==0.10.2, in the systemd, adding Environment="FORWARDED_ALLOW_IPS=192.168.1.185" resolved the problem for me, and now the request.url has correct scheme. :) Thanks, @tomchristie

In my case, using traefik and uvicorn, I could set the environment variable in my docker-compose.yml

[abe-winter]

fwiw I solved this in my own case by replacing request.url_for(...) with str(request.app.url_path_for(...)). url_path_for gives a relative path and doesn't require the framework to correctly infer http / https.

[wookiesh]

Unfortunately, I'm still facing this issue.
General description of the layout, 5 nodes docker swarm, ingress network active, traefik as tls terminating proxy and dispatcher to 5 instances of a fastapi based container.

I tried setting the proxy headers and forwarded-allow-ips in the entrypoint:
ENTRYPOINT [ "uvicorn", "--host", "0.0.0.0", "--proxy-headers", "--forwarded-allow-ips='*'" ]

but unfortunately, while everything is sent, received over https, the URL's produced by url_for() are still constructed with HTTP.

Have I missed some traefik required setting or something else ?
And sorry to comment on a closed issue.

[dnshio]

Same here. fastapi==0.103.1

I can see 'x-forwarded-proto': 'https' in the header but it's still generating http URLs.
I have configured uvcorn with --proxy-headers

Am I missing something?

Also, I share the same view as others in this thread. url_for should just generate relative URLs. The purpose of url_for was to refer to endpoints by routes/functions rather than hardcoding the URL. Generating a full scheme URL adds zero value in my opinion.

[Chris927]

@wookiesh, @dnshio, you could try this:

As parameter for --forwarded-allow-ips, Instead of '*', just use * (without the quotes), as in --forwarded-allow-ips=*.

My reasoning for why this may work: The single quotes '' around * may be required when uvicorn is invoked through a shell, to prevent the shell interpreting the * as "all files that match" (see bash's Filename Expansion for reference). As you use ENTRYPOINT in a Dockerfile, no shell is involved in the process: You should specify the value for the parameter without quotes.

[hemantapkh]

@tiangolo I am using gunicorn with uvicorn workers and the above mentioned solution is not working for me. However, I have achieved this by setting up a middleware and changing the scheme value from http to https.

@app.middleware("http")
async def middleware_events(request: Request, call_next):
    if settings.fastapi_env != "local":
        request.scope["scheme"] = "https"
        
    response = await call_next(request)

    return response

Please advice if this modification could potentially impact other parts of the application that rely on the original scope.schema value.

[marcelloinfante]

@wookiesh, @dnshio, you could try this:

As parameter for --forwarded-allow-ips, Instead of '*', just use * (without the quotes), as in --forwarded-allow-ips=*.

My reasoning for why this may work: The single quotes '' around * may be required when uvicorn is invoked through a shell, to prevent the shell interpreting the * as "all files that match" (see bash's Filename Expansion for reference). As you use ENTRYPOINT in a Dockerfile, no shell is involved in the process: You should specify the value for the parameter without quotes.

This worked for me! Thank you, @Chris927!

[wikiselev]

@wookiesh, @dnshio, you could try this:
As parameter for --forwarded-allow-ips, Instead of '*', just use * (without the quotes), as in --forwarded-allow-ips=*.
My reasoning for why this may work: The single quotes '' around * may be required when uvicorn is invoked through a shell, to prevent the shell interpreting the * as "all files that match" (see bash's Filename Expansion for reference). As you use ENTRYPOINT in a Dockerfile, no shell is involved in the process: You should specify the value for the parameter without quotes.

This worked for me! Thank you, @Chris927!

Worked for me, too! Thanks!

[rakeshsahni]

@wookiesh, @dnshio, you could try this:

As parameter for --forwarded-allow-ips, Instead of '*', just use * (without the quotes), as in --forwarded-allow-ips=*.

My reasoning for why this may work: The single quotes '' around * may be required when uvicorn is invoked through a shell, to prevent the shell interpreting the * as "all files that match" (see bash's Filename Expansion for reference). As you use ENTRYPOINT in a Dockerfile, no shell is involved in the process: You should specify the value for the parameter without quotes.

Resolving the FastAPI with Jinja2 Integration Issue

Understanding the Problem

When you don't specify the following settings in your FastAPI application:

1. "proxy_headers": True, (Corresponding to --proxy-headers)
2. "forwarded_allow_ips": "*", (Corresponding to --forwarded-allow-ips=*)

The application will redirect the URL to http://localhost:port_no or IP:port instead of the desired URL. This can cause issues when you need to serve SSL certificates or static files for your Jinja2 frontend.

Resolving the Issue

To fix this issue, you need to make the following changes:

1. Set the Proxy Headers and Forwarded Allow IPs:

   • In your FastAPI application, make sure to set the "proxy_headers" and "forwarded_allow_ips" configuration options.
   • This ensures that your application handles the correct URL, even when deployed behind a proxy (like Nginx).

2. Use Gunicorn as the Process Manager:

   • Instead of using Uvicorn as the process manager, consider using Gunicorn.
   • Gunicorn is generally more efficient than Uvicorn for production environments, as it can better utilize system resources.

3. Create a Custom Uvicorn Worker:

   • Create a custom Uvicorn worker class that inherits from UvicornWorker and sets the necessary configuration options.
   • This allows you to centralize the configuration settings and apply them consistently across your application.

Here's an example of the custom Uvicorn worker implementation:

# custom_worker.py
from uvicorn.workers import UvicornWorker

class CustomUvicornWorker(UvicornWorker):
    CONFIG_KWARGS = {
        "loop": "auto",  # Use 'auto' to automatically choose the best loop, 'uvloop' can be specified for performance.
        "http": "auto",  # Use 'auto' to automatically choose the best HTTP protocol support, 'httptools' can be specified.
        "lifespan": "on",  # 'on' to enable lifespan support.
        "proxy_headers": True,  # Corresponding to `--proxy-headers`
        "forwarded_allow_ips": "*",  # Corresponding to `--forwarded-allow-ips=*`
    }

4. Set up the Systemd Service:
   • Create a Systemd service file to manage your FastAPI application.
   • In the service file, use the custom Uvicorn worker you created earlier.

Here's an example Systemd service file:

# /etc/systemd/system/website.service
[Unit]
Description=Gunicorn instance to serve web
After=network.target

[Service]
User=ubuntu_username
Group=www-data
WorkingDirectory=/home/ubuntu_username/web
Environment="PATH=/home/ubuntu_username/web/venv/bin"
ExecStart=/home/ubuntu_username/web/venv/bin/gunicorn --workers 3 --worker-class custom_worker.CustomUvicornWorker app.main:app --bind 0.0.0.0:8000

[Install]
WantedBy=multi-user.target

By implementing these changes, your FastAPI application with Jinja2 integration should now work correctly, both locally and when deployed on an Ubuntu server with Nginx.

Happy Coding 🤗

[Zverik]

Alas this did not help me with an Apache reverse proxy. What helped are those two lines in the virtual host definition, along the ProxyPass:

ProxyPreserveHost On
RequestHeader setifempty X-Forwarded-Proto "https"

The latter requires "headers" module enabled.