deployment on nginx - https protocol

[jaysnm]

Hi
My appreciations to the team maintaining this project. It adds lots of capabilities not supported by rio-viz. Have found it incredible at what it does!

Problem description

I have been able to experiment COG tiler capabilities on local setup with nginx used as a proxy server. Moving to production, I introduced certificates routing all traffic to https. A /cog/metadata request returns expected data from my local setup (http) but fails on production setup (https) with

CURL error: Failed to connect to domain.xx port 443: Connection refused
'/vsicurl/https://domain.xxx/stac/catalog-item/my-item.tif' not recognized as a supported file format.

Same errors are emitted when calling tilejson endpoint to render tiles on map.

The /cog/viewer endpoint generates html content with tilejson_endpoint and metadata_endpoint defined on http protocol despite that I'm running the service on https.

const tilejson_endpoint = "http://jdomain.xx/cog/tilejson.json"
const metadata_endpoint = "http://domain.xx/cog/metadata"

Environment Information

I'm running developmentseed/titiler docker image with the port exposed being used by nginx to proxy_pass the service.

[vincentsarago]

@jaysnm I don't think this is a bug or has anything to do with TiTiler?

I think the error you are seeing is related to GDAL libcurl not being able to use ssl.

I'm going to move this to the discussion board and at the same time it could help if you provide more info about how you installed titiler. my first hint is you could try to set https://github.com/developmentseed/titiler/blob/master/Dockerfile#L3

[jaysnm]

Thanks @vincentsarago
Setting CURL_CA_BUNDLE environment variable works, however not consistently! An endpoint, say /cog/info?url=https://domain.xx/cogeo.tiff will return expected data sometime and other time return

CURL error: Failed to connect to domain.xx port 443: Connection refused
NoneType: None

or

162.11.55.1:39690 - "GET /cog/info.geojson?url=https%3A%2F%2Fdomain.xx%2Fcogeo.tif HTTP/1.1" 500
Read or write failed. IReadBlock failed at X offset 0, Y offset 0: IReadBlock failed at X offset 0, Y offset 1: TIFFReadEncodedTile() failed.
NoneType: None

CURL error is the most frequent even on tile generator endpoint /cog/tiles/WebMercatorQuad
these are the other environment variables I have on the container

- CPL_TMPDIR=/tmp
- GDAL_CACHEMAX=75%
- GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
- GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
- GDAL_HTTP_MULTIPLEX=YES
- GDAL_HTTP_VERSION=2
- PYTHONWARNINGS=ignore
- VSI_CACHE=TRUE
- VSI_CACHE_SIZE=536870912
- WORKERS_PER_CORE=1

[jaysnm]

Now works fine after doing a mix-match of tiangolo/uvicorn-gunicorn:python3.8 and osgeo/gdal:ubuntu-small-3.3.1 docker images to provide GDAL/OGR libs. Once again thanks @vincentsarago

[georgr]

I ran into a similar problem with nginx as reverse proxy. Some endpoints seem to subsequently call other endpoints and use http instead of https.

e.g.: putting a COG url into the https://titiler.xyz.dev/cog/viewer input box results in following error in the dev console in the browser.

Mixed Content: The page at 'https://titiler.xyz.dev/cog/viewer' was loaded over HTTPS, but requested an insecure resource 'http://titiler.xyz.dev/cog/metadata?url=https://example.com/cog.tiff&max_size=256'. This request has been blocked; the content must be served over HTTPS.

In this setup nginx terminates SSL and forwards the request via http to the titiler container.
The url_for function seems to get the http protocol via the request.base_url, since the request only sees the http request called by nginx and gives the http protocol to the cog_demo metadata context variable.

Does this assumption seem correct?

[vincentsarago]

@georgr This your assumption is correct. We could maybe add an environment variable used in

titiler/src/titiler/core/titiler/core/factory.py

Lines 123 to 129 in 2335048

	def url_for(self, request: Request, name: str, **path_params: Any) -> str:
	"""Return full url (with prefix) for a specific endpoint."""
	url_path = self.router.url_path_for(name, **path_params)
	base_url = str(request.base_url)
	if self.router_prefix:
	base_url += self.router_prefix.lstrip("/")
	return url_path.make_absolute_url(base_url=base_url)

to force HTTPS.

[georgr]

@vincentsarago Thank you for your review, I´ll try to create a pull request

[vincentsarago]

@georgr just looking at starlette repo there seems to have some discussion related. could you check them before?
encode/starlette#604
encode/starlette#538

[georgr]

@vincentsarago thank you for hinting the starlette issues!

After digging the starlette docs I found the solution:

The reverse proxy has to set the x-forwarded-proto header. ingress-nginx in k8s is providing the header with the default configuration. The headers that are forwarded can be checked f.e. with the echo-server container.

To get titiler to use the x-forwarded-proto header, the environment variable FORWARDED_ALLOW_IPS has to be set to either the ip address of the reverse proxy or in a k8s environment to "*" (k8s distribute the container with a dynamic internal ip).

Working k8s definition to get it up and running:

spec:
      containers:
      - image: developmentseed/titiler:latest
        imagePullPolicy: Always
        name: titiler
        ports:
        - containerPort: 80
        env:
        - name: FORWARDED_ALLOW_IPS
          value: "*"

[vincentsarago]

🥳 thanks @georgr , I'll add a line in the docs pointing to this thread 🙏

[gagecarto]

I had similar issues.. Adding this line to my NGINX server block cleared things up

proxy_set_header X-Forwarded-Proto $scheme;

Found the info here https://stackoverflow.com/questions/34656273/how-to-handle-nginx-reverse-proxy-https-to-http-scheme-redirect