'Missing' tiles at various zoom levels

[kiambogo]

First off, I wanted to say how impressed I am with this project - fantastic stuff! :)

When rendering some road data, I am noticing that there are a significant number of tiles missing (not rendering) at various zoom levels (in particular, lower zoom levels). If I zoom in quite a lot, then I get a better result. I have tested this in Mapbox, OpenLayer, and X-Ray/Inspector all with the same result. The only thing that I found in common with the missing tiles is that they contain >= 1000 features. I noticed that you added a limit of 1000 features when simplify is true - I tried to set this to false, as well as running 0.7.1 but I get the same result. The data is in the correct projection, and I can view it in QGIS from Postgis without issue.

Do you have any idea what could be causing this, and how I could remedy this?

For reference, this is OSM data that was inserted into Postgres with osm2pgsql default schema.

Mapbox:

Inspector:

[pka]

I've experienced these missing tiles myself, but your feedback motivated me to investigate this deeper. I couldn't reproduce it reliably with my setup, but I think I found the reason. It seems to be caused by the connection keep-alive handling of the integerated web server (hyperium/hyper#368). I therefore increased the default number of threads from 4 to 8 and decreased the the keep-alive timeout from 75s to 5s.
Are you able to build the current master branch for testing, or shall I make a binary available?

[kiambogo]

Thanks for the response!

Running the server on master yields huge improvements!
OpenLayers:

This is much better :) I don't want to pivot this issue too much, but Mapbox is giving some very strange results:
Mapbox:

(ignore the blue line, I added that manually). As you can see, there are a lot of line artifacts displayed as well. These don't exist in OpenLayers. I know that #18 seems pretty similar, in which you point out invalid geometries. Given that OpenLayers seems to be perfect, and that this is vanilla OSM data, would you have any ideas as to what is causing that?

Thanks for the help!

[pka]

I've imported a little osm country file from geofabrik with osm2psql (0.82) but couldn't reproduce these artefacts. Could you point me to your data - e.g. a metro extract?
Another difference could be the PostGIS version in use (SELECT postgis_full_version()). I'm testing with PostGIS 2.2.2.

[kiambogo]

Data is sourced from http://download.geofabrik.de/north-america/canada/ontario.html . I downloaded the PBF, used Osmosis to parse out a subsection with the bounding box tool

osmosis --read-pbf ./ontario-latest.osm.pbf --bounding-box top=43.753963 bottom=43.561912 left=-79.632868 right=-79.194903 --write-xml toronto.osm

then loaded the data into Postgres (fresh Postgres 9.6 with Postgis 2.3.2)

osm2pgsql -s -C 1024 --proj 3857 -d osm2 -j ~/Downloads/toronto.osm.pbf -H localhost

I'm wondering if Postgis 2.3.2 vs 2.2.2 is causing a difference, although I'd be surprised if that was the issue.

[kiambogo]

I'm not exactly sure what I did, but I am no longer getting the artifacts. I believe that the issue was around projections, and converting between them though. I can try to pinpoint what the issue is if you would like.

Unfortunately, I'm experiencing the missing/incomplete tile issue which I opened this ticket for initially :( I am running off of master, with 8 threads and a 5sec keep-alive timeout and am still seeing significant loss. I've tried tweaking these values, but to no avail.

Do you have any thoughts/recommendations on how to resolve this?

[kiambogo]

It looks like Hyper 0.11.0 was just released today, and includes asynchronous IO. I believe (from reading other issues similar to this) that this may help solve this problem.

Any chance on getting t-rex updated to use Hyper 0.11.0? :) I'd love to help with a PR, but my Rust is lacking.

[pka]

Missing tiles:
Will turn off keep-alive for the next release and migrate to Hyper 0.11.0 as soon as Nickel migrates.

[pka]

Artifacts in MB viewer:
In the meantime I saw the artifacts twice with my own datasets, but couldn't look deeper into it. At least in one case, clipping was apparently turned off, so big geometries may have caused the problem. I saw a message in the browser console saying this. Could you have an eye on the console in the browser next time you see this?

[ftheriault]

Hi guys,

I also have the same issue, rendering with OL. I used osmosis and osm2pgsql to import the dataset. The console displays "references" warning from ol.js, but no hard error. Yesterday, I tried with a manual build of t-rex, but the issue is still there.

I am quite new with all this, but if I can be of any help, let me know.

[pka]

Hi Frederic,
The keep-alive fix has been been commited this morning (European time). Can you try again with 9deaf62 or newer?
MB artifact fix is following in a few minutes.
Pirmin

[kiambogo]

Hi @pka . I tested with the latest patch, with keep-alive disabled, and confirmed that it was disabled in the logging. I still am experiencing the same issue though.

I've noticed that it's the tiles with the most densely populated geometries, and at a high zoom level that this occurs. Furthermore, the data seems to be fine in the bottom left corner, and then is blank at the top right hand corner of the tile. Could there be some memory allocation issue here? Perhaps the vector or other data structure that the data is being pushed to is filling up and maxing out? Just a thought.

[pka]

For fixing MB viewer artifacts see #18 (comment)
Don't forget to clear the browser cache before testing again!

[pka]

@kiambogo ok, problem number 3 in this issue... Looks like you are hit by the feature limitation (1000 per tile). You can increase or comment query_limit in the config file or use the serve command with --simplify=false which also turns off feature limiting. Don't forget to clear the browser cache before testing again!

[ftheriault]

Hi @pka ,

Unfortunately for me, it did not help, even with the --simplify=false. I am starting to think that perhaps it comes from the importation, if you do not have the issue on your end.

[kiambogo]

@pka That was my suspicion, but I have tried setting the query_limit to 10000 and I get the same results. I verified that there is no issue with the config by removing the LIMIT from the SQL in postgis.rs as well.

I have disabled browser cache.

[pka]

I'm now also testing an osm2pgsql import of Toronto and get similar effects mostly on levels < 12. The import looks complete (tested with QGIS). Using the inspector viewer I see that most tiles with gaps have often nearly 5000 features, even without query_limit. No explanation for that - will take a sleep first...

[kiambogo]

I was able to confirm that the issue is due to the limit in the SQL. I exported the SQL statements run against the DB, created them into a view in the db and ran it with QGIS to check if there were issues prior to encoding. It turns out that, with the map data I have been using, there is not much difference between using a limit of 1000 vs a limit of 10,000. When I adjusted the limit to 1mil I was able to get a complete map rendering; similarly, excluding query_limit yields a complete map.

[ftheriault]

It is indeed because of the query_limit, though now even though I have a decent computer, t-rex takes most of the CPU and has problem serving the maps. Thanks @kiambogo for pointing to the error.

[pka]

2f0c053 is now emitting an info message when features are limited by query_limit. Thanks a lot for your input!