Allow developers to clear all tile caches

[averas]

There is (as far as I know) no way of directly modifying or clearing the tile cache. In cases where the client is made aware that the server side vector tiles have been updated, is the supported way of clearing the tile cache simply removing and adding the tile source? Should there be a discrete API method to clear the tile cache?

[lucaswoj]

related to #1947

[lucaswoj]

Per #3715, it may be useful to allow a developer to clear a specific tile rather than all tiles.

[averas]

With the new smart setStyle() I provoke a tile cache flush on vector tile sources with this ugly workaround:

...
style.sources['mySource'].dirty = Math.random();
map.setStyle(style);
...

... which does work. However, after a while these messages start to show up on the console:

mapbox-gl.js:103 Uncaught TypeError: Cannot read property 'state' of undefined
at t.reloadTile (mapbox-gl.js:103)
at mapbox-gl.js:103

... which probably is because this workaround is incompatible with #3944. Works fine with 0.31.0.

[karlguillotte]

I am the same error. I am not doing that magic to put source dirty. I am using the new smart setStyle() out of the box.

[jaapster]

I get the same error ... without calling setStyle

[anandthakker]

@karlguillotte and @jaapster -- when you see this error in the console, what symptoms do you see in the rendered map? Could you by any chance provide a minimal example that reproduces the error?

[jaapster]

@anandthakker I don't see any symptoms in the rendered map, just the error messages. I'll try and reproduce.

[karlguillotte]

I actually never saw that error myself and I can not reproduce it. We are using Sentry as error reporting tool and that error has been reported many times. I am sorry, it can be hard for you, I tried to reproduce, but I can not find a way to reproduce.

[musicformellons]

I have a datasource consisting of vector tile points. When I currently add a new point on my client, the server updates the relevant tiles. Then it shows up in my client at all layers (when zooming in & out after adding the point) except on the layer on which the point was added as that tile did not refresh yet...

So I could use this feature! An ETA maybe? Any suggestions how to deal with my issue in the meantime?

[musicformellons]

@averas Did you ever find a working solution for this?

[jignesh-techvasu]

This is big use case with us as well. We are creating vector tiles and allowing user to modify the shapes. Flow is like

1. User update geometry
2. App updates tiles based on posted geometry
3. App is reloading layer

Here , there is no explicit way to tell mapbox that we don't want to use cached layer after update! It will keep on loading old tiles even if we want it to reload layer !

[joedjc]

@jfirebaugh Ok great thanks - I hope this issue can be pushed forward before 0.43.0 is released it would prevent me and i'm sure others from upgrading at the moment.

If it could work like this as well that would be a bonus:

Preferably, it would wait until the new tiles have been received before removing the old ones so there is no 'flicker' (where nothing is visible between removing the old tiles and adding the new ones)?

[musicformellons]

Can't upgrade before this is working.

[musicformellons]

Uh, any progress on this maybe?

[musicformellons]

Mmmh, I just tried and upgraded from version 0.42.2 to 0.49.0 and without changing any code all layer tiles seem to refresh... Quite a surprise there...?!

[musicformellons]

I don't know what feature takes care of this, but the issue I described on 12 sept 2017 is solved. Also the precision of the point at different layers is improved somehow. Anyway, I am really happy I upgraded.

[ChrisLoer]

@musicformellons glad to hear this! Although I'm also not sure what change you're seeing. 😅

Just to recap, the current state of this ticket is that we recognize the value of an explicit tile cache invalidation API for cases where you have some external way of knowing that some or all tiles are outdated. The current mechanism of removing and re-adding the source should work but is a cumbersome way to accomplish a re-load. The cache is supposed to respect HTTP max-age headers.

[arturgaleno]

I have a use case very similar to what @jignesh-techvasu described, and as @ChrisLoer said if the cache respect HTTP Cache-Control headers it would be great. Any new about this issue guys?

[asheemmamoowala]

if the cache respect HTTP Cache-Control headers it would be great.

As of v1.1.1 of the library there are some changes with relation to tiles being cached. Mapbox-hosted tiles are cached using the browser's Cache interface. This caching honors the http Cache-Control headers, but is separate from the browsers default cache. Since these resources are in a cache controlled programmatically by gl-js, they can be cleared if needed, using the Map#clearStorage API. To re-iterate - this will only affect mapbox-hosted tiles.

For self hosted tiles, or other non-mapbox hosted tiles, the browser cache will be used and GL-JS will respect that Cache-Control & Expires HTTP headers. There is no way for the library to clear that browser cache. If the tiles were served with a long TTL but need to be cleared, the best approach IMO is to use a cache-busting query parameter in the tile url, for example:
http://mytiles.server.com/{z}/{x}/{y}?dt=${Date.now()}

[bergkvist]

@asheemmamoowala Cache-Control/Expires headers are certainly a nice feature, but what if you want any changes that happen to be reflected within a few seconds?

Setting Cache-Control to a few seconds would be a way of doing this, but this kind of constant polling/reloading would use a lot of unnecessary bandwidth.

It would be really useful if there was some way to manually expire the tiles from within the client. Rather than reloading all of the tile data every few seconds, the client could simply wait for a signal to reload over a WebSocket connection or something similar.

[asheemmamoowala]

what if you want any changes that happen to be reflected within a few seconds
It would be really useful if there was some way to manually expire the tiles from within the client.

If the headers on the tile are longer than the TTL for the data, then the browser will cache those URLs and respond to subsequent tile requests with the already cached data. This is part of the browser's default caching behavior.
As I mentioned before:

the best approach IMO is to use a cache-busting query parameter in the tile url, for example:
http://mytiles.server.com/{z}/{x}/{y}?dt=${Date.now()}

This can be done using Map#transformRequest or by swapping the tiles urls on the source definition.

[bergkvist]

@asheemmamoowala

I'm familiar with/and have used the cache-busting query earlier based on your suggestion. (Currently only when I refresh/reload the website, in which case it works great)

Map#transformRequest

To me it seems like Map#transformRequest is a callback that is only called when a fetch is actually happening (allowing me to add cookies and whatnot). This would mean that transforming the url into a cache-busting url would not do me any good, since the cache would already have to be expired for this callback to be called. (Or the website would have to be refreshed) Please correct me if I'm wrong.

Tiles urls

I can't seem to find a function for changing the tile urls. Changing them manually does not cause mapbox to notice and reload the data.

(Removing and re-adding a source also does not seem to be possible when layers depend on it.)

Let's assume I have the following structure:

const map = new mapboxgl.map({
  container: 'map',
  center: [-122.420679, 37.772537],
  zoom: 13,
  style: {
    version: '8'
    sources: {
      'open-street-map': { 
        type: 'raster',
        tiles: [ 'http://a.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}@2x.png' ]
      },
      // vector tile source which might change/need to be refreshed at a moments notice.
      'electrical-grid': {
        type: 'vector'
        tiles: [ `http://some.url/{z}/{x}/{y}.pbf?dt=${Date.now()}` ]
      }
    },
    layers: [{
      id: 'open-street-map',
      type: 'raster',
      source: 'open-street-map',
      minzoom: 0,
      maxzoom: 22
    }, {
      id: 'electrical-grid-lines',
      source: 'electrical-grid',
      type: 'line',
      'source-layer': 'lines',
      paint: { /* ... */ }
    }, {
      id: 'electrical-grid-circles',
      source: 'electrical-grid',
      type: 'circle',
      'source-layer': 'circles',
      paint: { /* ... */ }
    }]
  }
})

// bust electrical grid cache, forcing a reload. might be neccesary in response to new data/measurements.
function reloadElectricalGrid() {
  // This does not work, because the library does not detect/react to the change. No new request is sent
  map.getSource('electrical-grid').tiles[0] = `http://some.url/{z}/{x}/{y}.pbf?dt=${Date.now()}`
}

[bergkvist]

I found a solution by looking through the source code!

Essentially:

/* my tile source id is "electrical-grid" */
function reloadElectricalGrid(map) {
  // Set the tile url to a cache-busting url (Thanks @asheemmamoowala):
  map.getSource('electrical-grid').tiles = [ `http://some.url/{z}/{x}/{y}.pbf?dt=${Date.now()}` ]
  
  // Clear the tile cache for a particular source
  map.style.sourceCaches['electrical-grid'].clearTiles()

  // Load the new tiles for the current viewport (map.transform -> viewport)
  map.style.sourceCaches['electrical-grid'].update(map.transform)

  // Force a repaint, so that the map will be repainted without movements
  map.triggerRepaint()
}

NOTE

This causes the layer to visually flash/dissappear for less than a second. This is a clear indication it is working, but this could be a bit annoying if these reloads happen often. This happens because the cache is empty while the new tiles are being fetched. (update is asynchronous)

A solution that avoids this "flicker" might be creating a new source, letting it load the new tiles in the background, before somehow switching.

[zhenyanghua]

I found a solution by looking through the source code!

Essentially:

/* my tile source id is "electrical-grid" */
function reloadElectricalGrid(map) {
  // Set the tile url to a cache-busting url (Thanks @asheemmamoowala):
  map.getSource('electrical-grid').tiles = [ `http://some.url/{z}/{x}/{y}.pbf?dt=${Date.now()}` ]
  
  // Clear the tile cache for a particular source
  map.style.sourceCaches['electrical-grid'].clearTiles()

  // Load the new tiles for the current viewport (map.transform -> viewport)
  map.style.sourceCaches['electrical-grid'].update(map.transform)

  // Force a repaint, so that the map will be repainted without movements
  map.triggerRepaint()
}

NOTE

This causes the layer to visually flash/dissappear for less than a second. This is a clear indication it is working, but this could be a bit annoying if these reloads happen often. This happens because the cache is empty while the new tiles are being fetched. (update is asynchronous)

A solution that avoids this "flicker" might be creating a new source, letting it load the new tiles in the background, before somehow switching.

@bergkvist, Do you think you solution has any difference than remove the source and remove the layer, then add the source and layer back for a certain layer? These two approach seems both present a short gap while the new source is loading. Have you managed to have a seamless source swapping solution yet?

[everhardt]

I'm using the code above by @bergkvist, but noticed that if the tab is in the background, the sourceLayer is not available until the tab is in the foreground again (at least in the browser I'm using, Chrome). In the mean time, if layers that use this sourceLayer are updated (for example added), mapbox throws errors like

mapbox-gl.js?e192:33 Uncaught TypeError: Cannot read property 'size' of undefined
    at $i (mapbox-gl.js?e192:33)
    at Object.symbol (mapbox-gl.js?e192:33)
    at co.renderLayer (mapbox-gl.js?e192:33)
    at co.render (mapbox-gl.js?e192:33)
    at r._render (mapbox-gl.js?e192:33)
    at eval (mapbox-gl.js?e192:33)

and

mapbox-gl.js?e192:29 Uncaught TypeError: Cannot read property 'get' of undefined
    at Aa (mapbox-gl.js?e192:29)
    at e.queryRadius (mapbox-gl.js?e192:29)
    at tl.loadVectorData (mapbox-gl.js?e192:29)
    at i.a (mapbox-gl.js?e192:33)
    at Eu.process (mapbox-gl.js?e192:29)
    at MessagePort.Bu.MessageChannel._channel.port2.onmessage (mapbox-gl.js?e192:29)

(similar to #8233 and #742)

[bergkvist]

@zhenyanghua I think both approaches should be quite similar. I haven't tried removing the "flash", but what I would look for: Some way of creating an invisible layer, waiting until all the tiles in the viewport have finished loading - and removing the previous layer whilst simultaneously making the new version of the layer visible.

@everhardt My guess is that this has something to do with "map.transform" being null or missing some internals when the browser tab is not active. You could try console.log(map.transform) to see if it changes when the browser tab is hidden.

[everhardt]

@bergkvist There's no change. I've noticed that these two lines are enough to trigger the errors:

map.style.sourceCaches['electrical-grid'].clearTiles()
map.style.sourceCaches['electrical-grid'].update(map.transform)

If I look at https://developers.google.com/web/updates/2017/03/background_tabs, I see that requestAnimationFrame is paused (or ignored?) when the tab is in the background. Might that have to do with it?

[rogeraustin]

I am using the following code to refresh all tiles when I receive a notification (via a websocket) that the tiles have changed on the server. The refresh is perfectly smooth with no flicker and I haven't seen any problems with refreshing while in the background. The server sets max-age=86400 on the tiles so the code needs to both bust the browser cache (using Cache-Control:max-age=0 in the request) and also set the tile expiration time in the sourceCache. It would be easy to extend this to refresh only a particular set of tiles. Note that using Cache-Control to bust the cache is better than changing the url because it forces the browser to revalidate its cached resource rather than requesting a completely new resource, allowing the server to return 304 if a particular tile has not actually changed (which is usually the case in my app).
In response to a notification:

      const sourceCache = map.style.sourceCaches['my-source'];
      for (const id in sourceCache._tiles) {
        sourceCache._tiles[id].expirationTime = Date.now() - 1;
        sourceCache._reloadTile(id, 'reloading');
      }
      sourceCache._cache.reset();
      map.triggerRepaint();

And my transformRequest

transformRequest: (url, resourceType) => {
        if (resourceType === 'Tile') {
          return {
            url: url,
            headers: { 'Cache-Control': 'max-age=0' }
          };
        }
      }

It would be nice if something like this could be wrapped up into a public refreshTiles method on the source, so that I don't have to worry about implementation details changing in future, but it is only half a dozen lines of code so should be no big deal to rework.

[nmahser]

Hey @rogeraustin, I'm trying to avoid caching map tiles into "mapbox-tiles" cache storage. I tried adding `headers: { 'Cache-Control: 'max-age=0'. However, I get Cors error. I was wondering if there was a way to disable mapbox caching?

[douglasg14b]

resetting the cache and triggering a repaint doesn't fetch new tiles unfortunately... The view will not change. till the user zooms out or in, but the current zoom level still stays broken regardless. So, the cache reset posted above seems largely ineffective.

Also map.style.sourceCaches['my-source']; should now be `map.style._sourceCaches['other:my-source'];

[douglasg14b]

After some more testing, this works intermittently. One action will cause a tile re-fetch, then the next won't, and the next will, then the next won't.

Even using source.setSourceProperty() which apparently is supposed to do something here has the same behavior. ed27d23