We generate all the vector tiles for zoom levels 0 to 15+ and regenerate affected tiles when the underlying OSM data changes. For higher zoom levels (we plan to stop at z18 at this point), serve pieces of lower zoom tiles. This table bellow summarizes our tile storage needs for one style. For the first iteration, we plan to store up to level 15. In order to greatly reduce the storage requirements, we will use ‘solid detection’ – a check to see if the tile consists entirely of one type of polygon (e.g. water or forest), without any other features. If so, we simply do not store the tile, and during rendering, we use over-zoom. Over-zooming is when we know that a tile does not contain more information than the corresponding piece of a tile in the lower zoom level, so instead of storing a tile, we simply extract a part of the tile above when needed. Some tiles could get marked as dirty during the OSM import, or as part of the data layer SQL adjustments. Tilerator has a job queue to process such tiles. Additionally, server could regenerate dirty tiles on the fly if user requests them. This is especially relevant to OSM itself, where the result of editing should be immediately visible, but could benefit WMF as we increase the OSM pull frequency. As a side result to tile iteration problem, we came up with a simple way to index any tiles regardless of the zoom level. We use a single 64bit integer to represent both X and Y coordinates together, and iterate it from 0 to 4Z. For each value, extract X value by combining every odd bit and Y value from the even bits: So if we look at this iteration sequence, the value 35 would be: This is the simplest approach – each tile is stored as an individual file. We have been told of an existing implementation that stores tiles as files up to level 14 (max 358 million), by increasing inodes count. By default, a 800GB partition has 51 million inodes. We are worried about the overhead here, especially since we will almost certainly need to store tiles for more than one data source later. implements this approach There is an existing implementation called mbtiles, that stores multiple vector tiles together in one sqlite file. From what we heard, even Mapbox itself sees this as a dead-end approach for server-side tile storage, even though it is used heavily by Mapbox studio for storage and uploading. We might however use some ideas from it or even fork’n’rewrite it to our needs. This snippet generates a two column list, where the second column is the ‘group size’ – how many identical tiles were found. E.g. 1 means that this tile is unique, 2 means there exists exactly one other tile with the same content, etc. The first column is how many groups with that size were found. E.g. when groupsize=1, this is the number of unique tiles, for groupsize=2, it is the number of pairs found. The total number of stored tiles is the sum of these two columns multiplied. The task of tile generation might take considerable time – depending on the zoom level (number of tiles), SQL server speed, and the number of servers doing the rendering. In order to manage this process, we plan to use a priority-queue, with persistent storage to mitigate crashes. Source.