Seeding Tile Caches by Traffic-Weighted Priority

Order the seed queue by observed request counts rather than by tile coordinates, so that the first minutes of a seed run warm the tiles readers actually ask for — which matters both when the seed is racing live traffic after a purge and when it is interrupted before it finishes.

Core Algorithm and Workflow

Demand for map tiles is extremely uneven. Readers look at cities, coastlines and the areas a particular product is about; large parts of any coverage are requested rarely or never. A seed ordered by tile coordinate spends its first hours on whatever happens to sort first, which correlates with nothing.

Building a demand ranking is four steps:

  1. Aggregate the logs. Parse access log lines into z/x/y keys and count. Discard non-tile paths, error responses and obvious abuse.
  2. Decay older observations. Weight yesterday more heavily than last week, so a shift in interest is reflected rather than averaged away.
  3. Promote low zooms unconditionally. A zoom 6 tile costs almost nothing and is requested in nearly every session. It should be seeded before any high-zoom tile regardless of its individual count.
  4. Roll up to metatiles. The renderer produces metatiles, so the queue should be ordered by metatile — otherwise the seed asks for four tiles from one render pass at four different times.
Where the benefit lands when a seed is interrupted Cumulative share of future requests served plotted against seed progress. The coordinate-ordered queue delivers benefit roughly linearly, reaching 20 per cent at 20 per cent progress. The demand-ordered queue reaches 78 per cent of future requests within the first 20 per cent of the work. A marker at 20 per cent progress shows the difference an interruption would make. 0% 50% 100% requests served seed progress 0% 50% 100% demand-ordered coordinate-ordered 20% progress: 78% vs 20% Both curves end at the same place. What differs is everything that happens before they get there.
If a seed always ran to completion undisturbed, the order would not matter. Seeds are interrupted by deploys, quotas and operators often enough that it does.

Production-Ready Python Implementation

import math
import re
from collections import defaultdict

TILE_PATH = re.compile(r"/tiles/(\d{1,2})/(\d+)/(\d+)\.(?:png|pbf|mvt)")


def aggregate_demand(log_lines, day_of, half_life_days: float = 3.0) -> dict:
    """Decayed request counts per tile key from access logs.

    `day_of(line)` returns the age of a line in days. Recent requests are
    weighted more heavily so a shift in interest is reflected rather than
    averaged out over the whole window.
    """
    counts = defaultdict(float)
    for line in log_lines:
        m = TILE_PATH.search(line)
        if not m or " 200 " not in line:
            continue
        z, x, y = (int(g) for g in m.groups())
        weight = 0.5 ** (day_of(line) / half_life_days)
        counts[(z, x, y)] += weight
    return dict(counts)


def seed_queue(candidate_keys, demand: dict, cheap_zoom_max: int = 10,
               metatile: int = 4) -> list:
    """Order tiles for seeding: cheap zooms first, then by decayed demand.

    Keys are rolled up to metatile groups so the seed renders at the same
    granularity the tile server does.
    """
    groups = defaultdict(float)
    members = defaultdict(list)
    for z, x, y in candidate_keys:
        gx, gy = x // metatile, y // metatile
        groups[(z, gx, gy)] += demand.get((z, x, y), 0.0)
        members[(z, gx, gy)].append((z, x, y))

    def rank(group):
        z, _, _ = group
        tier = 0 if z <= cheap_zoom_max else 1     # cheap zooms unconditionally first
        return (tier, z if tier == 0 else 0, -groups[group])

    return [members[g] for g in sorted(groups, key=rank)]

Ranking cheap zooms by zoom rather than by demand inside their tier is deliberate: at zoom 6 the whole coverage is a few thousand tiles, they are requested in nearly every session, and rendering them takes minutes. Sorting them against each other by demand optimises something that does not need optimising.

The decay half-life of three days is a starting point rather than a constant. A product with strong weekly seasonality wants a longer half-life so weekend patterns are not treated as noise; a news-driven map service wants a shorter one.

Performance Tuning and Cartographic Best Practices

  • Checkpoint the queue position. A seed that is interrupted at eighty per cent should resume, not restart. Writing a cursor every few hundred groups costs nothing and turns an interruption from a disaster into a pause.
  • Roll up before ordering, not after. Ordering individual tiles and then grouping them re-sorts the groups arbitrarily and loses the ranking.
  • Exclude keys already fresh. Issue conditional requests and skip anything still valid. On an incremental re-seed this routinely removes most of the queue.
  • Keep the aggregation cheap. A week of logs for a busy service is tens of millions of lines. Aggregate incrementally as logs arrive rather than reprocessing the window each time a seed is planned.
  • Re-rank periodically, not per seed. The demand ranking is a slowly changing artefact. Recompute it daily, store it, and let each seed read the stored ranking — which also makes seed behaviour reproducible for debugging.
How quickly the ranking should follow a shift in interest Weight given to a tile whose requests moved from one region to another six days ago. With a flat average over the window, the old region still outranks the new one. With a three-day half-life the new region has overtaken it. With a one-day half-life the ranking tracks the last day closely and becomes noisy. flat average old region — still ranked first new region half-life 3 days old region new region — correctly promoted half-life 1 day tracks yesterday; one quiet day reshuffles the queue The half-life is the one parameter worth tuning, and it should follow the product's own rhythm: longer for a service with weekly seasonality, shorter for one driven by events.
A flat average is a half-life of infinity, which is why it keeps ranking a region nobody has visited for a week. The failure is not subtle once it is framed that way.

Integration and Next Steps

The ranking this produces feeds the concurrency-bounded seeder described in Tile Seeding and Cache Warming, and the same ordering is what makes a staged recovery after a purge tractable — re-seeding the top of the ranking before reopening traffic is far cheaper than letting readers generate the load. Because the ranking is derived from logs, it also answers a question that comes up separately: which parts of the coverage are worth investing render quality in at all.

One ranking, three consumers A demand ranking feeding three uses. Seed ordering consumes it directly. Purge staging uses it to decide which region to invalidate first so the busiest tiles are re-warmed while traffic is lightest. Quality investment uses it to identify the small fraction of the coverage that most readers actually see. demand ranking recomputed daily seed ordering the first minutes warm what readers ask for purge staging invalidate the busiest region when traffic is lightest quality investment which fraction of the coverage most readers actually see The third use is the one nobody plans for and the one that changes cartographic priorities.
The ranking is cheap to compute and answers questions well beyond seeding — including which parts of a map are worth a cartographer's attention.

Frequently Asked Questions

How much log history is needed to rank tiles usefully?

Roughly a week. Long enough to average out a single unusual day, short enough that the ranking still reflects current interest. Ranking is a much easier problem than prediction: even a noisy week cleanly separates tiles receiving thousands of requests from those receiving none, and that separation is the whole of what the ordering needs. Extending the window to a month adds precision to a comparison that was never close.

What about tiles with no history at all?

They rank last and render on demand, which is the right outcome — a tile nobody has ever requested is the definition of one not worth pre-rendering. The genuine exception is newly published coverage, where there is no history because there was no data. Seed those from a geographic rule for the first few days and let observed demand take over once it exists, rather than treating the absence of logs as evidence of absence of interest.

Should the ranking use unique visitors or raw request counts?

Raw counts. The cache serves requests, not people, and a single automated client requesting one tile repeatedly is a perfectly good reason to have that tile warm. Filter out clear abuse before aggregating — a client requesting the entire pyramid sequentially is scraping, not using the map — but do not deduplicate by client, because that optimises for a metric the cache does not experience.

Does the seed order matter if the whole job will finish anyway?

Yes, in two situations that both occur regularly. A seed following a purge is racing live traffic, and every minute during which the top of the ranking is already warm is a minute of load the render pool does not have to absorb. And seeds get interrupted — by a deploy, a quota, an operator with a different priority — so whatever work completed before the interruption should be the work that mattered most. The ordering is free; both situations are common.

Back to Tile Seeding and Cache Warming