Edge cache, admission and warming

Every request to an app’s hostname passes the edge. Three things there keep the realm fast and steady under load: a cache in front of the apps, admission control that sheds load fairly when the edge is past its limit, and warming that keeps a cold copy of an app away from visitors until it is ready.

The edge cache

Caching is a stage of the edge middleware pipeline, the Edge Response Cache. The realm’s pipeline sets the defaults; an organisation or an app can override them. In the stage you add cache buckets: a Public cache, or a Private cache (per user), each with an optional path filter (/static/*, *.js) and a Default edge TTL. Edge rules and edge scripts can set their own times.

What the edge keeps:

• answers to GET and HEAD with status 200 and a time to keep;
• never an answer with Set-Cookie, Cache-Control: no-store, or private in a public cache;
• never an answer whose Vary names something other than Accept-Encoding, so no client is handed another’s variant.

How it behaves:

• One fetch for a cold page. When many requests ask for a page the cache doesn’t have, one goes to the app and the others wait for its answer. A rush on a new page reaches the app once. If that one fetch fails or the answer may not be kept, the others ask for themselves.
• Stale answers, if the site asks for them: Serve stale while refreshing answers from the old copy for so many seconds after it expired, while one request fetches a new one; Serve stale on app errors answers from the old copy when the app fails or answers 5xx. The app’s own stale-while-revalidate and stale-if-error come first, and its must-revalidate or no-cache turn them off for that answer.
• Purge. Any change to a site, its rules or its stages empties that site’s cache.

The cache lives in the edge’s memory, shared by all sites: 32 MiB by default, and no single answer over 2 MiB is kept (larger ones stream through). The least recently used answers go first. These are edge settings in deploy.json (settings.edge.ingress.cacheBytes, cacheObjectBytes); raise the budget on an edge with memory to spare.

Admission control

When the edge is busy beyond what it can serve well, it answers some requests at once with 503 and Retry-After, instead of answering every request late. It decides right after reading a request’s head, before any other work, from two signals: the number of requests in flight, and how far behind its own scheduling is.

The order is fair:

1. Below the limit, every request passes.
2. Past it, the busiest sites are shed first: a site holding more than its share of the requests in flight, and any name no site has.
3. Further past it, every customer site is shed.
4. The realm’s own traffic (the desktop, the API, sign-in) is shed only last, so you can still reach the Manager while your sites are under a flood.

The thresholds are edge settings (settings.edge.admission): kneeInFlight (4,096 requests in flight), siteKneeInFlight (1,024 for one site), tenantInFlight (8,192), realmInFlight (16,384), and the lag limits lagKneeMs (10 ms) and lagLimitMs (50 ms). The defaults suit an edge with a worker per CPU core; tune them per deployment.

Request limits per site and per address

Two token buckets limit how fast requests may come:

An IPv6 client counts as its /64. Each bucket saves up two seconds of its rate for bursts. Both are policies in Policies: set them for the realm, or for one organisation or project. Below the realm a value can only be lower, never higher, than the one above it.

Slow clients

A client that sends or reads too slowly loses its connection rather than holding resources: a request head must arrive within 30 seconds, a request body may pause for 60 seconds, and a client that reads nothing for 60 seconds while the edge has bytes for it is cut. Each is an idle limit, never a limit on a whole transfer, so a large upload or download that keeps moving is never cut. The settings are clientHeadTimeoutMs, clientBodyIdleMs and clientWriteStallMs; 0 turns one off.

Warming new copies

An app can list warm paths: pages the edge asks a new copy for, with each of the app’s hostnames, after its health check passes and before it gets any visitor. A framework that renders a page on first request then has it ready before the first real visitor arrives.

• Set them in the app’s Hostnames tab, under Warm paths, or in the app’s velrix.json as deploy.warmPaths (up to 32 paths, each starting with /). When velrix.json sets them, the tab shows them read-only.
• A copy is warmed once, when it first comes up or comes back after failing. A copy that is up is not warmed again.
• Warming is part of being healthy. A warm path that answers anything but 2xx or 3xx, or warming that takes longer than 30 seconds in all, keeps the copy out of service. In a deploy, the new revision then fails its health check and is rolled back, so a broken page never reaches a visitor.
• Warming fills the app’s own caches. The edge cache fills on the first real request.

Next

• Public address and hostnames
• Apps
• Policies