← Blog

Billing every API request exactly once, without a database on the request path

Theo Mathieu ·

The first post was about the geocoding engine. This one is about the smaller service in front of it: the edge that checks API keys, applies rate limits, spends credits and counts usage for billing. It is written in Rust like the engine, runs on Kubernetes as several identical replicas, and one rule shaped all of it: nothing on the request path waits for a database.

The edge lives in a private repository, so I’ll describe it rather than link to it. The one code sample below is a simplified version of the real query.

Why not write to the database on every request

The obvious design is to look up the key and write a usage row in Postgres for every request. It’s simple and exact, and it’s what I considered first. The problem is throughput. In the engine’s benchmark, a single geocoder pod served 1,700+ searches a second (methodology). If every one of those requests also writes to Postgres, the database has to sustain the same rate, for all customers at once, and keep sustaining it as the geocoder scales out. The engine would only be as fast as the billing database behind it.

Latency points the same way. The geocoder’s average processing time is 5 ms (how it’s measured). A database round trip on every call would add its own latency on top, and a slow or unavailable database would become a slow or unavailable API.

So keys, limits and balances live in memory, and Postgres is updated in the background.

Kubernetes first

Everything runs on Kubernetes, and I designed it that way from the start rather than moving it there later. Each part is a Deployment with several identical replicas spread across nodes, and none of them assumes it is the only copy:

  • The geocoder is stateless and read-only. Each pod pulls a complete, immutable index from an S3-compatible store in an init container and serves it from a read-only volume. A small leader, one of two pods sharing a Kubernetes Lease, starts a build Job when the index named by the chart’s settings isn’t in the store yet, and rolls out the serving pods once it is. New pods load their index before old ones stop, so a rebuild never interrupts serving (engine Helm chart).
  • The edge, the subject of this post, runs three replicas by default. Each has its own in-memory snapshot, its own counts and its own random ID. The replicas never talk to each other; they only meet in Postgres, every few seconds.
  • The web app (marketing site, dashboard and signup) also runs three replicas.

In front of the edge, an L7 ingress balances each HTTP request and gRPC call across pods, not each connection. Rollouts start a new pod before removing an old one, and disruption budgets keep node drains to one pod at a time.

Most of what follows is about how the edge replicas agree on keys, limits and money without asking each other.

What a request does

For each request, the edge:

  1. hashes the API key with SHA-256 and looks it up in an in-memory snapshot of active keys;
  2. takes a token from the account’s per-second bucket;
  3. for billable operations, checks the account’s remaining credits;
  4. calls the geocoder;
  5. counts the request if it succeeded.

“Succeeded” is narrow on purpose. Only a JSON 2xx reply or a gRPC reply whose final grpc-status is 0 is billed. Over gRPC, an HTTP 200 can still end with an error status in the trailers, so the edge buffers unary gRPC replies through their trailers before it counts anything. Backend failures return a generic error and aren’t billed. Autocomplete and region lookups need a key and share the rate limit, but they don’t spend credits.

The remaining credits are worked out locally: the balance from the last Postgres load, minus the requests this replica has counted that the balance doesn’t include yet. When that reaches zero, billable calls get a 429. There is no metered overage.

Each replica subtracts only its own counts. Requests served by the other replicas reach its balance through Postgres, after their next sync and its next reload. That gap of a few seconds is where the small overshoot described at the end comes from.

The 5-second sync

Every 5 seconds, each edge replica runs one sync. It takes its pending counts as a batch, gives the batch the next sequence number and writes it in a single transaction:

BEGIN;
-- Adds nothing if this batch was already written.
INSERT INTO edge_flush (edge_id, seq)
VALUES ($1, $2) ON CONFLICT DO NOTHING;

-- Only when that insert added a row:
INSERT INTO usage (key_id, day, count)
SELECT * FROM UNNEST($3::uuid[], $4::date[], $5::bigint[])
ON CONFLICT (key_id, day)
DO UPDATE SET count = usage.count + EXCLUDED.count;

-- Lock the accounts' credit grants, then spend the batch
-- from the grant that expires soonest.
SELECT 1 FROM credit_grant
WHERE user_id = ANY($6) ORDER BY id FOR UPDATE;
UPDATE credit_grant SET used = used + ...;
COMMIT;

The edge_flush row is what makes it exactly once. edge_id is a random ID each edge process picks when it starts, and seq numbers its batches, so replicas flush independently and their batches never collide. If the transaction fails, or commits but the reply never arrives, the batch stays in memory with the same sequence number and is sent again on the next sync. A retry of a batch that already committed hits the existing row and adds nothing; a retry of one that didn’t commit writes it for the first time. Either way, it lands once. A test against a real Postgres database flushes the same batch twice and checks the count. The row is written even when the batch is empty, so it also serves as the replica’s heartbeat.

Credits are spent in the same transaction. Each paid period and each free month is a separate grant with its own expiry, and a batch is spent from the grant that expires soonest. Locking the account’s grant rows first means two replicas flushing at the same moment take turns instead of spending the same credits twice.

After the commit, the sync reloads active keys, each account’s per-second limit and balance, and the number of edge replicas that flushed in the last 15 seconds. A batch stays counted against the local balance until a reload that includes it succeeds, so there is no moment where requests are neither in Postgres nor in memory.

For Postgres, this means one write transaction per replica every 5 seconds, with rows per key and day rather than per request.

Rate limits without shared state

Rate limits are the part I simplified most. A strict per-account limit across several replicas needs a shared counter, usually Redis, consulted on every request. Instead, each replica takes its share. With N replicas that sent a heartbeat in the last 15 seconds, an account’s bucket on each replica refills at limit / N tokens a second and holds at most the larger of limit / N and 2 tokens.

As an example, take the free plan’s 2 requests a second across 5 replicas. Each replica refills at 0.4 a second and holds 2 tokens. Starting from full buckets, a client can make up to 10 requests, and then the refill settles at 2 a second overall. The two-token minimum absorbs short bursts when the load balancer happens to send a few requests to the same replica. Retry-After is rounded up to the next local token, so an empty bucket in that example answers with 3 seconds rather than a fixed 1.

Scaling needs no configuration. A new replica’s first sync is also its first heartbeat, so the others count it at their next reload, within 5 seconds, and lower their share. When a replica stops, its last heartbeat ages out after 15 seconds and the others raise their share at their next reload. While that happens, the shares are briefly off, which is one of the inaccuracies listed below.

When Postgres is down

If Postgres is unreachable, the edge keeps serving from its last snapshot and keeps counting in memory, and the counts are written once the database is back. I chose to use a snapshot of any age rather than turn away paying customers. Signup, dashboard changes and the website demo do stop, because they need the database. The edge’s liveness check doesn’t depend on Postgres, so an outage never makes Kubernetes restart pods that are holding cached keys and unflushed usage.

A new edge pod can’t load keys without Postgres, so it stays not ready while the existing pods carry the traffic. A rollout during an outage stalls instead of replacing working pods with ones that can’t serve.

Shutting down without losing usage

Rollouts are the other moment where counts could go missing. Kubernetes replaces edge pods one at a time, starting the new one before the old one stops. On SIGTERM, the edge drains in-flight requests for up to 30 seconds, stops its background sync, then freezes billing and flushes everything still pending, retrying for up to 15 seconds. A request that finishes after billing is frozen gets an “unavailable” error instead of a successful reply that would never be billed. If the final flush still fails, the edge logs the lost usage and exits with a nonzero status. These budgets, a 10-second preStop delay and the error reporter’s 2-second close timeout all fit in Kubernetes’ 60-second grace period.

What this costs

The design accepts a few known inaccuracies. I’d rather list them here than have someone find them later:

  • A replica that crashes loses up to one sync interval, 5 seconds, of counts. Those requests are never billed.
  • A balance can be overspent by roughly one sync of traffic plus bucket bursts. At the Growth plan’s 25 requests a second, that is about 125 requests plus bursts.
  • Rate limits are approximate. Burst capacity grows with the number of replicas, restarts and changes in the replica count can briefly allow extra bursts or throttle early, and uneven routing can return a 429 below the account’s nominal rate.
  • A persistent connection pinned to one pod only gets that pod’s share of the limit, so the load balancer has to balance each HTTP request and gRPC call, not each connection.
  • Successful replies are charged when they are prepared, before delivery. A client that disconnects while the reply is on its way is still billed for the work.

Most of these fall in the customer’s favour. Lost counts are never billed, and an overspent balance means a few free requests rather than an unexpected invoice, because there is no overage to bill.

The improvement I’d like to make later is sticky routing per account. If the ingress sent every request from one account to the same edge replica, that replica would hold the account’s whole rate limit and see all of its unflushed usage. Limits and balance checks would then be exact, without Redis and without a network call on the request path. It isn’t built yet, and it brings its own trade-offs. The ingress only sees the API key, not the account, so an account with several keys needs a way to land on the same replica. A very busy account would load a single replica. And when replicas are added or removed, some accounts move to another replica and start there with a fresh bucket.

Shared counters in Redis would also make the limits exact, but at the cost of a round trip on every request, and I’d rather not pay that. Keeping Postgres in the background also keeps another option open: because the edge only talks to Postgres in the background, a second region with its own edge and geocoder could use a distant database without slowing anyone down.

Try it

The free plan includes 50,000 requests a month at 2 requests a second, with no card. Create a free account, make a few calls, and you’ll see them in the dashboard’s usage, written by the sync described above.