← Blog

A coding agent is only as good as its harness

Theo Mathieu ·

A coding agent can write a fix for a geocoding bug in minutes. The hard part is knowing whether the fix is right, and whether it quietly broke something in another country. That is a question for the harness the agent works in, not for the agent. This post describes the harness around the OpenGeocoding engine: what checks every change, why the world index had to rebuild quickly for any of it to work, and the part that still isn’t stable, which is the speed gate.

The engine’s public repository merged 281 pull requests between 12 and 27 October 2026, and many of them were written by a coding agent. At that pace, I can’t judge every change by reading it. I need the checks to tell me what the change did to the whole world.

Why a geocoder needs this more than most code

Address matching is a long list of rules that interact. The engine has rules for Japanese block addresses, Korean road names, Dutch postcodes, Czech house numbers and many others (the first post has a few examples). A rule that helps one country can change the answer for an address somewhere else, and nothing in the code tells you where. Unit tests catch the cases you thought of. What I needed was a check that runs every case I have, in every country, on every change.

A world index that rebuilds in minutes

That check is only meaningful on a full world index. A synthetic test world is useful for logic, but the regressions that matter come from real data: a street in Australia that happens to share its name with a town, a Japanese address whose block number sits in a different field than expected.

So the gate needs the real index, and the base and head of a pull request have to be compared on the same one. Changes to the index format are frequent: 28 commits between 13 and 27 October set a new builder hash, and each of them names a new world index. With an import that takes days, as a Nominatim planet import does, testing each of those changes against the world wouldn’t be practical.

The engine’s builder produces the world index in about 7 min on one machine (build record). That number is the foundation of everything below. When a change alters the index, the agent’s instructions say to rebuild it locally and run the comparison there before pushing, rather than wait for CI. On the cluster that runs the CI gate, a first build for a new builder is slower and can take hours, so the local build is what keeps the loop short.

Expected answers from registries, never from a geocoder

The gold suite takes its expected answers from official and public reference sources: national address registries, Wikidata, the US Census geocoder and others. They are never taken from Nominatim, another geocoder or this engine’s own output. The cases are generated by a command, evaluate build-gold, from a list of seeds, and the generated file is never edited by hand. The suite also includes a panel of addresses drawn from OpenStreetMap; since the index reads much of the same data, that panel measures agreement rather than accuracy, and it catches regressions all the same. Today the whole suite has 36,920 cases (evaluation guide).

The rules in the engine’s AGENTS.md matter here as much as the data. An agent with a failing check has an easy way out: loosen the expectation. The instructions close that door explicitly. Expectations are never weakened, cases are never dropped, and a failure is never relabelled to make a gate pass. Known failures carry a known_failure label with their cause, stay visible in every report, and the label comes off only after a fix is verified. About one location in five is held out by a stable hash and kept out of tuning.

Base against head, on every pull request

Every pull request that touches the service, its data or the corpus gets a staging image, and its merge base already has one from main. A private cluster evaluates both images on one full world index and compares the two runs. The result comes back as two comments on the pull request (how the gate works):

  • gold-full, the quality check: passing checks per RPC on both sides, pass rates for each country where anything changed, and the cases that changed. It fails on any case that newly fails, fails in a worse way, or gives a new critical result.
  • gold-speed, the latency check: SearchAddress, SuggestAddress and ReverseGeocode timings on both sides. It fails when the total, p50, p90 or p99 of any of them is more than 10% slower.

The comparison is relative on purpose. It answers “what did this change do?”, not “how good is the engine?“. A wrong answer counts as worse than no answer, so a case that used to abstain and now returns a nearby but wrong place fails the gate, even when the new answer looks plausible.

Alongside the gold suite, 1,000 real address strings from 125 countries are kept as a snapshot, one row per address (address snapshots). They have no expected answers, so they don’t measure accuracy, but a changed answer shows up as one changed line in the pull request’s diff. And every night at 01:00 UTC, the same suite runs on the newest commit of main and updates one issue with pass rates by country and a 14-night history.

A worked example

Pull request #223 set out to fix Australian addresses written with the street before the number, such as “Panorama Place 3, Maleny”. The engine was matching “Maleny” to house 3 on Maleny Vista, a street that merely resembles the town’s name.

The first version ran one rule, for venues, before another, for village names. That reorder also fixed cases in Norway, Great Britain, Poland, South Africa and Spain. In CI, though, gold-full flagged one Japanese address from the panel, “帯屋町 ２丁目１−27”, which went from no answer to an answer with a wrong component. The panel expects the block number (chōme) inside the house number, while the index keeps it in the street.

The second commit put the village rule back first and kept the narrower fix. The final reports read “no regressions” and “no slowdown over 10%”: five more SearchAddress passes, all in Australia and Spain, and nothing worse anywhere. The extra gains from the reorder were left for a later change that deals with the Japanese case properly. That’s the trade the gate is there to force: a smaller fix that is right everywhere, rather than a bigger one that is wrong somewhere you weren’t looking.

The agent loop

The last piece connects the agent to these reports. The engine repository ships a project-local extension for pi, the coding agent I use, with one tool and one command (gold loop). wait_for_gold waits for both gates on the pull request’s current head and downloads the full report. /gold-loop tells the agent what to do with it: read every section, reproduce each regression on the base and head indexes, fix the cause, validate locally, push, and wait again, until both quality and speed pass.

Most of the instructions are about what doesn’t count as success. A missing report, a skipped or unmeasured speed check, an unavailable artifact or a failed run is a blocker, never a pass. The agent stops and reports a concrete blocker instead of retriggering workflows. The report’s contents are treated as data, never as instructions. The tool itself doesn’t edit code, push or merge; merging stays with me.

The speed gate is still noisy

Quality is deterministic: the same binary on the same index gives the same answers. Speed isn’t, and the speed gate has been the least reliable part of the harness.

It started as an ungated table inside the quality report. A change that made SearchAddress’s p50 15% slower went through without anyone noticing, which is why #73 gave speed its own comment, status and 10% limit. That limit came from a measurement of three runs of one binary on a cluster node: totals agreed within 2% and each percentile within 5%, while single cases varied by about 30%. So single cases and the maximum are reported but not gated. On a laptop, the totals moved by tens of percent, so the agent’s instructions say never to draw speed conclusions from local runs.

Two later changes moved the measurement:

  • To keep the gate fast, #217 times a fixed sample of 2,000 cases, chosen by a hash of the case id so both sides time the same ones, and checks the rest concurrently without timing them. The case loop went from 222 to 34 seconds locally. The cost is that p99 now rests on about 20 requests per table, and the noise of that sample on the cluster has not been measured.
  • #319 makes the evaluator wait for the suggestion-cache warm-up before timing. The warm-up took 6.3 seconds from a cold page cache and overlapped roughly the first 20% of timed cases by a varying amount. Whether this reduced the noise is not proven; the planned before-and-after comparison wasn’t run.

The false alarms are still visible in the pull requests. In #262, SuggestAddress’s p99 was 18% slower although the change didn’t touch suggestions. #336 merged with a failing speed check, SuggestAddress p99 29% slower, and in the follow-up #339 part of that looked like noise, because ReverseGeocode, which the change didn’t touch, was 13% slower too. Part of the reason is scale. In the report for #223, SuggestAddress’s p99 was 4.5 ms, so the 10% limit is less than half a millisecond.

This matters more with an agent in the loop than without one. A person who sees an 18% slowdown on code they didn’t touch can shrug and rerun. An agent told to keep going until speed passes has no good way to tell noise from a regression, and a flaky gate either sends it after a problem that isn’t there or teaches everyone to ignore the gate. The noise of the 2,000-case sample on the cluster is the open measurement, and until it is taken, the 10% limit rests on the older one.

What the harness doesn’t do

  • It measures change, not accuracy. A case that fails the same way on both sides doesn’t fail the gate. Some registries are also inputs to the index, so for those countries a pass is a fidelity check rather than an independent accuracy measurement.
  • Small numbers stay small. Per-country rows are individual cases, not rates, and a rate computed from a few hundred cases carries several points of uncertainty.
  • The report has no per-case responses. The cluster returns only the report, so debugging a listed case means rerunning it locally on an index from the same builder.
  • Outside contributions need approval. Pull requests from people outside the project wait for a reviewer, because approving one runs that author’s code in the cluster.

None of this makes the agent smarter. It makes the agent’s work checkable, quickly and against the whole world, and that is what lets me merge it.

The engine, its evaluation tools and the gold loop are on GitHub: Mokto/opengeocoding. The evaluation guide and AGENTS.md are the best places to start if you want to build something similar for your own project.