← Blog

What "accurate" means for a geocoder, and how I decide a change is safe

Theo Mathieu ·

“How accurate is it?” sounds like one question, but a geocoder answers several at once. Did it find anything? Is the answer in the right country? Is the point close enough to be useful? Is it as precise as the address you sent, a house for a house and a town for a town? And when it returns a house number or a postcode, does that agree with yours?

This post explains how I score those questions for OpenGeocoding, how close I count as close enough, which losses I accept when I change the engine, and how I keep the test set from grading the index on its own data. The first post has the pass rates; this one is about what stands behind them.

A pass has several conditions

The scorer looks only at the first result. A best-of-several answer would flatter every service, and an application usually takes the first one. It ignores the confidence each service reports, because those numbers don’t mean the same thing from one provider to the next. A case then passes when the answer meets all of these, taken from the case’s expectation (what a pass means):

  • the country is one of the allowed countries;
  • the point is within the case’s distance tolerance;
  • the precision is within the allowed range, so a house case can’t pass on a street;
  • a house number or postcode it returns agrees with the request.

Failures are not one bucket either. The evaluation guide separates too_coarse (right country or region, but not the precision the address asked for), abstain, near_miss, far_miss (more than 100 km away), wrong_country and a few others. They matter because they aren’t equally bad. A country answer to a street address is a different failure from a point in another country, and the second is the more serious one.

How close is close enough

The tolerance depends on what the case asks for, and the corpus records it per case instead of using one global number:

  • Registry houses: the tolerance is recorded on each case. The US sample uses 150 m, or 300 m where the register places the point at a parcel centroid, and a separate sample of US addresses with loosely placed points is scored at 500 m and reported apart (US sample).
  • Landmarks: 1 km. Those cases measure coarse coverage, not the position of an entrance.
  • Company-address town cases: 5 km, rising to 10 km from 50,000 people and 20 km from a million (GLEIF cases).
  • Indonesian villages: twice the radius of a circle with the village’s area, between 2 and 25 km.

None of this tells you whether a point is the front door. The scorecard says so directly: the pass rates don’t measure postal deliverability, entrance accuracy or unit correctness. Whether 150 m is good enough depends on your workload. For enriching company records with a town and a region it is more than you need. For routing a courier to a flat it is not enough.

Which losses I accept when I change the engine

A pass rate says how good the engine is. A change needs a different question: what did it break? Every pull request that touches the service, its data or the test corpus is evaluated against the commit it branched from, on the same index, and the two runs are compared case by case (gates).

The rule has one exception:

  • A case that passed before and fails now fails the change, as does a case that moves to a worse failure class and a new critical result.
  • The exception is the panel built from OpenStreetMap addresses, which measures agreement with data the index itself reads. There I accept at most 5 locations that newly fail, and only when at least 5 times as many newly pass and nothing else fails.
  • Losses on registries, Wikidata, GLEIF and constructed cases are never traded for gains elsewhere, because those measure accuracy against sources outside the index.
  • Speed is a separate gate: the change fails if SearchAddress, SuggestAddress or ReverseGeocode gets more than 15% slower in total, at p50, p90 or p99.

So the acceptable loss is zero where the evidence is independent, and a small, explicit trade where it is not. Two experiments in the engine’s notes show why I keep it strict. A change to town matching reached 2,360 passing towns, but introduced 52 regressions where a coarse answer or an abstention became a point more than 5 km from the expected one. I dropped it. The version I kept finds 1,880 and, in the local comparison, adds no newly failing check. Another prototype gained 17 passes by trusting a place-type word alone, and it selected the wrong Samoa namesake, so I rejected that too.

A gate like this measures change, not accuracy. It tells me a commit didn’t make the cases I know about worse. It says nothing about the addresses nobody wrote a case for.

Not grading the index on its own data

A geocoder tested on the data it was built from will look good. The corpus has several defences against that, and each has a limit.

Expected answers come from outside the engine. Wikidata, national address registries, the US Census geocoder and similar sources provide them, never Nominatim, another geocoder or this service. GeoNames and Overture, which the index also reads, only chose which places to ask about.

Registry samples start from the registry. Cases first drawn from the index’s own Overture data and then checked against a register can’t show an address the index lacks, so recall stays invisible. The registry samples are drawn from the register first, and the index is asked about whatever was drawn (registry-first samples).

Overlap is reported, not hidden. Several registries used as expectations are also inputs to the index, so in those countries a pass measures fidelity to the register. The OpenStreetMap panel measures agreement with the index’s own source. Wikidata towns partly come from GeoNames, so some town gains reflect a shared origin. The reports group results by kind of evidence so the independent sources are not blended with the self-referential ones.

Balance, and how much a number can carry

Counts per country are not equal, and the headline rate depends on that. Town cases take up to 100 settlements per country, company-address cases up to 50 and the landmark house-coverage cases 20, so no country dominates those groups. The registry samples differ in size: from 120 cases for French Polynesia to 2,794 for South Korea, and they make up most of the 29,013-query selection that the published pass rate is computed on. That is why the scorecard also breaks results down by cohort and by registry, and why the cohorts differ in level and difficulty.

Variants of one location (a typo, a missing country) share a key, so they don’t count as independent places. About one location in five is held out from development and reported separately. With around 250 cases in a slice, the evaluation guide notes, an aggregate rate carries several points of uncertainty, and a single country’s row is one piece of evidence rather than a rate. I read the per-country tables that way, and I’d ask you to as well.

One more limit applies to all of it: this is a vendor-run evaluation on my own corpus, as the scorecard states at the top, and human review of the cases is still pending.

What to do with this

Use the same method on a sample of your own addresses. Decide first what close enough means for your workload: a town for enriching profiles, a house within 150 m for deliveries. Then geocode a few hundred addresses from each country you care about and look at the failures, not only the rate. The free plan has 50,000 requests a month with no card, and the bulk geocoding guide has a script that geocodes a CSV. The methodology is in the evaluation guide, and if a country of yours behaves worse than these tables suggest, I’d like to hear about it.