# Document Extraction Accuracy: Build a Field-Level Eval Set Before You Trust Any Model

> A single “99% accurate” number tells you nothing about whether an extraction pipeline is safe to automate. Here is how to build a field-level eval set, pick the right metrics, and turn accuracy into a routing decision instead of a marketing claim.

- Source: https://parserai.co/blog/document-extraction-accuracy-field-level-eval-set
- Published: 2026-09-11

---
Every extraction vendor, model and in-house prototype will quote you an accuracy number. None of them mean the same thing. Before you let a pipeline touch a payment run or a ledger, you need your own measurement — one that reports accuracy per field, on your documents, with the errors that would actually cost you money separated from the ones that would not.

## Why a single accuracy number is useless

“99% accurate” is not a claim you can act on, because it hides three choices the person quoting it already made for you: what counts as a unit, what counts as correct, and which documents were measured. Change any one of them and the same pipeline can be reported at 99% or at 60% without anybody lying.

Take a fifteen-field invoice. If you score at the character level, a model that got every field right except a due date is still above 99% — the wrong date is a handful of characters out of a few hundred. If you score at the document level, that invoice is a total failure. Both numbers describe the same extraction. Only one of them predicts what your team will experience on Monday, and it depends entirely on whether a wrong due date matters in your process.

The second hidden choice is the document mix. Accuracy measured on clean, born-digital PDFs from three cooperative suppliers tells you nothing about the 200-dpi fax someone scanned sideways. Almost all the real error in production extraction lives in a small tail of hard documents, and almost all the impressive benchmark numbers come from samples that quietly exclude it.

That tail is also where the reading step and the interpretation step get blamed for each other's mistakes. Before you tune a model, it is worth being clear about which layer you are measuring — the [walkthrough of OCR extraction and IDP](https://parserai.co/blog/ocr-extraction-in-2026-how-to-automate-document-processing-for-faster-more-accurate-workflows) separates capture, extraction, and validation, and an eval set is what tells you which of the three is actually costing you.

An accuracy number you did not define is a marketing artifact. An accuracy number you measured is a routing decision.

## Pick the right unit: fields, not documents

Score at the field level, then roll up. Field-level scoring is the only granularity that tells you where to spend effort: if your total and tax ID are near-perfect and your line items are at 84%, the fix is a line-item problem — a table-parsing problem — not a “the model is bad” problem. A document-level number would have told you only that a fifth of your invoices fail.

Keep the document-level number too, because it is what your reviewers feel. A document is correct only if every field you care about is correct, so document accuracy is roughly the product of the field accuracies — which is why a pipeline that looks excellent per field can look mediocre per document. That gap is not a contradiction; it is the arithmetic, and seeing both numbers side by side is what keeps expectations honest.

_Illustrative field-level accuracy for an invoice schema — the weakest fields, not the average, decide how much review you need._

Notice what the chart makes obvious and an average would hide: two fields carry nearly all the risk. Everything else is effectively solved. That is the normal shape of an extraction problem, and it is the reason field-level measurement pays for itself on the first run.

_Illustrative: the same pipeline scored at four levels of granularity, before and after a round of schema and prompt work. Document-level accuracy is always the harshest view._

## Build the golden set

A golden set is a fixed sample of your own documents with hand-verified correct values. It is the single most valuable asset in an extraction project, and it is cheap: a few focused days of work from someone who already knows the documents.

Size it by strata, not by instinct. You want roughly 150–300 documents, stratified so that every layout family, language, scan quality and edge case you actually receive is represented by at least 10–15 examples. Ten documents per stratum will not give you a tight confidence interval, but it will reliably tell you when a stratum is broken — and that is the decision you are trying to make.

Deliberately over-sample the tail. Your production mix might be 80% clean digital PDFs, but if your golden set matches that distribution, the hard cases contribute so few documents that a real regression in them cannot move the number. Sample the hard strata up, score each stratum separately, and weight back to production volumes only when you want a single headline figure.

Freeze it, and keep it out of the prompt

The moment a golden document ends up in a few-shot prompt, a fine-tune, or a “here are examples of tricky suppliers” instruction block, it stops measuring anything. Keep the eval set in a separate store from prompt assets, and keep a small holdout you never look at until a release candidate is final.

## Normalize before you compare

Most naive eval harnesses report failures that are not failures. `18450.9` versus`18450.90`, a tax ID with punctuation versus without, `14/08/2026` versus `2026-08-14`, two spaces instead of one — string equality calls all of those wrong, and if you let it, you will spend your first week chasing formatting instead of extraction.

Define a normalizer per field type and apply it to both sides before comparing: numbers to a fixed precision, dates to ISO, identifiers stripped of punctuation, text case-folded with whitespace collapsed. Write them down in the eval record, because a normalizer is a policy decision — deciding that punctuation in a tax ID does not matter is fine, as long as the system consuming the data agrees.

eval-record.json — one document, gold vs. predicted

```
{
  "doc_id": "inv-2026-00418",
  "source": "supplier-batch-b/scan-lowres.pdf",
  "strata": { "layout": "supplier_b_v2", "language": "pt-BR", "quality": "scan_200dpi" },
  "gold": {
    "invoice_number": "4181-A",
    "issue_date": "2026-08-14",
    "due_date": "2026-09-13",
    "supplier_tax_id": "12345678000195",
    "currency": "BRL",
    "total_amount": 18450.90,
    "line_items": [
      { "description": "Rolamento 6204-2RS", "qty": 120, "unit_price": 24.5, "amount": 2940.0 },
      { "description": "Correia dentada HTD-5M", "qty": 40, "unit_price": 61.2, "amount": 2448.0 }
    ]
  },
  "predicted": {
    "invoice_number": "4181-A",
    "issue_date": "2026-08-14",
    "due_date": "2026-08-13",
    "supplier_tax_id": "12.345.678/0001-95",
    "currency": "BRL",
    "total_amount": 18450.9,
    "line_items": [
      { "description": "Rolamento 6204-2RS", "qty": 120, "unit_price": 24.5, "amount": 2940.0 },
      { "description": "Correia dentada HTD 5M", "qty": 40, "unit_price": 61.2, "amount": 2448.0 }
    ]
  }
}
```

Here the predicted output differs from gold in three places, but only one of them is real: the tax ID differs by punctuation, the total by a trailing zero, and the due date by a month. After normalization, two of those collapse into matches and the third stands out as the defect it is.

scored.json — what the harness writes back

```
{
  "doc_id": "inv-2026-00418",
  "normalizers": ["strip_punctuation:tax_id", "iso_date", "decimal_2dp", "casefold_collapse_ws"],
  "fields": {
    "invoice_number":  { "status": "match" },
    "issue_date":      { "status": "match" },
    "due_date":        { "status": "mismatch", "gold": "2026-09-13", "got": "2026-08-13", "severity": "high" },
    "supplier_tax_id": { "status": "match", "note": "equal after strip_punctuation" },
    "total_amount":    { "status": "match", "note": "equal after decimal_2dp" },
    "line_items":      { "status": "partial", "precision": 1.0, "recall": 1.0, "field_errors": 1 }
  },
  "document_status": "fail",
  "blocking_fields": ["due_date"],
  "route": "human_review"
}
```

## Metrics that survive contact with production

For scalar fields, exact match after normalization is enough. For repeated structures such as line items, use precision and recall over the rows: precision tells you how many extracted rows were real, recall how many real rows were found. A model that silently drops the last row of a table on page two has perfect precision and terrible recall, and only one of those numbers will warn you.

Track two failure classes separately, because they need opposite fixes. A **miss** is a field the model left empty or marked unknown; it is annoying and visible, and it routes cleanly to a human. A **fabrication** is a confident, well-formatted value that does not appear in the document; it is rare and far more expensive, because nothing downstream flags it. Count fabrications on their own, and treat any increase as a release blocker even when overall accuracy improved.

Finally, measure calibration, not just correctness. If your pipeline emits confidence, bucket the predictions and check whether the 0.9-and-above bucket really is right about 90% of the time. Confidence that does not track accuracy is worse than no confidence at all, because it invites you to auto-approve exactly the wrong things.

150–300

documents in a first golden set

10–15

per stratum, minimum

2

failure classes to separate

## Turn the numbers into routing

The point of measuring per field is that you can then automate per field. Once you know invoice number, issue date and total are at 97–99% and due date is at 89%, you do not have to choose between “automate everything” and “review everything”. You auto-post the strong fields, send documents whose weak fields are blocking to a review queue, and show the reviewer only the fields that need a decision.

That is what the `blocking_fields` and `route` keys in the scored record above are for. Mark each field in your schema as blocking or advisory, and let the eval set tell you which is which: a field is blocking if a wrong value causes a payment error, a posting error or a compliance problem. Everything else can go through with a flag.

- A frozen golden set, stratified by layout, language and scan quality, kept out of every prompt.
- A normalizer per field type, written down and shared with the consuming system.
- Per-field exact match, plus precision/recall for repeated rows like line items.
- Fabrications counted separately from misses, with a release-blocking threshold.
- Confidence buckets checked against observed accuracy at least once per model change.
- Blocking vs. advisory marked per field, so routing follows the measurement.
- The whole suite runnable in CI on every prompt, schema or model change.

## Run it on every change, and watch for drift

An eval set that runs once is a slide. An eval set wired into CI is a safety net. Every prompt edit, schema change, model upgrade and post-processing tweak should run the suite and print a per-field diff against the last accepted run — the useful output is not the score, it is the list of documents that changed verdict.

Production drifts even when you change nothing: a supplier redesigns its layout, a scanner is replaced, a new client arrives with documents nobody anticipated. Sample a small slice of live traffic for human verification every week and plot it. A sustained dip means the world moved, and the fix is usually to add the new stratum to the golden set rather than to touch the prompt.

_Illustrative weekly accuracy on a sampled slice of live traffic — a dip that recovers after the new layout is added to the golden set and handled._

The test to apply before automating

For each field, ask: what happens if this value is wrong and nobody notices? If the answer is “nothing much”, automate it now. If the answer involves money leaving the company or a filing being wrong, it is a blocking field — and it stays in review until your own numbers, on your own documents, say otherwise.

## Common questions

### How many documents do I really need?

Enough per stratum to notice a break, not enough to publish a paper. Ten to fifteen documents in each layout or quality group, adding up to a couple of hundred, is a working first set. Grow it by adding the documents that failed in production — that is where the next regression lives.

### Can I use a model to grade the output instead of humans?

For free-text fields, a model grader is a reasonable way to judge equivalence. For the numbers, dates and identifiers that drive payments, compare against human-verified gold values — those fields are exactly where a grader's mistakes are least acceptable and normalized equality works perfectly well.

### What if accuracy is good but reviewers still complain?

You are probably measuring the wrong fields, or measuring the right fields on the wrong documents. Ask the reviewers for twenty recent documents they had to fix by hand, score those, and you will usually find a stratum missing from the golden set.

## Where this leaves you

Accuracy is not a property of a model; it is a property of a model, a schema, a document mix and a definition of “correct”, measured together. Build the golden set once, normalize before comparing, score per field, separate fabrications from misses, and the number stops being a claim you have to take on faith and becomes the thing that decides which documents a person ever has to look at.

Parser

## See it on your own documents.

In the demo we process your documents live — no slides, no fluff.

[Book a demo](https://parserai.co/demo)