Synthetic training data for document AI

Document understanding is bottlenecked by data. The public benchmarks most teams fall back on were built to measure progress, not to train production systems. FUNSD, the Form Understanding in Noisy Scanned Documents dataset published by Jaume, Ekenel, and Thiran at ICDAR-OST 2019, contains exactly 199 forms — 149 training and 50 test. CORD, the Consolidated Receipt Dataset from Park et al., contains roughly 1,000 receipts. SROIE, the ICDAR-2019 Scanned Receipt OCR and Information Extraction challenge, ships 973 scanned receipts. IIT-CDIP Test Collection 1.0, the largest public document corpus, is unlabeled.

Those volumes are useful as yardsticks. They are not enough to train a production document extractor on a new form type. A single enterprise workflow — a W-2 ingestion pipeline, a CMS-1500 claims classifier, a 1040 field extractor — routinely needs thousands to tens of thousands of labeled examples to reach usable accuracy on the long tail of layouts, fonts, stamp overlays, and field combinations that real documents contain.

Collecting that volume from real documents runs into three walls. First, privacy: tax returns, medical claims, and identity documents carry regulated PII, PHI, and financial data that cannot simply be uploaded to a labeling vendor. Second, cost: manual bounding-box and entity annotation on dense forms is the expensive kind of human labeling — minutes per page, not seconds — and a modest training set can burn five-figure budgets before the first model converges. Third, coverage: real-document corpora reflect the sampling bias of whoever collected them. You get whatever forms that team happened to scan, not the full distribution of layouts your production system will see.

Synthetic document generation flips all three constraints. There is no real PII to exfiltrate because every field value is drawn from a coherent fictitious identity. There is no human labeler because the generator emits the ground truth — it knows what it drew, down to the pixel. And the distribution is yours to control: you can oversample the layouts, fonts, and anomalies your downstream pipeline struggles with.

The canonical unit in our output is the WordAnnotation. Every word the renderer places on the page carries its text, its pixel bbox ( x, y, width, height), a page number, a field_id tying it back to the source form definition, an entity_type (e.g. answer, label, header), and a linked_field_id when the word is a label pointing at a value field. Every downstream model format is a projection of that core record.

• FUNSD linking. Words with entity_type = "label" become question entities; their linked words become answer entities; the linked_field_id edges become the FUNSD linking pairs.
• LiLT input. Word text becomes the token sequence, bboxes are normalized into LiLT’s [x0, y0, x1, y1] 0-1000 scale, and the entity type becomes the per-token class label.
• Donut prompt. The structured answer tree is serialized into Donut’s <s_key>value</s_key> target grammar, keyed by field_id. No bboxes are needed — Donut is OCR-free.
• YOLO labels. Word bboxes are grouped by field_type to form class regions, then normalized to YOLO’s (cls, cx, cy, w, h) 0-1 space using the page dimensions the document was rendered at.
• BIO sequence. The word list is tokenized in reading order; the first word of each entity span gets a B- tag, continuations get I-, and unlabeled whitespace gets O. The canonical BIO sample above is generated directly from the same fixture.

Because every format is derived from one underlying annotation graph, switching architectures is free. You don’t have to regenerate your batch to experiment with a Donut target instead of a LiLT tagger; one call gives you both.

The math for synthetic is straightforward and ugly for the alternative.

Volume. A labeled FUNSD-scale dataset of 199 forms takes minutes to generate and a few hundred credits to render. A dataset an order of magnitude larger than CORD’s 1,000 receipts costs less than the coffee budget for a single human labeling session. The constraint on dataset size stops being your bank account and becomes your training infrastructure.

Cost. Published benchmarks for professional document annotation vary, but typical rates run a few minutes per page for dense forms at a few tens of cents per labeled field. For a W-2 with dozens of labeled fields, that’s real dollars per document before quality review. The synthetic-equivalent marginal cost is whatever a single render costs — orders of magnitude less, and the labels come out right the first time.

Privacy. A real document dataset imports every compliance obligation the source documents carried. Real W-2s bring tax-data handling rules. Real CMS-1500 claims bring HIPAA obligations. Real driver’s licenses bring identity-document handling requirements. Synthetic documents bring none of them, because no real person’s data ever touched the pipeline.

Iteration speed. Real datasets are static snapshots. Need to stress-test your model against a new layout variant, a new optional field, or a new date format? With synthetic data you change the form definition and regenerate in minutes. With real data you launch another labeling project and wait.

Most document-AI systems in production fall into one of two architectures. Synthetic data slots naturally into both.

Two-stage pipeline: detection then extraction

The classical pattern is a detector that finds field regions followed by an extractor that reads structured values from those regions. A YOLOv8 region detector trained on synthetic bounding boxes localizes header, line-item, total, and party regions on an invoice; a LiLT or LayoutLMv3 tagger then consumes each region and emits structured key-value pairs. For the detection half, see our guide on synthetic training data for YOLOv8. For the extraction half, see synthetic training data for LiLT.

End-to-end pipeline: OCR-free image-to-JSON

The newer approach skips OCR entirely. Donut, Pix2Struct, and related vision-language models take the raw page image and emit structured JSON directly, bypassing the OCR-then-parse pipeline. These models need a lot of training data to learn both the visual reading and the structured decoding, which is exactly the kind of volume synthetic data unlocks. For the full end-to-end pattern, see synthetic training data for Donut.

A third pattern worth mentioning is the classification head: a lightweight model that decides which template a document belongs to before routing it to a specialized extractor. Synthetic data is especially useful here because you control the class balance exactly — real-world document streams are always skewed, and a classifier trained on the real skew will quietly misclassify the long tail.

Synthetic document data exists outside the regulatory frameworks that govern real personal and financial records. That’s not a loophole — it’s the direct consequence of the data never having described a real person.

Every identity in a SymageDocs generation batch is drawn from a coherent fictitious persona. The name on a synthetic W-2 is fictional; the SSN is a syntactically valid but unassigned value; the EIN, address, and wage numbers are generated so the overall record looks plausible but corresponds to no real taxpayer. The same applies to CMS-1500 patients, driver’s licenses, passports, and every other form class we support.

The practical consequences:

• HIPAA. Synthetic CMS-1500 and healthcare claims data contain no PHI by construction, so training a claims model on them does not trigger HIPAA handling obligations.
• GLBA / financial data. Synthetic tax and banking documents carry no customer information, so they sidestep the safeguards rule entirely.
• GDPR / CCPA. Synthetic data is not personal data. It can be moved across borders, shared with vendors, and retained indefinitely without DSAR, erasure, or processing-basis obligations.
• Model sharing. A model trained on synthetic data can be published, open-sourced, or handed to a partner without the reidentification risk that comes with models memorizing rows of real training data.