Schema-first extraction: every value tied to the passage behind it

Extraction is the step that turns pages and documents into rows: a company's headquarters city, a contract's renewal date, a product's dimensions, the sponsor of a clinical trial. Language models made this far easier to start, because you no longer write a parser for every layout. They also made it easier to get wrong at volume, because a model asked to fill a field will usually fill it, whether or not the source says anything. This page covers how to design extraction you can check: the schema, a supporting passage for every value, honest unknowns, and validation that still works on the last document of a large batch.

Getting the material and reading it are separate jobs

"Extracting data from websites" bundles two problems that fail in different ways.

• Acquisition gets the content: fetching pages, rendering JavaScript, following pagination, downloading PDFs, calling an API or exporting from a system. Its failures are blocked requests, login walls, cookie banners captured instead of content, and layouts that change overnight. It also carries obligations: a site's terms, its robots rules, rate limits, and the difference between public pages and pages behind an account.
• Extraction reads that content and produces fields. Its failures are wrong values, values with no basis, and inconsistent formats.

Keep the two apart in your pipeline. At acquisition time, store the raw content with its source address, the retrieval time and a hash of what you received. Extraction then always runs against a known snapshot, and any value can be traced back to exactly the text it came from. For the acquisition side, see what web scraping is and choosing a scraping API; for pulling clean article text specifically, see article extraction.

Design the schema before the prompt

Most extraction quality is decided before a model sees a single page. A good schema answers, for every field:

• Type and format. Dates in ISO 8601, amounts as a number plus a separate currency code, countries as codes rather than free text.
• Allowed values. If a field is a category, list the categories. "Other" is a valid option; an invented category is not.
• Cardinality. One value or a list? A company can have several offices; a contract has one effective date.
• A definition with edge cases. "Headquarters" means the registered or stated head office, not the largest office and not the address of a parent company.
• A status alongside the value. Whether the value was found, not stated, ambiguous or conflicting.

Writing the schema as JSON Schema lets you validate every output mechanically. Here is one field, designed with evidence in mind:

"headquarters_city": {
  "value": "string or null",
  "status": "found | not_stated | ambiguous | conflicting",
  "evidence": [
    {
      "source": "document id or page address",
      "retrieved_at": "timestamp of the stored snapshot",
      "quote": "exact passage from the source",
      "location": "page, section or element"
    }
  ]
}

A supporting passage for every value

The most useful single rule in extraction is that no value enters your table without the passage that supports it. The passage turns review from "is this right?" into "does this sentence say this?", which a person can answer in seconds. It also enables a cheap automatic check that catches a whole class of errors: verify that the quote actually appears in the stored source text, after normalizing whitespace and punctuation. A quote that cannot be found was invented, and so, probably, was the value next to it.

Presence is necessary but not sufficient. A real sentence can still fail to support its value: a quote about a regional office attached to the headquarters field, or a date that belongs to a different contract in the same file. Check that the value is contained in, or derivable from, the quote (for numbers and dates this can be done in code), and sample the rest for human review. A citation points to evidence; it does not prove the reading is right. Citations that hold up goes deeper into what makes a pointer trustworthy.

Unknown instead of guessed

A model's instinct is to be helpful, and in extraction helpfulness looks like a plausible value in every cell. Plausible values are the worst kind of error, because they survive a quick glance. Make unknown a first-class answer.

• Allow null in the schema and require a status that explains it.
• Distinguish not stated (the source was read and says nothing), not checked (the relevant page was never retrieved), ambiguous (the text supports more than one reading) and conflicting (two sources disagree). Each has a different next step.
• Forbid inference from outside the source: no guessing a country from a domain ending, no filling a founding year from general knowledge.
• Make downstream code respect it. An unknown revenue must not become zero in a spreadsheet, and an empty string must not count as a match.

field              value       status       evidence
headquarters_city  Rotterdam   found        about page: "Our head office is in Rotterdam"
founded_year       null        not_stated   checked: about, contact, footer
employee_range     null        conflicting  careers page "over 100" vs about page "around 60"

A table with honest gaps is worth more than a full one, because you know which cells to trust. Missing evidence is an open question explains why a gap should stay visible rather than be smoothed over.

Validation that works at scale

Reviewing every row by hand stops being possible quickly. What replaces it is a set of layered checks, each catching a different failure.

1. Structure. Validate every record against the schema: types, allowed values, required fields. Reject and retry rather than repair silently.
2. Evidence. Confirm that each quote exists in the stored source and that numeric and date values can be found inside their quotes.
3. Consistency. Apply rules across fields (an end date after its start date) and across records (one company, one headquarters; no duplicate rows for the same entity).
4. Drift. Watch the share of each status per batch. A sudden rise in "not stated" usually means the acquisition step broke, for instance because a site began returning a consent page, not that the world changed.
5. A labelled sample. Keep a set of documents with hand-checked answers and re-run it whenever you change the schema, the prompt or the model. Compare field by field, not with a single overall score.
6. Coverage. Report what you attempted alongside what you got. A row count says nothing about how many matching entities you missed.

Record the schema version, the extractor version and the source hash on every row. When something changes, you can re-extract only what is affected and still explain any value long after the run.

Where GoCux fits

GoCux does not crawl the open web, so acquisition stays with whatever tool and permissions you use for it. When web material is needed, it arrives through an authorized browser or export step and is imported together with the address of the original page and the date it was fetched. From there, Extraction works on the material you supply: each value comes with the passage behind it, and a value the source does not contain is left as unknown rather than filled with a guess. When the job is to find every candidate that meets a set of criteria, FindAll returns a table in which each row carries its own evidence, and it says plainly where coverage stops instead of rounding the gaps away. Exporting the result counts as a consequential action, so the work pauses and lists the files that would leave. Bring your own evidence describes working from material you already hold.

Extraction checklist

• Is raw content stored with its source address, retrieval time and hash before extraction runs?
• Does every field have a type, allowed values, cardinality and a written definition with edge cases?
• Does every value carry an exact quote and its location?
• Is each quote checked automatically against the stored source?
• Can a field be unknown, with a status that says why?
• Is inference from outside the source explicitly forbidden?
• Do structural, evidence and consistency checks run on every batch?
• Do you track status shares per batch to catch broken acquisition?
• Is there a labelled sample you re-run after every change?
• Do reports state coverage, not just the row count?