Pulling clean article text and metadata out of web pages

Fetching a web page is easy. Getting the article out of it is not. A typical news or blog page carries navigation, cookie banners, related-story rails, newsletter prompts, comment threads and a footer, and the few paragraphs you actually want sit somewhere in the middle. If an agent or a retrieval pipeline consumes the whole page, it reads the noise as content. This guide covers what a good article extraction step should return, how boilerplate removal works and where it fails, how to recover authors and dates you can defend, what robots rules and licensing mean for your pipeline, and how to keep the source reference attached so that every downstream quote can be traced back.

What an extraction step should return

Before comparing libraries or hosted APIs, decide on the record you want at the end. Text alone is not enough. Without metadata you cannot rank by recency, attribute a quote, or tell two versions of the same story apart. A useful target record looks like this:

{
  "requested_url": "https://news.example.org/story?id=812&utm_source=feed",
  "final_url":     "https://news.example.org/news/city-council-budget",
  "canonical_url": "https://news.example.org/news/city-council-budget",
  "http_status":   200,
  "retrieved_at":  "YYYY-MM-DDThh:mm:ssZ",
  "title":         "City council approves revised budget",
  "authors":       ["A. Writer"],
  "published":     {"value": "YYYY-MM-DD", "source": "json-ld"},
  "modified":      {"value": "YYYY-MM-DD", "source": "meta"},
  "language":      "en",
  "site_name":     "Example News",
  "text":          "...main body as plain text or markdown...",
  "headings":      ["Background", "What changes"],
  "content_hash":  "sha256:...",
  "extractor":     "extractor name and version",
  "license_note":  "unknown"
}

Three fields carry more weight than they appear to. final_url and canonical_url let you deduplicate the same article reached through feeds, tracking parameters and syndication. The source next to each date records where the date came from, which matters as soon as sources disagree. extractor lets you reprocess only the pages a faulty version handled, once you have fixed it.

How boilerplate removal works, and where it breaks

Most main-content extractors combine a handful of signals:

• Semantic markup. Elements such as article and main, and the schema.org articleBody property, point straight at the content when a site uses them properly.
• Text density. Blocks with long runs of prose and few links score high; blocks made mostly of links (menus, tag clouds, "read next" boxes) score low.
• Structure and class names. Containers named like comment, sidebar, promo or share are penalized; names like content or post-body are rewarded.
• Cross-page comparison. On sites you process often, blocks that repeat on every page are template, not content. Comparing a few pages from the same site removes them reliably.

Open-source options include Mozilla's Readability library (the algorithm behind Firefox Reader View) and Trafilatura, a Python library for main-text and metadata extraction. Hosted article extraction APIs wrap similar ideas together with page rendering and scale. Whatever you choose, test it on the pages that usually defeat extractors:

• Pages that build their body with JavaScript, where a plain HTTP fetch returns an empty shell. The page on headless browser APIs covers when rendering is worth the cost.
• Live blogs and timelines, where the "article" is a stack of short timestamped entries.
• Articles split across several pages, or truncated behind a "show more" control.
• Consent or paywall interstitials. An extractor that returns the paywall notice as the article is worse than one that fails, because it looks like success.

That last point generalizes: design for visible failure. If the extracted text is very short relative to the page, contains a consent phrase, or matches the text of another page on the same site, flag it instead of passing it on.

Authors and dates you can defend

Bylines and dates look trivial and are a frequent source of quiet errors. One page can hold the publication date, an update date, the dates of linked stories, a copyright year in the footer and the timestamps of comments. Check sources in a fixed order and record which one you used:

1. Structured data. JSON-LD of type NewsArticle, BlogPosting or Article, with datePublished, dateModified and author. The vocabulary is documented at schema.org.
2. Meta tags. Open Graph properties such as article:published_time, and the meta names common publishing systems emit.
3. Markup in the body. A time element with a datetime attribute near the headline, or an element marked up as the byline.
4. Visible text. "Published" or "Updated" lines, parsed with the page's language and locale in mind.
5. The URL. A date segment in the path is a weak hint, useful only to break a tie.

A few rules save trouble later. Keep published and modified dates separate, and never present a modified date as the publication date. Store timestamps with their timezone, or note that none was given. When sources disagree by more than a day, keep both and mark the field uncertain. For authors, split multiple names, drop prefixes like "By" and role suffixes, and do not mistake a newsroom desk or an organization for a person. If no author is stated, store unknown rather than filling in the site name.

Robots rules, site terms and licensing

Being technically able to fetch a page does not settle whether you may fetch it, store it or reuse its text. Treat those as separate questions. What follows is general orientation, not legal advice; take specific cases to counsel.

• Robots rules. The Robots Exclusion Protocol, specified in RFC 9309, tells automated clients which paths a site asks them not to crawl. Check it before fetching, cache it sensibly, and identify your client with an honest user agent and a contact address. Page-level signals such as a robots meta tag or an X-Robots-Tag header carrying noarchive or nosnippet say something about storage and display as well.
• Site terms. Many publishers restrict automated access or reuse in their terms. Some offer feeds or licensed APIs instead, which are usually the cleaner path for a recurring need.
• Copyright. Extracting text so a colleague can read and analyze it internally is a different use from republishing it or showing it to customers. Short quotations with attribution and a link are a common pattern. Keeping full text indefinitely deserves an explicit decision with a named owner.
• Licenses on the page. If a page declares a license, such as a Creative Commons license, record it in the metadata so downstream users know what attribution and reuse it requires.
• Logins and paywalls. Content behind a login does not become public because you happen to hold an account. Do not let a pipeline reuse personal credentials or work around access controls.

Keep the source reference attached

An extracted article becomes useful when someone can later ask "where does this sentence come from?" and get an answer in seconds. That only works if the reference travels with the text through every step: chunking, summarizing, embedding, quoting.

• Give each article a stable identifier derived from the canonical URL and the content hash, so a changed article is stored as a new version instead of silently replacing the old one.
• Keep the retrieval timestamp. A page is evidence of what the site said at that moment, not forever.
• When you split text into chunks, store the article identifier and character offsets on every chunk, so a quote can be located in the original.
• Where the terms allow it, keep the raw HTML snapshot next to the extracted text. When your extractor changes you can reprocess without fetching again, and you can show what the page looked like.
• When a summary or an agent's answer cites the article, check that the quoted passage really appears in the stored text. A link shows where to look; it does not show the claim is there. The note on citations that hold up goes further.

Evaluate extractors on your own pages

Vendor demos use friendly pages. Build a small test set from the sites you actually depend on, including the awkward cases above, and write down the expected title, authors, dates and the first and last sentence of the body for each page. Run every candidate, compare by hand, and note how each one fails: missing paragraphs, leftover boilerplate, the wrong date source, or an empty result reported as success.

Libraries give you control and run next to your code. Hosted APIs take on rendering and scale; check what they keep from your requests, for how long, and what their terms let you do with the output. For fields beyond the article itself, such as prices or specifications, see structured data extraction with AI; for choosing the fetching layer, see the scraping API guide.

Where GoCux fits

GoCux does not fetch or crawl web pages, so it is not an article extraction API. It works on the material once it is in hand. Web content enters a GoCux project only through an authorized browsing or export step, and on import it keeps the original page reference and the date it was retrieved, which is the provenance this guide argues for. A link to a page behind a login is not treated as permission to get past it.

From there, Extraction pulls the values you ask for from the imported material and attaches the supporting passage to each one; a value the text does not state stays unknown. Knowledge search answers questions across a project's documents and returns passages with their citations. The page on web sources and responsible agent access explains how web material is handled.

Checklist for an article extraction step

• The output record is defined: requested, final and canonical URL, retrieval time, title, authors, dates with their source, language, text, hash and extractor version.
• Failures are visible: short, duplicated or consent-page text is flagged, not passed on.
• Published and modified dates are kept apart, with a timezone or an explicit "none given".
• Unknown authors and dates are stored as unknown, never filled with guesses.
• Robots rules are checked, the client identifies itself, and the terms of every recurring source have been read.
• Storing and reusing full text has an owner and a decision; page licenses are recorded.
• Every chunk carries the article identifier and offsets, and quotes are checked against the stored text.
• Candidates were tested on your own difficult pages and checked by hand.