Parsing PDFs and messy documents

What it isCopy Anchor Link

A PDF file says "draw this character at this spot". It does not say "this is a paragraph" or "this is a table". Plain text extraction often gives broken lines, repeated page headers, tables flattened into one long row and two-column pages read straight across. A scanned PDF is only a picture, so a normal parser returns nothing.

A parser tries to recover the structure: headings, paragraphs, tables and the reading order. Simple libraries such as PyMuPDF and pdfplumber work on PDFs that contain real text. Tools such as Docling and Unstructured also detect the layout. OCR (optical character recognition) reads text out of scanned images, and some vision models can read a page image directly.

A worked exampleCopy Anchor Link

A report page has two columns and a small table. Plain extraction reads across the page, so the first column's sentence "Net revenue rose 4 percent" is interleaved with the second column's text about shipping. The chunk makes no sense, and its embedding is a blur of both topics. A layout-aware parser returns column one in full, then column two, and turns the table into rows such as "| Year | Revenue |". Now a question about revenue finds the table row.

How to check itCopy Anchor Link

Before you index thousands of pages, parse ten and put them beside the originals. Look for tables that fell apart, text in the wrong order, footers repeated on every page and pages that came back empty. Strip repeated headers and footers. Keep the page number and the section title with each chunk, so answers can cite where they came from.

How a parser rebuilds structureCopy Anchor Link

A PDF with a text layer stores each character with an x and y position on the page. A parser works in steps:

1. Group characters into words and lines by their positions.
2. Group lines into blocks, using the gaps between them.
3. Decide the reading order, for example all of column one before column two.
4. Detect tables from aligned columns or ruled lines, and write them out as rows.
5. Drop or tag repeated headers and footers.

Simple libraries do the first steps. Layout-aware tools add a model that looks at the page shape and labels each region as heading, paragraph, table or figure.

A second example: a scanned pageCopy Anchor Link

A scanned contract has no characters at all, only pixels. OCR finds regions that look like text, then guesses each character. It makes typical mistakes: a zero read as the letter O, or "rn" read as "m". A clause number or an amount can change without any warning. Some vision models read the page image directly, but they can also misread digits or tidy up wording, so check amounts against the original.

Trade-offsCopy Anchor Link

Simple libraries are fast and free but ignore layout. Layout-aware tools are slower and need more computing power. Vision models handle messy pages best and cost the most per page. Choose by document type: clean exported PDFs rarely need more than a simple library, while scans and multi-column reports do.

What goes wrongCopy Anchor Link

The failures are quiet. A table becomes one long line, so no chunk holds a clean row. Page numbers appear in the middle of sentences. A page returns almost nothing because it was an image.

To confirm, count characters per page. Any page with fewer than about 20 characters deserves a look. Then pick one known phrase from the original and search the parsed text for it. If you cannot find it, the parser lost it.

Common mistakeCopy Anchor Link

Treating "I extracted the text" as the end of ingestion. A scanned file that quietly returns an empty string will never be retrieved, and nobody notices until a user asks about it.