Real-world HTML is often messy with malformed markup, complex whitespace, tables, and boilerplate content. Good text extraction handles these challenges gracefully.
Malformed HTML
Common issues include missing closing tags, improperly nested elements, invalid characters, and mixed encodings. The HTML5 parsing specification handles most issues with browser-like error recovery. Libraries like jsdom (JS), BeautifulSoup (Python), and Nokogiri (Ruby) implement forgiving parsing.
Whitespace Handling
HTML collapses multiple spaces to one and treats newlines as spaces, except in pre tags which preserve formatting. Text extraction should produce readable output by preserving paragraph breaks, removing excessive whitespace, and converting br tags to newlines.
Tables and Lists
Tables can be extracted as row-by-row text, tab-separated values, markdown tables, or structured data depending on the use case. Lists should preserve their structure with markers like dashes or numbers in the plain text output.
Boilerplate Removal
Real web pages include headers, navigation, sidebars, ads, footers, and cookie notices. Main content extraction identifies the article body using techniques like content-to-boilerplate ratio analysis, DOM depth analysis, and machine learning classifiers to focus on relevant text.