“Converting HTML to text” in Liu et al. (2023)

Companion repo for “Evaluating Verifiability in Generative Search Engines”.

Converting HTML to text

To extract text from HTML pages, we first used single-filez to download cited webpages and their associated assets (e.g., CSS and images). Then, we use the Chrome DOM Distiller to extract the “readable” portion of the page (this is the view that appears when you use “Reader Mode” in the Chrome browswer). Finally, we used Trafilatura to extract the text from the DOM-distilled HTML.