jump to search:⌘/ select search:⌘K navigate results:↑/↓ navigate suggestions:↑/↓ close suggestions:esc

ROBOTS.TXT PARSER

[cat:weblinks]

tags: robots-txt

November 17th, 2023

Will Critchlow’s ROBOTS.TXT PARSER on Nov 17, 2023

PARSE YOUR ROBOTS.TXT FILE THE SAME WAY GOOGLE’S CRAWLERS DO

Daniel S. Griffin, Ph.D.

About | Research | CV | ARCHIGNES

I study searching and build tools to make it better. How do people think about it, talk about it, and practice it? Admit it, extend it, repair it, own it? Share about it? Learn about it? Complain about it? We need better ways to think about, talk about, and do search. How can we make it better? Can we build tools to improve exit and voice?

- Care and Repair of Search Engines
- notes 2024-02-21
- notes 2024-02-12
- change notes 2024-02-12
- change notes 2024-02-07
- change notes 2024-02-05
- 2024-02-05
- “Search has become ubiquitous but that does not mean that search has been solved.”
- Human-centered Evaluation and Auditing of Language Models Workshop
- tooling to support people in making hands-on and open evaluations of search

Griffin. Situating Web Searching in Data Engineering (dissertation)
Griffin & Lurie. Search quality complaints and imaginary repair (2022)
Goldenfein & Griffin. Google scholar – platforming the scholarly economy (2022)
Burrell, Kahn, Jonas, & Griffin. When users control the algorithms (2019)
Mulligan & Griffin. Rescripting search to respect the right to truth (2018)

Updates

Weblinks

Change notes

Excerpts

Annotated tweets

Random page

Website Last Updated: