Wonderstacked

Webmaster

Robots.txt Checker / Tester

The Robots.txt Checker fetches a site's robots.txt and tests whether a specific URL path is allowed for a given crawler. It uses the robots-parser library to apply real precedence rules, and also reports crawl-delay and any declared sitemaps.

Last updated · Free, no signup

Loading interactive Robots.txt Checker / Tester…

What does a robots.txt tester evaluate?

Conflicting Allow and Disallow directives are resolved according to the requesting bot and the specificity of each matching path. This tester applies that precedence to a supplied crawler identity and target address, then surfaces sitemap declarations and crawl-delay values that are easy to miss during a manual review.

The page uses robots-parser and chooses the most specific matching path in line with common crawler behavior. A permitted path can still be noindexed or inaccessible, while a disallowed path may remain indexed from links. Different crawlers can interpret unsupported directives differently, so test the actual user agent that matters.