Robots.txt Tester

Fetch a site's live robots.txt and test whether specific paths are allowed or disallowed for a given crawler. The file is requested and processed in memory, never stored.