Generate a robots.txt file for your site — allow every crawler, block them all, or list specific paths to disallow. Processed in memory and never stored.