robots.txt Checker

One line in this file can take a site out of every search engine, and it is normally found months later. This reads the rules per crawler and then fetches the sitemaps the file promises.

Domain: Scheme and path are optional — https://example.com/page works too.

Check robots.txt