robotstxt

A generator for websites that creates files for crawlers that search engines use so that the website will be indexed and better ranked in searches.