in

Robots.txt generator – Create a robots.txt file instantly

https://en.ryte.com/free-tools/robots-txt-generator/

Generate a robots.txt file for free with this easy-to-use tool. Make sure search engine crawlers are crawling and indexing your site. 

Search Engines are using robots (or so called User-Agents) to crawl your pages. The robots.txt file is a text file that defines which parts of a domain can be crawled by a robot. In addition, the robots.txt file can include a link to the XML-sitemap.

What do you think?