Shah J Ali
https://en.ryte.com/free-tools/robots-txt-generator/
Generate a robots.txt file for free with this easy-to-use tool. Make sure search engine crawlers are crawling and indexing your site.
Search Engines are using robots (or so called User-Agents) to crawl your pages. The robots.txt file is a text file that defines which parts of a domain can be crawled by a robot. In addition, the robots.txt file can include a link to the XML-sitemap.