Did you know? WordPress itself create a virtual robots.txt file if you don’t manually create a robots.txt file in the root folder. As mentioned its a virtual file and basically you cannot see or not able to find Wordpress virtual robots.txt location. Actually it virtually resides in the root directory. So don’t waste your time searching for it. That virtual robots.txt file become invalid after you create a robots.txt file and upload to your site’s root directory. Thats a physical robots.txt not virtual.
Robots.txt files are “Bots” stopper that stop search engine robots crawling your website files and folders. If you don’t need search engine to index a particular folder and its contents indexing to their search results, specify a “User Agent” and normally robots obey it like Google bots. Or you can allow only allow Google bots to crawl your site while denying all the other crawlers.
Robots.txt file come in handy in SEO rankings, by specifying your Sitemaps in the robots.txt file. So search engines can easily identify your sitemap and start crawling the urls in that sitemap. Checkout DigitPAGE’s Robots.txt file. That is a default robots.txt file and i added DigitPAGE’s xml Sitemap to it.