Our robots.txt file contains only this:
User-Agent: *
Disallow: /
However, we've been receiving the below errors frequently. Not sure why both Bingbot and Googlebot are looking for a Robots.txt folder
IP: 207.46.13.42
HTTP status code: 404
URL: http://www.website.net/robots.txt/
User Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
The following error occurred:
The controller for path '/robots.txt/' was not found or does not implement IController.
IP: 66.249.74.48
HTTP status code: 404
URL: http://www.website.net/robots.txt/
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
The following error occurred:
The controller for path '/robots.txt/' was not found or does not implement IController.
We can code around this, but the question is why these major bots are trying to search for this folder? The goal of our robots.txt file is to prevent bots from crawling the site. From every example that exists on the internet we have set up the robots.txt file correctly to accomplish this but we still run into these bots trying to access a folder that does not exist and MVC tries to handle it causing an exception.