Sharepoint 2013 Site Returns 304 Not Modified

1k Views Asked by At

One of the sharepoint 2013 site is not able to crawled by google bot. Google said, he is not able to reach Robots.txt. When i look from chrome developer tools, it seems server return 304 code.

How can i solve this problem?

EDIT: When i call the url without "www."(xxxxx.com/robots.txt) , server returning 200 ok.

User-agent: *

Allow: /

Disallow: /_layouts/
Disallow: /_vti_bin/
Disallow: /_catalogs/


Sitemap: http://www.xxxxxxx.com:80/sitemap.xml
2

There are 2 best solutions below

0
On BEST ANSWER

SOLVED:

Need to give access to crawler service on firewall.

Thanks.

0
On

Try this:

User-agent: *
Allow: /
Disallow: /_layouts/
Disallow: /_vti_bin/
Disallow: /_catalogs/
Sitemap: sitemap.xml