Cloudflare will add X-Robots-Tag to some pages

95 Views Asked by At

Recently I find Cloudflare will add X-Robots-Tag(noindex, follow) to some of our pages. For example:

https://www.datanumen.com/fr/blogs/chatgpt-join-drawing-exam/

The response header in Chrome is below:

enter image description here

To make sure the flag is set by Cloudflare, I do the following check:

  1. Check all .htaccess files and confirm there is no setting of X-Robots-Tag.
  2. Use www.webpagetest.org script to setDNS to our server's IP directly, to bypass Cloudflare, as below:
setDns www.datanumen.com ##.##.##.##
navigate https://www.datanumen.com/fr/blogs/chatgpt-join-drawing-exam/

Then the response header will NOT contain X-Robots-Tag any more, the result is at https://www.webpagetest.org/result/231201_AiDcS6_52V/1/details/#waterfall_view_step1

So how to prevent Cloudflare from adding X-Robots-Tag?

I check some related posts:

https://community.cloudflare.com/t/apo-cache-inserts-x-robots-tag-noindex-follow-header/377185 https://community.cloudflare.com/t/x-robots-tag-setting/58774 https://community.cloudflare.com/t/noindex-sent-to-google-crawling-bot/51113/10

but still cannot solve the issue.

0

There are 0 best solutions below