How can I get rid of the x-robots-tag in my http response header

Question

So I have a wordpress website on a server, on which the x-robots-tag is always included, because it's mainly a storage cloud hosting.

I want to get rid of the tag because with it my site won't get indexed for google and other search engines.

I tried to overwrite it in the .htaccess file and for some reason it works just fine for all html pages but all php pages don't change. I tried to unset the header, using functions.php and header.php of my wordpress theme but without any luck.

I used Header always unset x-robot-tag in the htaccess file.

Answer 1

If you're using Cloudflare's Flexible SSL then you can change your Site URLs to be HTTPS, a safe way to do this is adding this code to your wp-config.php

define('SP_REQUEST_URL', ($_SERVER['HTTPS'] ? 'https://' : 'http://') . $_SERVER['HTTP_HOST']);

define('WP_SITEURL', SP_REQUEST_URL);
define('WP_HOME', SP_REQUEST_URL);

It will use the requested protocol as the site URL, so in Cloudflare you can just create a page rule to send all HTTP traffic to HTTPS, that means the Site and Home URL will always be HTTPS (since you can now use HTTPS).

Your Yoast SEO or Other Plugin sitemap will then be in HTTPS. Be sure to turn on the other Cloudflare SSL settings such as automatic rewrites and also use the official Cloudflare plugin.

Answer : https://github.com/Yoast/wordpress-seo/issues/3877#issuecomment-347184510

Delete your sitemap from google webmaster console and resubmit again new one.

A few other solutions :

Change your seo plugin to another, or delete line if using google sitemap generator:

if(!headers_sent()) header('X-Robots-Tag: noindex', true, 200);

in /wp-content/plugins/google-sitemap-generator/sitemap-core.php

Attention This might cause another problem please backup your original files.