How to set missing X-Robots-Tag for HTTP 301 response?

Question

I want to set noindex x-robots tag for a particular bad search engine which index even redirect page, instead of final destination.

At the top of my root .htaccess file, I have added below rules.

<IfModule mod_headers.c>
Header add X-Robots-Tag "BadBot: noindex"
</IfModule>

It works this way.

Requesting http://example.com/page

SERVER RESPONSE: HTTP/1.1 301 Moved Permanently

Date: Mon, 17 Jul 2017 11:17:10 GMT
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive
Location: https://example.com/page


SERVER RESPONSE: HTTP/1.1 301 Moved Permanently

Date: Mon, 17 Jul 2017 11:17:11 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Location: https://www.example.com/page/
X-Robots-Tag: BadBot: noindex


SERVER RESPONSE: HTTP/1.1 200 OK

Date: Mon, 17 Jul 2017 11:17:13 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Robots-Tag: BadBot: noindex

Requested: http://example.com/page Final: https://www.example.com/page/

In the requested URL, the X-robots-tag is missing while forcing HTTPS. Is there any way tackle this issue?

Thanks

Answer 1

I think you made a small syntax error. Instead of Header add X-Robots-Tag it should be Header set X-Robots-Tag

<IfModule mod_headers.c>
Header set X-Robots-Tag "BadBot: noindex"
</IfModule>

Reference : https://yoast.com/x-robots-tag-play/

Answer 2

Header add X-Robots-Tag "BadBot: noindex"

As posted this directive would not have set the header on either of the 301 redirects. It would have only set the header on the final 200 OK response.

Aside: In order to analyse why you were seeing the header on one of the 301 redirects we would need to see where/how the redirects were implemented and whether you have other code that could be setting this header. It looks like there is something else going on.

The default condition for the header directive is onsuccess, which means the header is only set on 2xx responses. To set the header on all responses, including 3xx (redirects) and 4xx (not found) then you need to use the always condition.

For example:

Header always set X-Robots-Tag "BadBot: noindex"

Aside: No need to use add here; set is preferable, since you only want to set this header once.