I use remote: true
links a lot. In most cases, the response can be js
or html
, so searchbots don't cause a problem
But some links, are purely js response, and there is no HTML response (which causes exceptions and errors, which flood monitoring tools, email, alerts, etc)
How can I setup the link so searchbots (google,bing, other crawlers) won't try it?
Or - Is there a type of response I should return so the crawlers will know not to go back to that url?
rel="nofollow"
is what you can put on your links. that should prevent bots like google form crawling them https://support.google.com/webmasters/answer/96569?hl=en