Yoast SEO how to allow crawler bot

114 Views Asked by At

I got a job to get rid of "No information is available for this page" in a website. The website uses Yoast SEO, but it was disabled so I reenabled it and then I got a basic robots.txt like this

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

I have applied that settings for about six hours ago, tried to search it in Google, and still nothing changed. I feel anxious now. Is this enough for the crawlers to read the website? Do I miss something? Do I need to mess with .htaccess? I have zero exp in SEO, so any help would be very appreciated.

1

There are 1 best solutions below

1
On

copy and paste it in your robot.txt

User-agent: Googlebot
Disallow: 
User-agent: googlebot-image
Disallow: 
User-agent: googlebot-mobile
Disallow: 
User-agent: MSNBot
Disallow: 
User-agent: Slurp
Disallow: 
User-agent: Teoma
Disallow: 
User-agent: Gigabot
Disallow: 
User-agent: Robozilla
Disallow: 
User-agent: Nutch
Disallow: 
User-agent: ia_archiver
Disallow: 
User-agent: baiduspider
Disallow: 
User-agent: naverbot
Disallow: 
User-agent: yeti
Disallow: 
User-agent: yahoo-mmcrawler
Disallow: 
User-agent: psbot
Disallow: 
User-agent: yahoo-blogs/v3.9
Disallow: 
User-agent: *
Disallow: 
Sitemap: https://www.yoursitename.com/sitemap.xml