Add the robots.txt file to my WordPress site
Search engines, like Google, Yahoo!, and Bing, use robots to crawl your site's pages and media files in order to index them so they can be shown among search results. If you have the robots.txt file on your site, you can set up different directives to allow or deny crawling of specific sections of your site. Preventing robots from crawling less significant pages and files can help reduce server load. It'll also prevent robots from reaching their crawl quota per site before they get a chance to go through important pages.
- Sign in to WordPress.
- Install and activate the Yoast SEO plugin.
- In the left-side menu, select Yoast SEO and then select Tools.
- Select File editor.
- Under robots.txt, select Create robots.txt file. A new page with code will load, this is a developer view of your site created by Yoast for advanced troubleshooting. You can go back to the WordPress dashboard with your browser's back button.
Note: If you already have a robots.txt file on your site, you can edit the file.
When you go back to the File editor page in your WordPress dashboard, you'll see the newly created robots.txt file. On the server, the file is in the root folder of your WordPress site.
Related steps
- If you want to customize the robots.txt file on your site, check out Google's documentation for the list of Google crawlers and more details about the file syntax.
- To remove the robots.txt file from your site, connect to your hosting account with FTP or File Manager, find the root folder with your WordPress installation and delete the file.
More info
- For issues related to the Yoast SEO plugin, check out the plugin developer's official documentation and support forum.
- Yoast's ultimate guide to robots.txt
