How to correctly set robots.txt on WordPress website
File header rules:
The beginning of robots.txt file starts with User-agent:, which is used to specify search engine spiders. If you want to target Google search spiders, you can enter
User-agent: Googlebot
If you want to work on all search engines, enter
User-agent: *
Disallow rules:
Disallow: /abc means prohibiting access to abc.php, abc.html and all files under the abc folder.
Disallow: /abc/ means prohibiting access to all files under the abc folder, but not restricting abc.php and abc.html files.
Allow rules:
Allow rules are the same as Disallow.
Sitemap rules:
Sitemap is used to tell search engines the location of the sitemap
Sitemap: http://your domain name/sitemap.xml
Where sitemap.xml is the sitemap file of your website
To avoid including WordPress system files:
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
To avoid including duplicate content:
Disallow: /feed
Disallow: /articles/*/feed
The full text of robots.txt is summarized as follows:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: /feed
Disallow: /articles/*/feed
Sitemap: http://yourdomain/sitemap.xml
