Name Update Time
Netflix October 18, 2024 10:43 am
Disney+ October 18, 2024 11:03 am
Max October 18, 2024 11:34 am
ChatGPT 4 October 18, 2024 2:27 pm
Spotify October 18, 2024 11:49 am
Prime Video October 15, 2024 4:17 pm
Codecademy October 16, 2024 2:37 pm
Grammarly October 16, 2024 2:31 pm
Canva Pro October 16, 2024 2:44 pm
Udemy Premium Cookies September 2, 2024 2:53 pm

How to correctly set robots.txt on WordPress website

File header rules:
The beginning of robots.txt file starts with User-agent:, which is used to specify search engine spiders. If you want to target Google search spiders, you can enter

User-agent: Googlebot

If you want to work on all search engines, enter

User-agent: *

Disallow rules:
Disallow: /abc means prohibiting access to abc.php, abc.html and all files under the abc folder.

Disallow: /abc/ means prohibiting access to all files under the abc folder, but not restricting abc.php and abc.html files.

Allow rules:
Allow rules are the same as Disallow.

Sitemap rules:
Sitemap is used to tell search engines the location of the sitemap

Sitemap: http://your domain name/sitemap.xml

Where sitemap.xml is the sitemap file of your website

To avoid including WordPress system files:
Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /wp-content/plugins

Disallow: /wp-content/themes

To avoid including duplicate content:
Disallow: /feed

Disallow: /articles/*/feed

The full text of robots.txt is summarized as follows:
User-agent: *

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /wp-content/plugins

Disallow: /wp-content/themes

Disallow: /feed

Disallow: /articles/*/feed

Sitemap: http://yourdomain/sitemap.xml