How to Use Robots.txt Disallow to Block Pages from Search Engines

The robots.txt file is a simple yet essential tool for controlling how search engine crawlers (also called bots or spiders) interact with your website. It is a plain text file placed in your site's root directory (e.g., www.example.com/robots.txt) and provides instructions to bots on which parts of your site they can or cannot crawl. The proper use of robots.txt helps improve site performance, protect sensitive areas, and manage how search engines index your website.

Why is Robots.txt Important?

• Search Engine Management: It helps you prevent certain pages from being indexed, which can improve SEO by focusing bots on the most critical content.
• Reduce Server Load: Disabling specific files or directories can reduce the number of unnecessary server requests.
• Content Protection: It restricts crawlers from accessing confidential sections of your website, such as admin pages.

For more information about robots.txt, read our article.

While the robots.txt file is a helpful tool for controlling how search engine bots interact with your site, it has several limitations that should be considered:

• Bots Are Not Required to Follow Robots.txt Instructions:

       The robots.txt file is a voluntary standard, meaning that while most reputable search engines like Google, Bing, and Yahoo respect its directives, not all bots are obligated to follow these rules. Malicious bots or less compliant crawlers can ignore the instructions and access the content you're trying to block.

• Robots.txt Is Publicly Accessible and Can Be Bypassed:

       Anyone can view your robots.txt file by simply navigating to yourdomain.com/robots.txt. This makes it easy for anyone, including bots, to see which areas of your website you're trying to block. If you're using robots.txt to hide sensitive or private content, be aware that this approach won’t provide security—it only signals which areas you’d prefer search engines not to index.

• Cannot Be Used for Security Through Obscurity:

       The robots.txt file should not be relied upon as a security measure. Using it to "hide" content is ineffective, as determined individuals or bots can bypass the file's instructions. Sensitive or confidential data should be protected through other means, such as password protection or server-side access controls, and should not simply be omitted from search engine indexing using robots.txt.

The Disallow directive in robots.txt tells search engines not to crawl specific files, directories, or sections of your website. This is crucial when you want to restrict access to pages that shouldn't appear in search results or prevent bots from overloading your server with requests to less important site areas.

In a robots.txt file, the Disallow command follows the User-agent directive, which specifies the bot to which the rule applies. When you disallow a file or directory, bots are instructed not to access or index that content.

Example of Robots.txt Disallow:

User-agent: *
Disallow: /private-directory/

This means all bots (User-agent: *) are not allowed to crawl the private-directory folder.

The Disallow directive can be used in several ways, depending on what you need to block. Here are the most common use cases:

Block One File (Disallow Specific Files)

To block a single webpage from being crawled by bots, use the Disallow directive to specify the URL path.

Example:

User-agent: *
Disallow: /example-page.html

This prevents bots from crawling the page example-page.html on your site.

Block One Directory (Disallow Specific Folders)

If you want to prevent search engines from accessing an entire directory, use the Disallow directive with the folder path.

Example:

User-agent: *
Disallow: /images/

This prevents all bots from crawling the contents of the images directory.

Allow Full Access

You can leave the Disallow directive empty to allow bots to crawl the entire website without restrictions.

Example:

User-agent: *
Disallow:

This indicates that all pages are accessible to bots.

Hide the Entire Website from Bots (Disallow Specific Bots)

You can use a wildcard to block all bots' access to your website to completely hide it from search engines.

Example:

User-agent: *
Disallow: /

This directive prevents all bots from accessing any part of the website.

Disallow Specific Bots

You can target a specific bot by name to block it from crawling your site while allowing others to.

Example:

User-agent: Googlebot
Disallow: /

In this case, only Google’s crawler (Googlebot) is blocked from accessing the site, while other bots are allowed.

Allow and Disallow Together

   You can combine Allow and Disallow to fine-tune crawling. For instance, you can block a directory but allow a specific file within it.

       User-agent: *
       Disallow: /private/
       Allow: /private/public.html

Case Sensitivity

       The paths specified in Disallow are case-sensitive. Ensure the file and directory names match exactly.

One Directive Per Line

       Only one Disallow directive should be used per line to avoid confusion.

By adhering to these rules, you can control which parts of your website are accessible to search engine crawlers, optimize the site's indexing, and protect sensitive areas.

In the robots.txt file, wildcards such as an asterisk (*) can be used to manage access to specific pages and directories more flexibly. The asterisk (*) can be used as a wildcard to block multiple pages with similar patterns. It represents any sequence of characters, allowing you to create broader rules.

Example:

User-agent: *
Disallow: /private*/

This example blocks access to all URLs starting with "/private", such as "/private-data/" or "/private-docs/".

Example:

   User-agent: *
   Disallow: /*.pdf

   This blocks all URLs ending in ".pdf".

You can also use the $ symbol to indicate the end of a URL.

Example:

User-agent: *
Disallow: /*.pdf$

This rule prevents all PDF files on the site from being indexed.

Wildcards allow for more efficient indexing control, reducing the number of specific rules in the robots.txt file.

Using the Disallow directive in a robots.txt file helps control which parts of your website are crawled by search engines. However, mistakes in this file can lead to unintended consequences.

Here are some typical errors when using Disallow:

1.    Blocking Critical Pages. One of the most common mistakes is accidentally blocking essential pages from being crawled. For example:

Disallow: /

This rule blocks the entire website, preventing search engines from crawling any page. Be careful with broad rules like this, which can severely affect your site's visibility.

2.    Incorrect Directory Format. Forgetting to add a trailing slash when disallowing a directory can cause unexpected behavior. For example:

Disallow: /private

This would block only the specific "/private" page but not the entire directory. The correct way to block a directory is as follows:

Disallow: /private/

3.    Overusing Wildcards. While wildcards (*) are helpful in blocking multiple pages, overusing or placing them incorrectly can unintentionally block too much content. For example:

Disallow: /*.php

This rule blocks all .php pages, which may include important dynamic content.

Ensure that wildcards are only used when necessary and correctly.

4.    Mixing Allow and Disallow Incorrectly. Mixing Allow and Disallow directives without clear rules in complex setups can lead to unexpected results. For example:

Disallow: /private/
Allow: /private/data/

If not correctly configured, this might still block "/private/data/". Ensure that more specific Allow rules come after broader Disallow rules.

5.    Failing to Update the Robots.txt File. Over time, your website structure may change, but if the robots.txt file isn’t updated to reflect these changes, you might unintentionally block or allow pages. Regularly review and update your robots.txt file to match the current site architecture.

6.    Leaving Empty Disallow Rules. An empty Disallow rule like this:

Disallow:

doesn’t block anything but can be misinterpreted by some bots. Always ensure that the Disallow directive has a clear target.

7.    Blocking Resources Needed for Rendering. Blocking resources such as CSS, JavaScript, or image files can prevent search engines from properly rendering and indexing your website, leading to lower rankings. For example:

   Disallow: /scripts/
   Disallow: /images/

   This can lead to incomplete rendering of the page. It's important to ensure that search engines can access essential files required for proper display.

8.    Specifying multiple directories in one disallow instruction. Many site owners try to put all directories prohibited from indexing in one Disallow instruction.

Disallow: /css/ /cgi-bin/ /images/

This entry violates the standard, and it is impossible to guess how different robots will process it. The correct way is to write it like this:

Disallow: /css/ Disallow: /cgi-bin/ Disallow: /images/

9.    Listing all files. Another mistake is listing every file in a directory:

User-agent: *
Disallow: /AL/Alabama.html
Disallow: /AL/AR.html
Disallow: /Az/AZ.html
Disallow: /Az/bali.html

Instead, you can close the entire directory from indexing:

User-agent: *
Disallow: /AL/
Disallow: /Az/

10. Missing a Disallow instruction. Even if we want to use an additional directive and do not want to prohibit anything, it is best to specify an empty Disallow. According to the standard, the Disallow instruction is mandatory, and the robot can "misunderstand you".

This is wrong:

User-agent: googlebot

Host: www.example.com

This is correct:

User-agent: googlebot
Disallow:
Host: www.example.com

By avoiding these common errors, you can ensure that your Disallow rules effectively control which pages are crawled without negatively impacting your website’s SEO.

When using wildcards in the Disallow directive of a robots.txt file, it's easy to make mistakes that can affect how search engines crawl your site. Below are some common errors:

1. Overusing the Asterisk (*): Overly broad wildcard rules can unintentionally block important pages from being indexed. For example:

Disallow: /images*

This will block all pages with URLs starting with /images, including any subpages you want indexed.

2. Not Using the $ for File Types: If you intend to block a specific file type, forgetting to add the $ can block more pages than intended. For example:

Disallow: /*.pdf

This will block URLs that contain ".pdf" anywhere in the path. Adding the $ at the end ensures only PDF files are blocked:

Disallow: /*.pdf$

3. Misplaced Wildcards: Placing the asterisk in the wrong position can lead to unexpected behavior. For example:

Disallow: /blog*/

This may block all URLs starting with "/blog" but could miss variations like "/blog-archive/". Be careful to place the wildcard correctly based on your needs.

4. Using Multiple Wildcards Incorrectly: Combining too many wildcards in a single rule can make predicting which URLs are being blocked difficult. For example:

   Disallow: /*/docs*/images/*

   This could block more or fewer URLs than intended, depending on the structure of your URLs.

5. Forgetting to Test: Not testing the rules after adding wildcards can result in significant indexing issues. Always test your robots.txt with tools like Google Search Console to ensure the rules work as expected.

Avoid these errors to ensure efficient and accurate control over how search engines crawl and index your website content.

1. Test for Syntax Errors:

• It’s crucial to ensure that your robots.txt file is free from syntax errors. Even a small mistake can lead to unintended consequences, such as blocking essential pages from being crawled. Always double-check your syntax, especially when using directives like Disallow, Allow, or wildcards.

2. Use a Robots.txt Tester Tool:

• To validate the accuracy of your robots.txt file, it’s highly recommended to use a robots.txt tester tool. These tools, like the one available in Google Search Console, help you simulate how search engines interpret your file, identify any errors, and ensure that your site's instructions are followed as intended.

3. Regularly Review and Update Your Robots.txt File:

• As your website evolves, so should your robots.txt file. Review and update it regularly to reflect changes in your site's structure, content, and priorities. This will ensure that search engines continue to crawl the appropriate sections of your website while excluding unnecessary or outdated pages.

By adhering to these best practices, you can maintain a practical robots.txt file that enhances your site's SEO while avoiding common issues related to misconfiguration.

The Disallow directive in the robots.txt file is essential for controlling search engine crawlers and managing your site's visibility. Whether you need to block specific pages, entire directories, or certain bots, properly configuring your robots.txt file is crucial for optimizing your website's SEO and performance. Regularly review and update your robots.txt file to reflect your site's structure and goals.

Regular reviews of your robots.txt file are necessary to reflect any changes in your site's structure or content strategy. It's recommended that you use Google Search Console to test the file and prevent any issues.

It’s also important to regularly verify the availability of your robots.txt file. It should always return a 200 response code; otherwise, search engines will ignore all its directives if the file becomes unavailable (returns a 404 or 500 code). This could lead to unwanted indexing of pages you intended to block.

To avoid such issues, check the file's availability daily or at least weekly. For added convenience, you can use the Atomseo Broken Link Checker at https://error404.atomseo.com/SeoListCheck. Add your robots.txt file's URL, and the service will monitor its status, notifying you immediately if any issues arise so you can address them quickly.

Ensure your robots.txt file is functioning correctly—check it now for free.