Robots.txt Generator

Optimize Your site in Search Engine

Robots.txt Generator

Default - All Robots are:  
Sitemap: (leave blank if you don't have) 
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo MM
  Yahoo Blogs
  DMOZ Checker
  MSN PicSearch
Restricted Directories: The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

The `robots.txt` file is a simple text file that webmasters create to instruct web robots (typically search engine crawlers) how to crawl and index pages on their website. This file is part of the Robots Exclusion Protocol (REP) and serves as a guideline for bots to understand which parts of a website should be accessed and which should be avoided. Here’s an in-depth look at how `robots.txt` works:

Basics of `robots.txt`

1. Location: The `robots.txt` file must be placed in the root directory of the website (e.g., ``). This is the standard location where web crawlers will look for the file.

2. Syntax: The file consists of one or more sets of instructions, each specifying user-agent directives followed by rules that allow or disallow access to certain parts of the website.

Key Components of `robots.txt`

1. User-agent: This directive specifies the name of the web crawler the rules apply to. A user-agent is the name of a web crawler, such as `Googlebot` for Google, `Bingbot` for Bing, etc. An asterisk (`*`) can be used to apply rules to all crawlers.

Example: `User-agent: *`

2. Disallow: This directive tells the crawler not to access a specific URL path. Each `Disallow` line applies to the user-agent specified in the preceding `User-agent` line.

Example: `Disallow: /private-directory/`

3. Allow: This directive, used more rarely, tells the crawler that it can access a specific URL path even if its parent directory is disallowed. This is particularly useful for allowing specific pages within a disallowed directory.

Example: `Allow: /private-directory/public-file.html`

4. Sitemap: This directive can specify the location of the website’s sitemap, which is an XML file that lists all the URLs on a site. This helps crawlers find and index all the content on the site.

Example: `Sitemap:`

Example of a `robots.txt` File

User-agent: *

Disallow: /private-directory/

Disallow: /temporary/

Allow: /public-directory/public-file.html


How `robots.txt` is Used by Web Crawlers

1. Fetching: When a crawler visits a website, it first looks for the `robots.txt` file in the root directory. If found, it reads the file to determine which parts of the site it can and cannot access.

2. Following Directives: The crawler follows the directives in the `robots.txt` file:

  • If a path is disallowed: The crawler will not visit or index that path.
  • If a path is allowed**: The crawler will visit and index that path unless it’s overridden by another rule.

3. Crawling Efficiency: By following `robots.txt` directives, crawlers avoid wasting resources on pages that webmasters don’t want indexed, making crawling more efficient.

Limitations and Considerations

  1. Non-Compliance: Not all web crawlers comply with `robots.txt` directives. Malicious bots and some lesser-known crawlers may ignore the file.
  2. No Security Guarantee: `robots.txt` is not a security mechanism. Sensitive information should not be placed in disallowed directories expecting it to remain hidden. Use proper authentication and authorization mechanisms for security.
  3. No Impact on Existing Indexing: Disallowing a URL in `robots.txt` does not remove it from a search engine’s index if it was previously indexed. For removal, use other methods like `noindex` meta tags or search engine removal tools.
  4. Case Sensitivity: URLs in `robots.txt` are case-sensitive. Ensure that the paths match exactly the URLs on the website.

Advanced Directives

  • Crawl-Delay: This directive sets a delay (in seconds) between successive crawl requests to the server. Not all crawlers support this directive.
  • Example: `Crawl-Delay: 10`
  • Host: This directive is used to specify the preferred domain for the site if it is accessible from multiple domains. It is primarily used by Yandex.
  • Example: `Host:`


The `robots.txt` file is a powerful tool in for managing how web crawlers interact with a website. By properly configuring `robots.txt`, webmasters can guide crawlers to index the most important parts of their site while avoiding unnecessary or sensitive areas, thereby optimizing the site’s presence in search engine results and conserving server resources.