Robots.txt and SEO: the complete guide to controlling crawlers

A single misplaced line in your robots.txt file can hide your entire website from Google. It has happened to companies that accidentally left a staging Disallow rule in place after launch, to site owners who blocked their CSS files and broke Google's ability to render their pages, and to developers who confused "blocking crawling" with "blocking indexing" and ended up doing neither correctly.

Robots.txt is one of the oldest files on the web. The Robots Exclusion Protocol was created in 1994, and the basic format has barely changed. But what has changed is how much is riding on it. Google now renders JavaScript, AI crawlers check it before scraping content, and crawl budget matters more than ever for large sites.

This guide covers everything you need to know about robots.txt for SEO: how the file works, what each directive does, how to manage AI crawlers, and the mistakes that cost rankings.

What is robots.txt?

Robots.txt is a plain text file that sits at the root of your website and tells web crawlers which pages they can and cannot access. When a crawler like Googlebot arrives at your site, the first thing it does is look for this file at example.com/robots.txt. If the file exists, the crawler reads the rules before visiting any other page.

The file follows the Robots Exclusion Protocol (REP), a standard created in 1994 that every major search engine respects. Google formalized its interpretation of the protocol as an internet standard (RFC 9309) in 2022, which clarified how it handles edge cases like conflicting rules and malformed files.

Two things robots.txt is not:

• It is not a security measure. The file is publicly accessible, and any bot can read it. Malicious crawlers can (and do) ignore it entirely. Never rely on robots.txt to protect sensitive content. Use proper authentication instead.
• It is not an indexing control. Blocking a URL in robots.txt prevents crawling, not indexing. If other pages link to a blocked URL, Google can still index it and show it in search results, just without a proper snippet.

How robots.txt works

When a crawler visits your site, it follows this sequence:

1. The crawler requests example.com/robots.txt before doing anything else.
2. If the file returns a 200 status code, the crawler reads and follows the rules inside it.
3. If the file returns a 404 (not found), the crawler assumes everything is allowed and crawls the entire site freely.
4. If the file returns a 5xx server error, the crawler treats it as a temporary problem and typically pauses crawling to be safe. Google will retry later.

Scope rules

Robots.txt only applies to the exact protocol and domain where it is hosted. This means:

• https://example.com/robots.txt controls crawling for https://example.com only.
• It does not apply to http://example.com (different protocol).
• It does not apply to https://blog.example.com (different subdomain).
• It does not apply to https://example.com:8080 (different port).

If you run multiple subdomains, each one needs its own robots.txt file. If you only use HTTPS (which you should), you generally only need one file on the HTTPS version.

Robots.txt syntax and directives

A robots.txt file is made up of one or more rule groups. Each group starts with a User-agent line that specifies which crawler the rules apply to, followed by one or more Disallow or Allow lines.

User-agent

This specifies which crawler the following rules apply to. Use * as a wildcard to target all crawlers:

User-agent: *
Disallow: /admin/

To target a specific crawler, use its name:

User-agent: Googlebot
Disallow: /internal-search/

Common user-agent names include Googlebot (Google search), Bingbot (Bing), Googlebot-Image (Google Images), and Yandex.

Disallow

Tells the crawler not to access the specified path. The path is relative to the root of the site:

Disallow: /private/
Disallow: /temp/old-page.html

An empty Disallow: with no path means "disallow nothing," which allows everything. A Disallow: / blocks the entire site.

Allow

Permits crawling of a specific path within a broader Disallow rule. This is useful when you want to block a directory but allow specific files within it:

User-agent: *
Disallow: /account/
Allow: /account/public-profile/

When a URL matches both an Allow and a Disallow rule, Google uses the most specific match (the longest path). If both paths are the same length, the Allow rule wins.

Crawl-delay

Some crawlers (like Bingbot and Yandex) support a Crawl-delay directive that specifies how many seconds to wait between requests. Google does not support this directive. To control Googlebot's crawl rate, use the crawl rate settings in Google Search Console.

Formatting rules

• Each directive goes on its own line.
• Lines starting with # are comments and are ignored by crawlers.
• Blank lines separate rule groups for different user-agents.
• The file must be plain text (UTF-8) and named exactly robots.txt in lowercase.
• The file size limit is 500 kibibytes (roughly 512 KB). Rules beyond this limit may be ignored.

Wildcards and pattern matching

Google and Bing support two wildcard characters that make rules more flexible:

Asterisk (*)

Matches any sequence of characters. Useful for blocking URL patterns:

# Block all URLs containing "?sort="
Disallow: /*?sort=

# Block all PDF files
Disallow: /*.pdf$

Dollar sign ($)

Marks the end of a URL. Without it, Disallow: /page blocks /page, /page/, /page-two, and /page/anything. With it:

# Only blocks /page exactly, not /page/about or /page-two
Disallow: /page$

Practical wildcard examples

# Block all URLs with query parameters
Disallow: /*?

# Block all URLs with a specific parameter regardless of position
Disallow: /*utm_

# Block faceted navigation URLs
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?sort=
Disallow: /*?filter=

Be careful with wildcards. A rule like Disallow: /*page blocks every URL containing the word "page" anywhere, including /homepage, /landing-page-design, and /about-our-page. Test wildcard rules thoroughly before deploying them.

Robots.txt and crawl budget

Every website gets a limited number of pages that Googlebot will crawl in a given session. This is your crawl budget, and robots.txt is one of the primary tools for managing it.

For small sites (under a few thousand pages), crawl budget is rarely a concern. Google can typically crawl the entire site without hitting limits. But for larger sites, wasting crawl budget on low-value pages means your important pages get crawled less frequently, which delays indexing and ranking updates.

Pages that waste crawl budget

• Internal search result pages: /search?q=shoes generates infinite URL combinations.
• Faceted navigation: /products?color=red&size=large&sort=price creates thousands of near-duplicate pages.
• Calendar pages: /events/2027/01/ generates infinite future-dated URLs.
• Session ID URLs: /page?sessionid=abc123 creates a new URL for every visitor.
• Print or PDF versions: /article/print/ duplicates existing content.
• Pagination beyond a reasonable depth: /blog/page/247/

Blocking these patterns in robots.txt frees up crawl budget for the pages that actually need to rank: your product pages, blog posts, service pages, and landing pages.

What to block (and what not to)

Safe to block

• Admin and backend pages: /admin/, /wp-admin/, /cms/
• Internal search results: /search, /*?s=, /*?q=
• Shopping cart and checkout: /cart/, /checkout/
• User account pages: /my-account/, /dashboard/
• Staging or testing paths: /staging/, /test/
• Infinite URL generators: calendar widgets, faceted navigation parameters, tag combinations

Never block

• CSS and JavaScript files: Google needs to render your pages to understand them. Blocking /assets/css/ or /assets/js/ breaks rendering and can drop your rankings. This was a common mistake years ago when Google could not execute JavaScript. It can now, and it needs full access to your resources.
• Images you want indexed: If you block image directories, your images will not appear in Google Images search. Only block images that are genuinely private.
• Pages with noindex tags: If a page has a <meta name="robots" content="noindex"> tag, the crawler must be able to access the page to read and obey that tag. Blocking a page in robots.txt and adding noindex is contradictory. The crawler cannot see the noindex tag if it cannot crawl the page.
• Your sitemap: Keep your XML sitemap accessible to all crawlers.

Robots.txt vs. noindex: understanding the difference

This is the single most misunderstood concept in robots.txt management. The difference is straightforward but critical:

• Robots.txt Disallow = "Don't visit this page." The crawler never downloads the content.
• Noindex = "Visit this page but don't put it in the search index." The crawler downloads the content, reads the noindex tag, and removes the page from search results.

If you want a page to disappear from search results, use noindex. If you want to save crawl budget by preventing the crawler from visiting a page that has no SEO value, use robots.txt. If you use both on the same page, the crawler cannot see the noindex tag (because robots.txt blocks the visit), and the page may still appear in search results based on external links.

When to use which

Goal	Use
Remove page from search results	Noindex meta tag
Save crawl budget on junk URLs	Robots.txt Disallow
Block crawling AND prevent indexing	Noindex (remove from robots.txt)
Consolidate duplicate content	Canonical tag
Permanently move a URL	301 redirect

The Sitemap directive

Robots.txt can include a Sitemap directive that points crawlers to your XML sitemap. This line can appear anywhere in the file and applies to all crawlers regardless of the User-agent groups above it:

Sitemap: https://example.com/sitemap.xml

Important rules for the Sitemap directive:

• Use the full URL including the protocol (https://). Relative paths do not work.
• The directive is case-insensitive (Sitemap: and sitemap: both work, but the URL itself is case-sensitive).
• You can list multiple sitemaps if you split them by content type or section.
• The sitemap URL does not have to be on the same domain as the robots.txt file. You can point to a sitemap hosted on a CDN or a different subdomain.

While Google discovers sitemaps through Search Console and other signals, including the Sitemap directive in robots.txt is a good practice. It helps other search engines and crawlers find your sitemap automatically.

Managing AI crawlers with robots.txt

In 2026, robots.txt is the primary mechanism for controlling whether AI companies can access your content. Most AI companies have introduced specific user-agent strings for their crawlers, and they respect robots.txt rules.

Major AI crawler user-agents

• GPTBot (OpenAI): Used for browsing and training. Block with User-agent: GPTBot.
• OAI-SearchBot (OpenAI): ChatGPT search specifically. Blocking GPTBot does not block this one.
• ClaudeBot (Anthropic): Claude's web crawler.
• PerplexityBot (Perplexity): Powers Perplexity's search answers.
• Google-Extended (Google): Controls Gemini/AI Overview training but does not affect Google Search crawling.
• Bytespider (ByteDance): TikTok's parent company crawler.
• CCBot (Common Crawl): Open dataset used by many AI companies for training.

Strategy: allow AI search, block AI training

Many site owners want their content to appear in AI-powered search answers but do not want it used for model training. The approach is to allow search-specific crawlers while blocking training crawlers:

# Allow search engine crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Check the llms.txt specification for an additional way to provide AI-friendly content summaries alongside robots.txt controls.

For a broader strategy on optimizing for AI search engines, see the Generative Engine Optimization service page.

Real-world robots.txt examples

Simple blog or portfolio site

User-agent: *
Disallow: /admin/
Disallow: /drafts/

Sitemap: https://example.com/sitemap.xml

Small sites usually need minimal rules. Allow everything except admin and draft pages.

WordPress site

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /*?s=
Disallow: /tag/*
Disallow: /author/*
Disallow: /*?replytocom=

Sitemap: https://example.com/sitemap_index.xml

The Allow: /wp-admin/admin-ajax.php line is important because many WordPress themes and plugins use AJAX calls that Google needs to render the page correctly. Blocking tag archives and author pages prevents thin content from wasting crawl budget. If you use Rank Math or Yoast, these plugins can manage your robots.txt for you.

E-commerce site

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /wishlist/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?min_price=
Disallow: /*?max_price=
Disallow: /search
Disallow: /compare/
Allow: /

Sitemap: https://example.com/sitemap.xml

E-commerce sites generate the most crawl budget waste through faceted navigation. Blocking filter and sort parameters keeps Google focused on your actual product and category pages.

Cloudflare Workers site (this site)

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://bikeshtamang.com.np/sitemap.xml

This site is a static site deployed on Cloudflare Workers. Since there are no dynamic URLs, faceted navigation, or internal search, the rules are simple: allow everything for search engines, block AI training crawlers, point to the sitemap.

How to test your robots.txt

Google Search Console

The Robots.txt Tester in Google Search Console lets you paste a URL and see whether Googlebot is allowed or blocked. It also highlights syntax errors and warnings in your file. This is the most reliable way to verify your rules work as intended.

Check the live file

Visit yourdomain.com/robots.txt in your browser. If you see the file contents, it is working. If you get a 404, no robots.txt exists and crawlers will access everything. If you get a 5xx error, fix your server configuration because some crawlers pause crawling entirely when the robots.txt returns a server error.

Validate syntax

Look for common formatting issues:

• Each directive must be on its own line.
• No extra spaces between the directive name and the colon.
• Paths are case-sensitive: /Admin/ and /admin/ are different rules.
• Make sure rule groups are separated by blank lines.
• Verify the file is served with a Content-Type: text/plain header.

Monitor crawl behavior

After making changes, watch the Crawl Stats report in Search Console over the next few weeks. You should see Googlebot stop requesting the paths you blocked and increase crawling on the paths you left open. If Googlebot is still hitting blocked paths, double-check your rules for typos or wildcard issues.

8 robots.txt mistakes that hurt SEO

1. Leaving staging Disallow rules in production

Staging sites typically have Disallow: / to prevent Google from indexing test content. When the site launches, this rule must be removed. Forgetting to do so blocks your entire live site from search engines. It is one of the most common (and most damaging) robots.txt errors. Always include a robots.txt review in your launch checklist.

2. Blocking CSS and JavaScript

Years ago, SEOs recommended blocking resource files. Google now explicitly warns against this. Googlebot renders pages using CSS and JavaScript, and blocking these files means Google sees a broken page. This can directly hurt your rankings and trigger warnings in Search Console.

3. Using robots.txt to hide pages from search results

Disallow does not equal deindex. If you want a page out of Google's index, use a noindex meta tag. Using robots.txt to "hide" a page just means Google will show the URL in results without a snippet, which looks worse than not appearing at all.

4. Blocking pages that have noindex tags

If you add noindex to a page and then also block it in robots.txt, the crawler cannot reach the page to read the noindex tag. The result: the page stays in the index with no snippet. Remove the Disallow rule so Google can crawl the page, read the noindex tag, and actually remove it from search results.

5. Overly broad wildcard rules

A rule like Disallow: /*page blocks every URL containing "page" anywhere in the path. That includes your homepage (/ typically redirects, but paths like /landing-page/, /pricing-page/, and /service-page/ are blocked). Always test wildcard rules against your actual URL structure before deploying.

6. Conflicting rules across user-agent groups

When you have separate blocks for User-agent: * and User-agent: Googlebot, only the most specific group applies. If your Googlebot block has no Disallow rules, Googlebot ignores everything in the * block. This catches people who add a Googlebot-specific rule but forget to include their general Disallow rules in that same block.

7. Using the wrong path format

Disallow: blog/ (without the leading slash) does not work. Paths must start with /. Similarly, Disallow: /blog blocks both /blog and /blog-post and /blog/anything. If you only want to block the /blog/ directory, use Disallow: /blog/ with the trailing slash.

8. Never updating the file after site changes

Robots.txt is not "set it and forget it." When you redesign your site, change URL structures, add new sections, or migrate platforms, your robots.txt rules need to be reviewed. Old rules may block new URLs or allow paths that no longer exist. Include robots.txt in your regular SEO audit checklist.

Frequently asked questions

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website that tells web crawlers which pages or sections they are allowed to access and which ones they should skip. It follows the Robots Exclusion Protocol, a standard that search engine crawlers like Googlebot and Bingbot voluntarily follow. The file does not force compliance. Malicious bots can ignore it entirely.

Does robots.txt block pages from appearing in Google?

No. Disallowing a URL in robots.txt prevents Google from crawling it, but the URL can still appear in search results if other pages link to it. Google will show the URL with a "No information is available for this page" message. To truly prevent a page from appearing in search results, use a noindex meta tag instead.

Where do I put the robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at example.com/robots.txt. It only applies to the exact protocol and domain where it is hosted. A robots.txt file at https://example.com/robots.txt does not apply to http://example.com or https://shop.example.com. Each subdomain and protocol variation needs its own file.

Should I block AI crawlers in robots.txt?

It depends on your goals. If you want your content to appear in AI-generated answers, allow their crawlers. If you want to prevent content from being used for model training while still appearing in AI search results, selectively block training crawlers (like GPTBot) while allowing search-focused crawlers (like OAI-SearchBot). Many AI companies now separate their search crawlers from their training crawlers.

Need help getting your robots.txt and crawl configuration right? I offer technical SEO services that include robots.txt auditing, crawl budget optimization, and indexing controls. Or start with an SEO audit to identify crawling and indexing issues across your entire site. Get in touch to discuss your project.