Robots.txt & Crawl Directives Template

Standard Production Site

A normal live site that wants full crawling plus a sitemap pointer.

Use when: you run a healthy production site and want search engines to crawl everything freely while finding your sitemap fast.

Directives

• User-agent: *
• Disallow: (leave the value blank to allow all)
• Sitemap: [https://example.com/sitemap.xml]

An empty Disallow means nothing is blocked. You do not need an Allow line for this; allow-by-default is the standard behavior.

Optional hygiene

• Disallow: [/wp-admin/] (block a real admin path if you have one)
• Allow: [/wp-admin/admin-ajax.php]

Note: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and add a noindex meta tag, or protect it behind a login. Place the file at the domain root: [https://example.com/robots.txt].

Staging, Dev or Pre-Launch Site

A non-public environment you must keep entirely out of search.

Use when: the site is a staging server, development copy, or pre-launch build that no one should find in search.

Directives

• User-agent: *
• Disallow: /

A single Disallow: / tells compliant crawlers to skip every URL on the host. This is intentional here and dangerous everywhere else.

Critical warning: never let this line reach production. The most common deindexing disaster is a Disallow: / copied from staging to the live site during deployment.

Stronger protection

• Robots.txt is a request, not a lock. Rogue crawlers ignore it.
• Add HTTP authentication (a username and password) so the environment is truly private.
• If you add a site-wide noindex as a backup, don't also block the page with Disallow: /: a blocked page never reveals its noindex tag.

Note: if a staging URL was already crawled, do not rely on Disallow: / to remove it. Blocking the URL also blocks Google from seeing a noindex tag. Allow crawling temporarily with noindex, or use the removal tools in Search Console.

E-commerce Crawl Control

A store that needs to steer crawl budget away from low-value URLs.

Use when: a store generates endless faceted, parameter, and account URLs that waste crawl budget and dilute signals.

Typical blocks

• User-agent: *
• Disallow: [/cart]
• Disallow: [/checkout]
• Disallow: [/account/]
• Disallow: [/search] (internal site search results)
• Disallow: [/*?*sort=] (a sort parameter anywhere in the query string)
• Disallow: [/*?*filter=] (faceted filters)
• Disallow: [/*.pdf$] (block any URL ending in .pdf: $ anchors the end)

Use * to match any sequence and $ to anchor the end of a URL. Paths are case-sensitive, so [/Search] and [/search] are different rules.

Caution: only block parameters that create true duplicates or thin pages. Blocking a parameter that changes the main product content (or your canonical URLs) can hide pages you want ranked. Prefer canonical tags for duplicate content and reserve robots.txt for crawl waste you never want fetched.

Allow Search Engines, Block Scrapers

Welcome major search bots while turning away aggressive or unwanted crawlers.

Use when: you want Google, Bing, and other major engines to crawl freely but want to discourage known scrapers and bandwidth-heavy bots.

Allow the engines you want

• User-agent: Googlebot
• Disallow: (blank)
• User-agent: Bingbot
• Disallow: (blank)

Discourage a specific crawler

• User-agent: [BadBot]
• Disallow: /

Each User-agent block is matched independently, and a bot follows the most specific group that names it. Keep a final User-agent: * group so unnamed bots still get clear instructions.

To curb compliant AI crawlers, name them too: for example User-agent: GPTBot with Disallow: /. This only stops crawlers that choose to honor robots.txt.

Reality check: robots.txt is voluntary. Well-behaved engines obey it; malicious scrapers and many AI crawlers ignore it and can even use it to find paths you named. For real enforcement, block by user-agent or IP at the server or firewall, or use rate limiting. Treat robots.txt as guidance, not security.

Sitemap & Multiple-Sitemap Declaration

Point crawlers to one or several sitemaps, including a sitemap index.

Use when: you want every crawler to discover your sitemaps, especially a large site split across multiple files.

Single sitemap

• Sitemap: [https://example.com/sitemap.xml]

Multiple sitemaps

• Sitemap: [https://example.com/sitemap-posts.xml]
• Sitemap: [https://example.com/sitemap-products.xml]
• Sitemap: [https://example.com/sitemap-images.xml]

Or one sitemap index

• Sitemap: [https://example.com/sitemap_index.xml]

Rules to follow:

• List each sitemap on its own line. There's no cap on the number of Sitemap lines, but each individual sitemap file is limited to 50,000 URLs and 50MB uncompressed: split larger sites across multiple files or a sitemap index.
• Always use a full absolute URL with the correct protocol (https) and host.
• The Sitemap directive is independent of User-agent groups, so place it anywhere in the file.
• Submitting sitemaps in Search Console and Bing Webmaster Tools is still recommended as a backup.

Pre-Publish QA Checklist

Catch the mistakes that quietly deindex a site before you ship the file.

Use when: you are about to publish or replace a robots.txt file and want to avoid the classic, costly errors.

Check each item

1. Confirm the file is at the root: [https://example.com/robots.txt]. A file in a subfolder is ignored.
2. Make sure there is no stray Disallow: / left over from staging.
3. Verify casing on every path; robots.txt is case-sensitive.
4. Check that CSS, JS, and image folders are not blocked, so Google can render pages.
5. Confirm each Sitemap line uses a full https URL that returns a 200.
6. Watch for Disallow: rules that are broader than intended (a trailing slash matters).
7. Remember that robots.txt does not deindex; pair blocks with noindex or removal tools where needed.
8. Re-test after any CMS or plugin update, which can overwrite the file.

Final test: fetch [https://example.com/robots.txt] in a browser to confirm it loads, then validate it with a robots.txt tester before and after going live.

Does Disallow remove a page from Google?

No. Disallow stops compliant crawlers from fetching the page, but a disallowed URL can still be indexed if other pages link to it. To keep a page out of the index, allow crawling and add a noindex directive, or protect it behind authentication.

Where does the robots.txt file have to live?

At the root of each host, so it resolves at https://yourdomain.com/robots.txt. A robots.txt placed in a subfolder is ignored. Each subdomain and protocol is treated separately, so https, http, and www versions may each need their own file.

Is robots.txt case-sensitive?

The path values are. Disallow: /Folder/ and Disallow: /folder/ match different URLs, so copy paths exactly as they appear in your site. Directive names like User-agent and Disallow are not case-sensitive, but matching your real URL casing is what matters.

What is the difference between Disallow and noindex?

Disallow controls crawling: it asks bots not to fetch a URL. Noindex controls indexing: it asks search engines not to show a page in results. If you Disallow a page, Google cannot see its noindex tag, so to deindex a page you must allow crawling and use noindex.

How do I block one bot but allow the others?

Give that bot its own group, for example User-agent: BadBot followed by Disallow: /, and keep a separate User-agent: * group for everyone else. Each bot follows the most specific group that names it, so the targeted bot is blocked while others stay allowed.

Will robots.txt stop scrapers and AI crawlers?

Only the ones that choose to obey it. Robots.txt is voluntary, so reputable search engines follow it while many scrapers and bots ignore it entirely. For real enforcement, block by user-agent or IP at the server or firewall level, or add rate limiting and authentication.