Robots.txt Generator

Page details

Robots.txt Generator – Build It in Your Browser

Build a robots.txt in your browser: 3 templates, Allow and Disallow rules, an optional sitemap line, then copy or download. Nothing is uploaded, no domain needed.

Build and download a valid robots.txt without leaving your browser.

Written for the crawlers that read this file before they read anything else

Nothing to install and nothing to configure. The file is assembled in the browser as you type.

Allow All opens the whole site to every crawler, Block All closes it, and Standard gives you a conventional starting point. Whichever you pick, every line stays editable.

Each rule is a user agent, an Allow or Disallow directive and a path. Add as many as the site needs, plus an optional Sitemap URL and an optional Crawl-delay. The * user agent is free; naming a specific bot or setting a crawl-delay takes a free account.

The output is plain text. Copy it to the clipboard, or download the file with a free account, then place it at the root of your site so it answers at yourdomain.com/robots.txt.

Robots.txt has a small vocabulary, and most of the confusion comes from assuming every crawler treats it the same way. This is what each line does and where it is honoured.

Names the crawler that the rules underneath it apply to. A group runs until the next User-agent line.

Required. AdsBot-Google does not match * and has to be named explicitly.

Tells the named crawler not to fetch any URL whose path starts with this value.

Google, Bing, Yandex and anything following RFC 9309

Carves an exception out of a broader Disallow so one path inside a blocked branch stays crawlable.