User-agent rules
Preview
User-agent: * Disallow: /admin
Compose user-agent rules and a sitemap, then copy or download robots.txt.
User-agent: * Disallow: /admin
Your vote is saved on this device only.
The robots.txt Generator builds a valid robots.txt file from structured input instead of hand-typed directives. You define one or more user-agent blocks, list Allow and Disallow paths one per line, and add a single Sitemap URL. The output updates live in a preview pane, so you can see exactly what crawlers will read and avoid typos that silently block or expose pages.
It is built for developers, SEOs, and site owners who want a quick, correct file without memorizing syntax. Everything runs in your browser: rules are assembled client-side, nothing is uploaded, and no account is needed. It is free to use, and you can copy the result to your clipboard or download a ready-to-deploy robots.txt in one click.
No. The file is assembled entirely in your browser, nothing is uploaded, and you do not need an account. It is free to use.
Use * as the User-agent and add the directory path, for example /admin, on its own line in the Disallow field.
Yes. Click Add rule and set the User-agent to a name like Googlebot or Bingbot, then give that block its own Allow and Disallow paths.
The tool outputs a single Sitemap line. To reference multiple sitemaps, point the URL at a sitemap index file that lists them.
Upload the robots.txt to the root of your domain so it is reachable at https://yourdomain.com/robots.txt, which is where crawlers look for it.