robots.txt Generator

SEO and marketing

You’ve checked one thing. We’ll check the whole site

A tool checks one thing. We’ll go through your whole site for free and send a report: what’s holding you back in Google and what to fix first.

Indexing and technical errors
Speed and Core Web Vitals
Meta tags, structured data, duplicates
A fix plan, by priority

Learn more: Website SEO audit

What robots.txt does

A file in the site root telling crawlers where they may go. It is guidance, not a server-level lock.

Saves crawl budget A crawler spends limited time per site. If it wanders through filter and sorting pages, it may never reach what matters.

Hides utility sections Admin, cart and checkout pages have no place in search results and need no crawling.

Points to the sitemap One Sitemap line with the full URL, and engines you never submitted to will still find it.

Different rules per crawler You can allow Google everything while blocking aggressive content scrapers outright.

Mistakes that cost you traffic

Disallow: / in production A leftover line from development that blocks the entire site. Traffic vanishes within a week; the cause takes months to find.

CSS and JS blocked Google renders pages like a browser. Without CSS and JS it sees a broken layout and judges accordingly.

Trying to hide a page Disallow does not remove a URL from results; it can stay there with no description. Hiding requires noindex.

An over-broad rule Disallow: /cat blocks /catalog and /category too. One letter and a whole section disappears.

What is usually blocked

Admin area The site control panel. It has no business in search, and extra attention to it is harmful.

Cart and checkout Pages that exist only for one visitor. To a search engine they are empty.

Filters and sorting The same catalogue in hundreds of combinations. The biggest crawl-budget drain in online stores.

Internal search Internal search results generate an endless supply of near-duplicate URLs.

What robots.txt is and how it works

robots.txt is a plain text file in your site root that a search crawler reads first. It tells the bot which sections it may crawl and which to skip. The file is made of blocks: User-agent defines which crawler the rule applies to, followed by Disallow and Allow.

Core directives

User-agent: * — a rule for every crawler at once.
Disallow: /admin/ — block crawling of a section.
Allow: /admin/ajax.php — an exception inside a blocked section.
Sitemap: — the full sitemap URL; several are allowed.
Crawl-delay: — a pause between requests (ignored by Google).

The most common mistakes

The first and costliest is forgetting to remove Disallow: / after moving from a staging domain. The site disappears from search within weeks. The second is blocking pages you actually want removed from the index: the crawler stops visiting them, but URLs already indexed stay in the results. Removal requires a noindex meta tag, and the page must remain crawlable for the bot to see it. The third is blocking CSS and JS: Google cannot render the page and rates the mobile version worse.

What an online store should block

There is no point crawling cart, checkout and account pages. Filters and sorting deserve special attention: URLs like /*?sort= create thousands of near-identical pages and drain crawl budget. Block them — and turn genuinely useful filter combinations into standalone landing pages with their own copy instead.

Once uploaded, check the file in Google Search Console under Settings → robots.txt. Google will show when it was last read and whether it contains errors.

Rate this tool

It tells us what to improve first

Be the first to rate 5.0 of 5 0 votes

FAQ

Why do I need a robots.txt file?

robots.txt tells search crawlers which parts of the site to visit and which to skip. It saves crawl budget: the bot does not waste time on carts, user accounts or filter pages and reaches your products and articles faster. Keep in mind that robots.txt controls crawling, not indexing. To keep a page out of search results you need a noindex meta tag.

Where should robots.txt be uploaded?

Only to the domain root — example.com/robots.txt. In a subfolder (example.com/site/robots.txt) the file will not work, crawlers simply do not look there. A subdomain needs its own file: blog.example.com has a separate robots.txt.

Can I block the whole site from Google?

Yes, the "block the whole site" preset produces Disallow: / — use it on staging domains only. On a live site this is the costliest SEO mistake there is: traffic disappears within a few weeks. Always remove the directive before launch and verify the file in Google Search Console.

Does Google respect the Crawl-delay directive?

No, Google ignores it — crawl rate is configured in Search Console instead. Bing and Yandex, however, do understand Crawl-delay. It only makes sense when bots genuinely load your server; in a normal situation leave it at 0.

How do I create a robots.txt file online?

Pick a preset — WordPress, OpenCart, online store or fully open site — and enter your domain: a Sitemap line pointing to https://your-domain/sitemap.xml is added automatically unless you list another sitemap. Add your own Disallow and Allow rules if needed, one per line. The file builds as you type, and the “Download” button saves a ready robots.txt — all that is left is to upload it to the site root.

Should I block AI bots and SEO crawlers?

These are two separate checkboxes. The first denies access to GPTBot, CCBot, ClaudeBot, anthropic-ai and Google-Extended — the agents through which site content is collected for AI; regular Googlebot is unaffected. The second blocks AhrefsBot, SemrushBot, MJ12bot, DotBot and BLEXBot: less server load, but data about your site in those services may stop updating. robots.txt is a request: well-behaved bots follow it, others ignore it.

What we offer

NEW CASE

Let's take the first step towards your website — write to us