robots.txt Generator

SEO and marketing

What robots.txt does

A file in the site root telling crawlers where they may go. It is guidance, not a server-level lock.

Saves crawl budget A crawler spends limited time per site. If it wanders through filter and sorting pages, it may never reach what matters.

Hides utility sections Admin, cart and checkout pages have no place in search results and need no crawling.

Points to the sitemap One Sitemap line with the full URL, and engines you never submitted to will still find it.

Different rules per crawler You can allow Google everything while blocking aggressive content scrapers outright.

Mistakes that cost you traffic

Disallow: / in production A leftover line from development that blocks the entire site. Traffic vanishes within a week; the cause takes months to find.

CSS and JS blocked Google renders pages like a browser. Without CSS and JS it sees a broken layout and judges accordingly.

Trying to hide a page Disallow does not remove a URL from results; it can stay there with no description. Hiding requires noindex.

An over-broad rule Disallow: /cat blocks /catalog and /category too. One letter and a whole section disappears.

What is usually blocked

Admin area The site control panel. It has no business in search, and extra attention to it is harmful.

Cart and checkout Pages that exist only for one visitor. To a search engine they are empty.

Filters and sorting The same catalogue in hundreds of combinations. The biggest crawl-budget drain in online stores.

Internal search Internal search results generate an endless supply of near-duplicate URLs.

What robots.txt is and how it works

robots.txt is a plain text file in your site root that a search crawler reads first. It tells the bot which sections it may crawl and which to skip. The file is made of blocks: User-agent defines which crawler the rule applies to, followed by Disallow and Allow.

Core directives

User-agent: * — a rule for every crawler at once.
Disallow: /admin/ — block crawling of a section.
Allow: /admin/ajax.php — an exception inside a blocked section.
Sitemap: — the full sitemap URL; several are allowed.
Crawl-delay: — a pause between requests (ignored by Google).

The most common mistakes

The first and costliest is forgetting to remove Disallow: / after moving from a staging domain. The site disappears from search within weeks. The second is blocking pages you actually want removed from the index: the crawler stops visiting them, but URLs already indexed stay in the results. Removal requires a noindex meta tag, and the page must remain crawlable for the bot to see it. The third is blocking CSS and JS: Google cannot render the page and rates the mobile version worse.

What an online store should block

There is no point crawling cart, checkout and account pages. Filters and sorting deserve special attention: URLs like /*?sort= create thousands of near-identical pages and drain crawl budget. Block them — and turn genuinely useful filter combinations into standalone landing pages with their own copy instead.

Once uploaded, check the file in Google Search Console under Settings → robots.txt. Google will show when it was last read and whether it contains errors.

FAQ

Why do I need a robots.txt file?

robots.txt tells search crawlers which parts of the site to visit and which to skip. It saves crawl budget: the bot does not waste time on carts, user accounts or filter pages and reaches your products and articles faster. Keep in mind that robots.txt controls crawling, not indexing. To keep a page out of search results you need a noindex meta tag.

Where should robots.txt be uploaded?

Only to the domain root — example.com/robots.txt. In a subfolder (example.com/site/robots.txt) the file will not work, crawlers simply do not look there. A subdomain needs its own file: blog.example.com has a separate robots.txt.

Can I block the whole site from Google?

Yes, the "block the whole site" preset produces Disallow: / — use it on staging domains only. On a live site this is the costliest SEO mistake there is: traffic disappears within a few weeks. Always remove the directive before launch and verify the file in Google Search Console.

Does Google respect the Crawl-delay directive?

No, Google ignores it — crawl rate is configured in Search Console instead. Bing and Yandex, however, do understand Crawl-delay. It only makes sense when bots genuinely load your server; in a normal situation leave it at 0.

What we offer

NEW CASE

Let's take the first step towards your website — write to us