robots.txt Generator

What robots.txt is and how it works

robots.txt is a plain text file in your site root that a search crawler reads first. It tells the bot which sections it may crawl and which to skip. The file is made of blocks: User-agent defines which crawler the rule applies to, followed by Disallow and Allow.

Core directives

• User-agent: * — a rule for every crawler at once.
• Disallow: /admin/ — block crawling of a section.
• Allow: /admin/ajax.php — an exception inside a blocked section.
• Sitemap: — the full sitemap URL; several are allowed.
• Crawl-delay: — a pause between requests (ignored by Google).

The most common mistakes

The first and costliest is forgetting to remove Disallow: / after moving from a staging domain. The site disappears from search within weeks. The second is blocking pages you actually want removed from the index: the crawler stops visiting them, but URLs already indexed stay in the results. Removal requires a noindex meta tag, and the page must remain crawlable for the bot to see it. The third is blocking CSS and JS: Google cannot render the page and rates the mobile version worse.

What an online store should block

There is no point crawling cart, checkout and account pages. Filters and sorting deserve special attention: URLs like /*?sort= create thousands of near-identical pages and drain crawl budget. Block them — and turn genuinely useful filter combinations into standalone landing pages with their own copy instead.

Once uploaded, check the file in Google Search Console under Settings → robots.txt. Google will show when it was last read and whether it contains errors.