What we offer

NEW CASE
Antana

We create digital solutions that work for businesses

Become a clientBecome a client
Give us a call +38 (066) 35-14-529

Let's take the first step towards your website — write to us

Close
Tools

Robots.txt Tester

SEO and marketing

You may also like

Whether a crawler may fetch a specific URL, and which rule decided it.

Free audit

You’ve checked one thing. We’ll check the whole site

A tool checks one thing. We’ll go through your whole site for free and send a report: what’s holding you back in Google and what to fix first.

  • Indexing and technical errors
  • Speed and Core Web Vitals
  • Meta tags, structured data, duplicates
  • A fix plan, by priority

Report in 1–2 business days. No strings attached — if everything’s fine, we’ll say so.

By clicking the button you agree to the privacy policy

What this tester does

It fetches your robots.txt, parses the rules and shows whether a crawler may reach a given URL. Crucially, it shows which rule decided.

A verdict for one specific URL Not “the file looks fine” but an answer for the page you care about. That is where surprises turn up.
The deciding rule We show the line and its number in the file. When the result surprises you, the cause is immediately visible.
Different crawlers Googlebot and GPTBot can get different answers for the same URL when each has its own rule group.
Syntax errors Lines without a colon, rules before the first User-agent, unknown fields — each renders part of the file inert.

How rule selection works

The logic is not obvious, and every misunderstanding starts here.

A specific group wins If a crawler has its own User-agent block, the * rules do not apply to it at all. They are replaced, not combined.
The longest match wins Among matching rules the one with the longer path applies. That is why Allow: /private/public/ overrides Disallow: /private/.
Ties go to Allow When lengths are equal, Allow beats Disallow. The specification says so explicitly.
The * and $ characters An asterisk matches any sequence, a dollar marks the end of the URL. Disallow: /*.pdf$ blocks the files but not URLs with parameters after them.
An empty Disallow A Disallow line with no value means “everything is allowed”. Not an error but the conventional way to open a site fully.

What robots.txt does not do

It does not hide a page from search Disallow forbids visiting, not displaying. If links point to the page, the URL can still surface without a description. Hiding requires noindex.
It does not protect data The file is public, so a list of forbidden sections is effectively a map of where to look. Private areas need a password.
It is not binding on everyone It is a convention. Search engines respect it; content scrapers often do not.
It does not cover subdomains Every subdomain has its own robots.txt. The main domain file does not apply to shop.example.com.

What to do with a surprising result

1 Look at the rule and line number Often the path is broader than you thought: Disallow: /cat blocks /catalog and /category too.
2 Check for a specific group If the crawler has its own User-agent block, the shared rules are ignored. You have to copy them across by hand.
3 Add an exception with Allow Rather than rewriting the ban, open the subsection with its own line. The longer rule wins.
4 Re-check after editing Edits take effect immediately, but Google re-reads the file with a delay — usually within a day.

Read more on the blog

robots.txt and AI Crawlers: How to Control Access to Your Site Language models read sites through their own crawlers. Who they are, how they differ, and what to do about access — with no promises about r

FAQ

Why do Googlebot and GPTBot give different answers for one URL?
Because each may have its own rule group. If robots.txt contains a dedicated User-agent block, the shared wildcard rules do not apply at all — they are replaced, not combined. This is the most common source of confusion.
Does Disallow remove a page from search?
No. Disallow forbids visiting, not displaying. If links point to the page, its URL can still appear with no description. Removing it needs noindex — and the page must stay accessible, or the crawler never sees that tag.
What if a rule blocks more than intended?
Add an exception with a longer Allow path — the longest match wins. Disallow: /private/ together with Allow: /private/public/ closes the section while keeping the subsection open. No need to rewrite the ban.
Does robots.txt cover subdomains?
No. Every subdomain needs its own file in its own root. Rules on example.com do not reach shop.example.com — search engines treat them as separate sites.
Why paste the file when I can just give a URL?
Three cases: the site is not live yet, the server blocks automated requests, or you are editing locally and want to check before uploading. A URL check is impossible then, while the rule logic is identical.
Telegram Viber Call us