Test any URL against your robots.txt rules. See instantly if Googlebot, Bingbot, or any crawler is allowed or blocked — and exactly which rule triggered it.
— or paste robots.txt directly below —
How Robots.txt Rules Work
Matching Logic
Prefix Matching
Robots.txt uses prefix matching. Disallow: /blog/ blocks any URL starting with /blog/ — including /blog/post-1, /blog/category/seo, etc.
Rule Priority
Specificity Wins
Google uses the most specific (longest) matching path. If Allow: /admin/public/ and Disallow: /admin/ both match, the longer rule wins — allowing /admin/public/.
Wildcards
* and $ Patterns
* matches any sequence of characters. $ anchors to end of URL. Example: Disallow: /*.pdf$ blocks all PDF files.
Important
Crawl ≠ Index
Robots.txt blocks crawling, not indexing. Google can still index a blocked page via links from other sites. Use noindex meta tag to prevent indexing.
Frequently Asked Questions
A robots.txt URL tester parses your robots.txt file, identifies all user-agent blocks, and evaluates each rule against your target URL. It finds the most specific matching rule (by path length) for the given user-agent and reports whether crawling is allowed or blocked. If no rule matches, the default is allowed.
Google uses the most specific (longest) matching path to determine priority. If an Allow and Disallow rule match with equal specificity, Allow takes precedence. For example, "Allow: /page" overrides "Disallow: /" for that specific URL, because /page is more specific than /.
No. Robots.txt rules are per user-agent. You can allow Googlebot to crawl a URL while blocking other bots, or vice versa. User-agent: * rules apply to all bots not specifically addressed. A bot first looks for rules matching its specific user-agent, then falls back to wildcard rules.
"Disallow: /" blocks the entire website for the specified user-agent. It matches every URL that starts with "/" — which is all of them. To unblock specific pages, use "Allow: /specific-page" before the Disallow rule, since more specific rules take precedence.
Google and Bing support two wildcards: * matches any sequence of characters, and $ anchors to the end of a URL. For example, "Disallow: /*.pdf$" blocks all URLs ending in .pdf, and "Disallow: /search*" blocks all URLs starting with /search.
No — robots.txt blocks crawling, not indexing. Google can still index a page it has never crawled if other pages link to it. To prevent indexing, use a "noindex" meta tag or X-Robots-Tag HTTP header on the page itself.
Is Your robots.txt Accidentally Blocking Google?
We've seen SaaS companies with noindex on their homepage. 10-minute automated audit finds every crawlability issue.