Robots.txt Checker
Check whether a specific path is allowed or blocked in a site's robots.txt.
What this tool does
Fetches a site's robots.txt file, parses its rule groups, and checks whether a specific path is allowed or blocked for a chosen user agent.
How to use it
Enter a site's URL, the path you want to check (defaults to /), and a user agent (defaults to *, matching any crawler with no dedicated group). The checker fetches /robots.txt from that site, finds the matching rule group, and reports whether the path is allowed or blocked and which rule decided it.
Example
For a site whose robots.txt disallows /admin/ but allows /admin/public/, checking /admin/public/page for * returns allowed, matched by the more specific Allow: /admin/public/ rule rather than the broader Disallow: /admin/.
Validation and safety
Rule matching follows the published robots.txt convention: the longest matching Allow or Disallow pattern wins, ties go to Allow, * matches any character sequence, and a trailing $ anchors the match to the end of the path. A site with no reachable robots.txt is treated as allowing everything, which is the standard default.
How to interpret the result
This shows what the rules say, not what every crawler actually does with them — well-behaved crawlers like major search engines follow robots.txt, but it is a voluntary convention, not an access control mechanism, and it does not stop a page from being accessed directly if its URL is known.
Limitations
Only the User-agent, Allow, Disallow, Crawl-delay and Sitemap directives are parsed; the group is chosen by an exact user-agent match or the * fallback, not by matching the longest partial product-token name the way some crawlers do for closely related agent strings.
Privacy
This tool sends the address you enter to Quiklio's own server, which makes the network request on your behalf — the target address is not sent to any third party beyond what a normal request to that address would reveal.
Frequently Asked Questions
- What does Robots.txt Checker do?
- Fetches a site's robots.txt file, parses its rule groups, and checks whether a specific path is allowed or blocked for a chosen user agent.
- Can this be used to probe internal or private addresses?
- Yes, this is blocked. Requests to localhost, private IP ranges and cloud metadata addresses are refused before any check runs, and the resolved IP is re-checked after DNS lookup to prevent bypassing this with a hostname that points to a private address at request time.
- Is the checked address shared with anyone else?
- This tool sends the address you enter to Quiklio's own server, which makes the network request on your behalf — the target address is not sent to any third party beyond what a normal request to that address would reveal.