Robots.txt Generator
Generate a robots.txt file to control which crawlers can access your site.
This tool runs entirely in your browser. Your files are never uploaded to a server.
User-agent: * Allow: /
What this tool does
Creates a UTF-8 robots.txt draft for the universal user-agent group. It can add an explicit root Allow rule, validated root-relative Disallow paths, and one fully qualified HTTP or HTTPS sitemap location.
How to use it
List one path per line beginning with a slash, optionally provide the canonical sitemap URL, and review the generated text before copying it. Publish the plain-text file at /robots.txt on the exact protocol, host, and port it governs.
Worked example
With /admin/ and /search?sort= as blocked paths, the result contains two Disallow records beneath User-agent: *. A sitemap value of https://example.com/sitemap.xml is emitted separately as an absolute Sitemap record.
Validation method
The generator trims and deduplicates paths, rejects lines without a leading slash, comments, and control characters, and accepts only credential-free HTTP or HTTPS sitemap URLs without fragments. Invalid inputs are reported and omitted from copied output.
How matching works
Robots rules are case-sensitive path-prefix patterns. Under RFC 9309 the most specific matching Allow or Disallow rule wins, and Allow should win when equally specific. Test important URLs against the crawler behavior you actually target.
Crawling is not indexing or security
Disallow asks compliant crawlers not to fetch a path; it does not authenticate content and a blocked URL may still be indexed from other signals. Protect private data with access controls and use a crawlable noindex response when removal from search is the goal.
Privacy and verification
Generation runs locally and Quiklio does not upload paths or sitemap addresses. Check the deployed file returns HTTP 200 as UTF-8 plain text at the site root, then test representative URLs in the relevant search engine tools; Google and RFC 9309 guidance reviewed September 2026.
Frequently Asked Questions
- Where should robots.txt be placed?
- At the root of your domain, e.g. https://example.com/robots.txt — it won't be read from any other location.
- Does Disallow guarantee a page stays out of Google?
- No — it only blocks crawling. Use a noindex meta tag or password protection to keep pages out of search results entirely.
- Why doesn't this generator add Crawl-delay?
- Crawl-delay is not part of RFC 9309 and Google does not document it as a supported robots.txt rule. Configure crawl behavior through mechanisms documented by the specific crawler instead.