Allow, disallow, and sitemap in one file
robots.txt lives at the site root. Crawlers read User-agent blocks, path rules, and Sitemap URL. This form builds that text live with default disallow /admin/ and a sample sitemap URL.
Copy robots.txt writes the textarea. Reset restores defaults. Paths normalize to lead with /.
No upload. You deploy the copied file to HTTPS hosting.
SEO tools on the same site
Meta tags for pages you allow: Meta Tags Generator.
Track campaign landing URLs separately with the UTM Builder; robots.txt does not encode UTM.
Convert release notes HTML to docs Markdown with the HTML to Markdown Converter.
Common patterns
Disallow /admin/ or /api/ private zones; Allow /public/ when you need an exception inside a broad disallow (order matters to crawlers; this tool lists allows before disallows as you enter them).
Sitemap should be the full absolute URL to sitemap.xml or sitemap index. Update when you change hosts.
Staging environments often use Disallow: / for everything. Swap user-agent and rules per environment in your deploy script.
Frequently Asked Questions (FAQ)
How are allow and disallow paths formatted?
One path per line in each textarea. Lines trim whitespace; empty lines drop. Paths without a leading slash get one added. Output uses Allow: and Disallow: directives under the User-agent block.
Where does the sitemap line go?
Sitemap: URL appears after a blank line following the rules block when the sitemap field is non-empty. Standard robots.txt placement.
Can I target one crawler?
Set User-agent to Googlebot, Bingbot, or * for all. One block per form submit; duplicate the output manually for multiple agents if needed.
Does this replace Search Console?
No. Copy robots.txt to your site root and test in GSC. Pair with the Meta Tags Generator for indexable page metadata.