Rules
Presets
robots.txt
Generate allow and disallow rules for one or more user agents
Add sitemap hints for crawler discovery
Start from common presets such as allow all, block all, and standard blog rules
Copy the finished robots.txt file for deployment
How it works
Build a robots.txt file to control crawler access:
- User-agent — which crawler the rule applies to (* = all)
- Allow/Disallow — permit or block specific paths
- Sitemap — tell crawlers where your sitemap is
Want to learn more?
Read the complete guide with examples and tips
Related seo tools
Continue with nearby tasks in the seo tools hub.
SEO
Sitemap XML Generator
Generate XML sitemaps for search engines
Open tool
SEO
Canonical URL Builder
Generate canonical URL tags to prevent duplicate content
Open tool
SEO
Redirect Chain Checker
Check URL redirects and redirect chains
Open tool
SEO
Hreflang Tag Generator
Generate hreflang tags for multilingual websites
Open tool
SEO
Meta Tag Generator
Generate HTML meta tags for SEO
Open tool
SEO
Schema.org Generator
Generate JSON-LD structured data for rich results
Open tool