Free Tool
Set what crawlers can access, add your sitemap, and get a valid robots.txt file — live preview, one-click copy, and a real download.
Honored by Bing and some other crawlers — Google ignores this directive entirely.
User-agent: * Allow: /
Upload the downloaded file to your site's root (e.g. https://yoursite.com/robots.txt) — it has no effect anywhere else.
It matters, but it's a small part of whether a site actually gets crawled, indexed, and ranked. Arcetis's Technical–Topical–Trust Model covers the rest of that infrastructure layer.
Book a Discovery Call