Simple public site
Allow normal crawling, block admin/login paths, and point crawlers to your sitemap.
User-agent: *
Disallow: /admin/
Disallow: /login/
Sitemap: https://example.com/sitemap.xml
Generate a robots.txt file for public sites, ecommerce stores, sitemaps, and AI crawler rules in seconds.
Block AI training and browsing bots from accessing your site. Checked bots will have a Disallow: / rule added to the output.
Use these as starting points in the robots.txt generator, then adjust the paths for your own site before you copy or download the file.