Free Robots.txt Generator

Generate a robots.txt file for public sites, ecommerce stores, sitemaps, and AI crawler rules in seconds.

How It Works

  1. Select a CMS template or start from scratch
  2. Choose which bots can access your site
  3. Add paths to allow or block
  4. Add your sitemap URL(s)
  5. Copy or download your robots.txt file

CMS Templates

Start from template:
Rule Group 1

AI Bot Controls

AI Crawler Controls

Block AI training and browsing bots from accessing your site. Checked bots will have a Disallow: / rule added to the output.

# Robots.txt # Generated by Atelier Tools User-agent: *

Robots.txt file examples

Use these as starting points in the robots.txt generator, then adjust the paths for your own site before you copy or download the file.

Simple public site

Allow normal crawling, block admin/login paths, and point crawlers to your sitemap.

User-agent: *
Disallow: /admin/
Disallow: /login/
Sitemap: https://example.com/sitemap.xml

Ecommerce store

Keep product and category pages crawlable while blocking cart, checkout, account, and internal search URLs.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/
Sitemap: https://example.com/sitemap.xml

AI crawler robots.txt

Ask common AI crawlers not to request your site. Compliant bots may follow these rules, but robots.txt is not access protection.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Help & Guides

Robots.txt generator guide FAQ

Related Tools

Canonical Tag Generator

Create canonical tags to help search engines index the preferred version of each page.

FAQ Schema Generator

Generate FAQPage schema markup from question-and-answer content for richer search listings.

Open Graph Debugger

Check Open Graph tags and preview how your links appear when shared on social platforms.

Heading Structure Checker

Review page headings to spot hierarchy issues that can affect accessibility and SEO.