15+ AI crawlers respect robots.txt differently. Generate one that explicitly handles them all. 0 signup, 100% client-side.

Your domain (optional, for sitemap line) Choose how to treat each crawler

🟢 Major AI crawlers

🟡 Secondary AI / search

🟠 Aggressive scrapers (often ignore robots.txt)

Your robots.txt

🔍 What's next?

You've told bots how to behave. Now check if they're actually behaving — and what other AI-readiness signals your site is missing.

Run Free AI-Readiness Audit (30 sec) →

Audit checks llms.txt, ai-plugin.json, JSON-LD, OpenAPI, hreflang, and 10+ more signals.

Why this tool?

In 2026, your site is crawled by dozens of AI bots — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, ByteSpider (TikTok), and a long tail of less polite scrapers.

Each respects (or ignores) different signals.
Most don't respect a blanket User-agent: *.
If you want some bots in and others out, you need explicit per-bot rules.

This tool generates a clean robots.txt with the 15+ most-relevant 2026 crawlers. Pick allow/block per bot. Add a sitemap line. Copy or download. Done in 30 seconds.

Then what?

robots.txt is one of 11+ signals AI agents use. Want to see the full picture? Run the free audit →

🤖 robots.txt generator for AI bots