Robots.txt Generator

Build a valid robots.txt with user-agent rules, sitemap links, and crawl directives.

Quick Presets

User-Agent Rules

Sitemaps

One sitemap URL per line

Additional Settings

Host (optional)

0 rules Ready

AI Search Insights for Robots.txt

⚡

AI bots have their own user-agents

GPTBot (OpenAI), Google-Extended (Gemini), Anthropic-AI (Claude), and CCBot (Common Crawl) each have dedicated user-agent strings. You can selectively block AI training crawlers while keeping search bots allowed — granular control matters.

✓

Blocking AI crawlers ≠ blocking AI citations

If your content is already indexed by search engines, AI answer engines can still cite it via search APIs (like Google's Grounding API). Robots.txt controls crawling, not citation. To actually control AI usage, combine robots.txt with meta robots and data-noai headers.

ℹ

Sitemap lines help AI engines too

AI crawlers read the Sitemap directive in robots.txt to discover content efficiently. Listing your sitemaps here is a low-effort signal that improves discoverability across both traditional search and AI-powered engines.

⚡

Crawl-delay is respected differently

Google ignores Crawl-delay (use Search Console instead), but Bing, Yandex, and most AI crawlers honor it. If your server is under load from AI bots, adding a crawl-delay for their user-agents is a practical throttle.

Robots.txt controls who crawls your site. PatchMySEO tracks who actually cites it. Try free audit →

What this robots.txt generator does

Choose which crawlers and paths to allow or block, and this free tool writes a valid robots.txt file — with your sitemap line — ready to upload to your site's root.

How robots.txt works

Location: it must live at yourdomain.com/robots.txt — crawlers only look there.
Allow / Disallow: rules control which paths a named User-agent may crawl.
Not for secrets: robots.txt is public and only requests crawling behaviour — use noindex or auth to actually hide pages.
Add your sitemap: include a Sitemap: line so search engines find all your URLs.

Frequently asked questions

What is a robots.txt file?

A robots.txt file at your site's root tells search-engine crawlers which paths they may or may not crawl. This tool builds a valid one from your choices.

Does robots.txt stop a page from being indexed?

No. It controls crawling, not indexing — a blocked page can still be indexed if it's linked elsewhere. To keep a page out of results, use a noindex tag or password protection.

Should I add my sitemap to robots.txt?

Yes. A Sitemap: line helps search engines discover all your URLs. This generator includes it for you.

Is this robots.txt generator free?

Yes, it's free, needs no sign-up, and runs in your browser.

Related free tools

XML sitemap generator · Redirect chain checker · Canonical URL checker · Meta tag generator · All free SEO tools