Generate a robots.txt file. Choose whether crawlers and AI bots may access your site, add your sitemap, then copy the result to your site root.
Updated
Your browser is preparing the tool. It runs 100% locally.
Choose whether to allow all crawlers and whether to explicitly allow AI bots (GPTBot, ClaudeBot, PerplexityBot and Google-Extended), add your sitemap URL, and the tool writes a robots.txt for your site's root. robots.txt is the standard way to guide crawlers, defined in RFC 9309, but it is advisory. Well-behaved bots obey it, and blocking a page doesn't guarantee it stays out of search. The file is generated in your browser.
The tool produces a short, valid robots.txt. It contains a User-agent star block set to Allow / or Disallow / by the first toggle, optional Allow blocks that name the major AI crawlers, and a Sitemap line pointing at your XML sitemap. Save it as robots.txt at the root of your domain.
With the AI toggle on, the file states outright that GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (Google's AI training) are allowed. Crawlers that honour robots.txt get a clear, public statement of your position.
robots.txt is plain text with a fixed grammar. A User-agent line names a crawler, and Allow or Disallow lines set what it may fetch, with a path of / meaning the whole site. The tool builds those lines from your toggles and adds the Sitemap directive so crawlers can find your URL list.
The file has exactly one home, yourdomain.com/robots.txt, and crawlers read it before fetching other pages. It became an official internet standard (RFC 9309) in 2022, formalising a convention that had been in use since 1994.
User-agent: *
Allow: / (or Disallow: / when 'allow all' is off)
(optional) Allow blocks for GPTBot, ClaudeBot, PerplexityBot, Google-Extended
Sitemap: your-sitemap-URLThe file can allow or disallow the whole site, with no per-path rules or crawl-delay. Switching the AI toggle off removes the explicit allow blocks but adds no Disallow, so those bots still follow the User-agent star rule.
| Inputs | sitemap URL, allow-all toggle, allow-AI toggle |
|---|---|
| Output | User-agent star rule, optional AI allow blocks, Sitemap line |
| Granularity | allow-all or disallow-all (no per-path rules) |
| AI bots named | GPTBot, ClaudeBot, PerplexityBot, Google-Extended |
| File location | yourdomain.com/robots.txt (root only) |
| Where it runs | Generated in your browser, never uploaded |
robots.txt is advisory. Reputable crawlers such as Google and Bing obey it, but the file enforces nothing, and some bots, including certain AI scrapers, ignore it. It asks crawlers to behave a certain way and can't lock them out.
Disallow is not the same as noindex. A page blocked in robots.txt can still appear in search if other sites link to it, and Google then shows it without a description. To keep a page out of search, use a noindex meta tag and leave the page crawlable, so don't block it here. Google has ignored noindex inside robots.txt since 2019.
Switching off 'Allow AI bots' doesn't block them. It removes the explicit Allow blocks, and those crawlers fall back to User-agent star, so they are still allowed while 'allow all' is on. Blocking an AI bot needs a Disallow rule for it, and this tool doesn't write one.
The tool only allows or disallows the whole site. To block a single folder, add a crawl-delay or set rules for one bot (apart from the AI block), edit the file by hand.
The file must sit at the root, as yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored, and the Sitemap line should be a full absolute URL.
Create a first robots.txt that allows crawling and points at your sitemap.
State openly that AI training crawlers are allowed, or edit the file afterwards to block them.
Switch to Disallow / to ask crawlers to skip a site that isn't ready.
Add the Sitemap directive so crawlers discover your URL list.
To keep a page out of Google, use a noindex meta tag instead of robots.txt. For per-folder rules or to block a particular bot, generate the base file here and edit it by hand.
The generated file is valid robots.txt with a sitemap line and optional AI-bot allow blocks. The page also explains three limits: robots.txt is advisory, Disallow is not noindex, and switching the AI toggle off doesn't block those bots.
Fill in your page title, description, keywords and author, and the tool writes a block of HTML head meta tags you can paste, with the title, description, viewport and robots tags included. The block updates as you type.
Generate Open Graph meta tags so your page shows a proper title, description and image when shared on Facebook, LinkedIn, WhatsApp and more. Fill four fields and copy the block.
Type a title, URL and meta description to see roughly how your page could look in Google, cut to length the way a real result is.
Turn a title or phrase into a lowercase, hyphenated URL slug, the readable tail of a web address. It updates as you type, in your browser.
Paste your text to get its Flesch Reading Ease score and US grade level, the two classic readability measures, updated as you type.
Compress an image to JPEG with a quality slider and watch the file size drop as you drag, in your browser.
Generate as many as you need at the top of the page. It runs on this device and needs no account.