Tell search engines what to crawl. Pick your rules and get a clean, ready-to-upload robots.txt you can copy or download.
A robots.txt file sits at the root of your domain and tells search engine crawlers which parts of your site they may and may not crawl. Get it wrong and you can accidentally hide your whole site, or expose pages you meant to keep quiet. This free generator builds a valid robots.txt from simple inputs, ready to copy or download.
Use * for all crawlers, or name a specific bot.
List the folders to block (admin, checkout) and any to explicitly allow.
Point crawlers to your sitemap, then copy or download the file.
This calculator is a taste of what Klairova does. Run real campaigns, track every click to every sale, send rate cards and invoices, and get paid in 160+ currencies. Free to use.
At the root of your domain, so it lives at example.com/robots.txt. Crawlers look there automatically; it will not work in a subfolder.
Not exactly. Disallow tells crawlers not to crawl a path, but a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag instead.
Small sites often do not, search engines crawl them fine by default. It is useful for steering crawlers away from admin or duplicate pages and for pointing to your sitemap. On Klairova your sites are crawlable by default with sitemaps handled for you.