Every website that takes SEO seriously needs a robots.txt file — but it's also one of the easiest files to get catastrophically wrong. A single misplaced line can block Google from crawling your entire site, wiping it from search results overnight. This guide explains exactly what robots.txt does, how the syntax works, and how to use the Robots.txt Generator to create a safe, correct file without touching a line of code.
What Is a Robots.txt File?
Robots.txt is a plain text file stored at the root of your domain — accessible at yourdomain.com/robots.txt. It follows the Robots Exclusion Standard, a protocol that search engine crawlers (bots) check before crawling a site. The file tells bots which parts of your site they're allowed to crawl and which they should skip.
Crucially, robots.txt is a request, not a lock. Well-behaved bots like Googlebot respect it. Malicious scrapers do not. If you need to prevent content from appearing in search results, use a noindex meta tag — not robots.txt.
Understanding Robots.txt Syntax
User-agent
Specifies which crawler the rules apply to. Use * for all crawlers, or name specific bots like Googlebot, Bingbot, or GPTBot.
User-agent: *Disallow
Tells the crawler not to visit these paths. Leave the value blank to allow everything.
Disallow: /admin/
Disallow: /checkout/
Disallow: /private/Allow
Overrides a Disallow rule for specific paths. Useful when you've blocked a directory but want to allow a specific file within it.
Disallow: /assets/
Allow: /assets/og-image.jpgSitemap
Points crawlers to your XML sitemap. Best practice is to include this in robots.txt so crawlers discover it without needing a Search Console submission.
Sitemap: https://yourdomain.com/sitemap.xmlA Standard Robots.txt for Most Websites
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/
Disallow: /cart/
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThis allows all bots to crawl the entire site except sensitive backend and user-account areas, and points them to the sitemap.
Common Robots.txt Mistakes That Hurt SEO
Blocking CSS and JavaScript Files
Google renders pages like a browser and needs access to CSS and JS to understand your content and layout. If you block these in robots.txt, Google may see a broken, unstyled version of your pages and rank them lower.
Disallow: /wp-content/ ❌ This blocks all CSS, JS, and mediaAccidentally Blocking the Entire Site
The most dangerous mistake: Disallow: / under User-agent: * blocks every crawler from everything. This is sometimes accidentally left in development configurations that get pushed to production.
User-agent: *
Disallow: / ❌ Never do this on a live siteInconsistent Trailing Slashes
Disallow: /admin only blocks the exact path /admin. Disallow: /admin/ blocks the directory and everything inside it. Always use trailing slashes for directories.
Robots.txt in the Wrong Location
The file must be at the root of your domain — yourdomain.com/robots.txt. A robots.txt at yourdomain.com/blog/robots.txt will be ignored entirely.
How to Use the Robots.txt Generator
- Open the Robots.txt Generator.
- Select which crawlers the rules apply to (all bots, Googlebot only, etc.).
- Enter the paths you want to block — such as
/admin/,/login/,/cart/. - Add any Allow overrides needed.
- Enter your sitemap URL.
- Click Generate, review the output carefully, and download the file.
- Upload
robots.txtto the root directory of your web server. - Test it immediately using Google Search Console's Robots.txt Tester tool.
Blocking AI Crawlers
Many site owners now want to block AI training crawlers like OpenAI's GPTBot or Common Crawl. You can add specific rules for these:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /This prevents these bots from using your content for AI training without affecting Googlebot.
FAQ
Does robots.txt prevent pages from appearing in Google search results?
No. Blocking crawling prevents Google from reading the page, but if other sites link to it, Google may still show the URL in results with no description. Use noindex meta tags to fully remove a page from search results.
Can I have multiple User-agent blocks?
Yes. You can add as many User-agent blocks as you need, each with their own Disallow and Allow rules. Each block must be separated by a blank line.
How quickly does Google respect changes to robots.txt?
Google re-crawls robots.txt frequently — usually within a few hours to a day. You can force a refresh using the URL Inspection tool in Google Search Console.
Should I use robots.txt to block duplicate content?
Not ideally. It's better to use canonical tags to signal the preferred version of duplicate pages. Blocking via robots.txt prevents crawling but may not resolve the duplicate content issue.