Configure crawler rules and get a ready-to-use robots.txt.
}
genRobots();
Presets included
- Allow all — let every crawler in
- Block all — hide from search engines (staging sites)
- Block AI crawlers — block GPTBot, CCBot, ClaudeBot, Google-Extended, and others from scraping your content
- Standard — allow crawling but block admin, API, and private paths
- Next.js / Astro / WordPress — framework-specific recommendations