Term · Search & content
robots.txt
robots.txt is a plain-text file at the root of a domain (/robots.txt) that tells web crawlers which paths they may or may not request. Search engines and many other bots respect it; it is a request, not a security measure.
[ 01 ]
Explained
Basic syntax
User-agent:— the crawler the rules apply to (*for all).Disallow:— paths not to crawl.Allow:— exceptions within disallowed paths.Sitemap:— the location of the XML sitemap.
A crawler follows the most specific group that matches its name; it does not combine rules from * with its own group.
What it controls
Crawling — which URLs bots fetch. Typical disallowed areas: admin panels, internal search results, carts and checkout, infinite filter combinations.
What it does not control
- Indexing: a disallowed URL can still appear in results if linked elsewhere. To keep a page out of search, use
noindexor authentication. See indexing. - Security: the file is public and advisory; private areas need access control.
AI crawlers
AI search and training crawlers — such as OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, or the Google-Extended token — can be allowed or disallowed individually. Blocking search-oriented AI crawlers can remove your content from those assistants' answers. See AI search visibility.
Common mistakes
Disallow: /left from a staging setup after launch.- Blocking CSS and JavaScript needed to render pages.
- Using robots.txt to hide sensitive pages.
- A specific crawler group that accidentally drops general restrictions.
[ 02 ]
FAQ
Frequently asked questions
Is robots.txt required?
Not strictly, but most sites benefit from one with a sitemap reference and sensible exclusions.
How quickly do changes apply?
Search engines re-fetch robots.txt regularly, usually within a day.