A robots.txt file is a short text file at the root of your site, at /robots.txt, that tells crawlers, the automated programs that visit sites to read pages, which URLs they may request. Google Search Central's introduction describes it as a way to manage crawler traffic first of all: keep your server from being overloaded with requests, skip unimportant or near-duplicate pages, and point bots at your sitemap, the list of public URLs you want found.
The same page puts the limit in bold: robots.txt is not a mechanism for keeping a web page out of Google. The file manages traffic. Keeping a page out of search takes something else, covered below.
Allow, Disallow and User-Agent: How to Use the Gate
Three directives do almost all the work:
- User-agent names which bot a rule is for. The name is the string the bot uses to identify itself, and an asterisk means all bots.
- Disallow tells that bot not to visit a folder or page.
- Allow makes an exception inside a blocked area: the bot may visit one specific page even though its parent folder is blocked.
A small file shows the shape:
User-agent: *
Disallow: /private-folder/
Allow: /private-folder/public-page.html
Read it line by line. The first line addresses every bot. The second keeps them out of /private-folder/. The third lets them back into one public page inside that folder. Keep the file at the root so crawlers find it before anything else, write one Disallow or Allow per line, and add a sitemap line if you use one, so bots can find your public URLs easily.
What a Block Does to Each Kind of File
Google's introduction splits the effect by file type, and the split matters more than most guides let on.
- Web pages (HTML, PDF and other formats Google can read): a block manages crawl traffic, for example when your server would be overwhelmed or when a section is full of unimportant or similar pages. A blocked page's URL can still appear in search results, without a description. Images, videos, PDFs and other files embedded in that page are excluded from crawling too, unless another allowed page references them.
- Media files (images, video, audio): a block manages crawl traffic and also keeps those files out of Google's results. It doesn't stop other pages or people from linking to them.
- Resource files (scripts, styles, unimportant images): you can block them only if pages still make sense without them. If their absence makes a page harder for Google's crawler to understand, leave them open, or Google won't analyze those pages well.
That last point catches sites that block a scripts folder to save crawl traffic and then wonder why their pages read badly to Google.
Block AI Crawlers with the Same Gate
Blocking an AI crawler reuses the same mechanism. Name its user-agent and disallow it:
User-agent: GPTBot
Disallow: /
GPTBot is one AI crawler's user-agent name, and the slash means the whole site. Add one block per bot, one named User-agent at a time. Google-Extended, another name site owners use when they mean to opt out of AI use, gets the same treatment.
A WordPress plugin shows what this looks like wired up. RankReady describes a control that lets a site owner allow or block 29 AI crawlers one by one, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bytespider, auto-synced to the site's robots.txt, plus a Content Signals option to say "index me, but don't train on me" in a form bots already read.
Fastly's guide to AI crawlers places the file inside a wider set of controls. A robots.txt block works when the crawler checks the file first, which responsible AI crawlers do. Site owners layer it with login or paywall requirements, rate limiting and bot-management tools for the crawlers that don't.
Robots.txt vs Noindex: Why a Block Doesn't Hide a Page
The file works as a guideline that well-behaved crawlers follow. Google's introduction says the instructions can't enforce crawler behavior: Googlebot and other respectable crawlers obey them, and other crawlers might not. Different crawlers can also interpret the same syntax differently, so the page points owners at the proper syntax for each crawler they address.
The practical consequence is that a blocked page can still show up in results. Google won't crawl or index the content blocked by the file, but it may still find and index a disallowed URL if other sites link to it. The address, and possibly public information such as the anchor text of links pointing at it, can then appear in search.
So use the file to save crawl budget, the share of attention a crawler will spend on your site, and to keep load off the server. When the goal is to keep a page out of search, use a password, a noindex tag (the marker that tells search engines not to show a page) or remove the page. If you choose noindex, remove the robots.txt block for that page: a crawler that can't read the page can't see the tag.
The same logic settles sitemap confusion. The sitemap lists public URLs for crawlers. Robots.txt manages which of them crawlers should request.
Check It and Watch It Change
Robots.txt shows up in routine publishing and monitoring stacks. One builder's publish checklist pings Bing and checks robots.txt alongside llms.txt and a newly added agent manifest. Another builder's monitoring build alerts on exactly this surface: the robots file changing, pages flipping to noindex, canonical tags (the markers that say which address counts as the main one) changing, HTTP errors, URLs dropping out of the sitemap, and clicks or impressions falling without explanation. A related French-language crawl stack crosses Screaming Frog, a crawl tool that reports status codes, canonical tags and indexing directives, with Search Console, analytics and speed data to catch the same drift.
The habits worth copying:
- Open
/robots.txtin a browser after every deploy and confirm it loads and says what you meant. - Run it through a robots.txt checker or tester to catch syntax slips.
- Keep a copy of the last known-good file, so a bad edit can be compared line by line.
- Alert when the file changes, not only when traffic falls.
