SEO glossary
robots.txt
robots.txt is a text file at the root of a domain that tells well-behaved crawlers which paths they may and may not request.
It controls crawling, not indexing, and confusing the two is the most expensive mistake in this file. A URL blocked in robots.txt can still appear in search results — Google just cannot see the page to know what is on it, or to read the noindex you put there. If you want a page out of the index, let it be crawled and serve noindex.
The rules are prefix matches against the path, and the most specific matching rule wins. Disallow: /admin blocks /admin, /administrator and /admin-tools alike, which catches people out more often than it should.
Two things worth doing: point at your sitemap with a Sitemap: line, and check the file after every deployment. A stray Disallow: / shipped by a staging config is the fastest way to lose every ranking a site has, and it looks like nothing in a diff.
Common question
- Does robots.txt remove a page from Google?
- No. It stops the page being crawled, which can leave it indexed with no description. Use a noindex tag on a crawlable page instead.
Check this on your own site
- Site Audit
Crawl your site and group what is broken
Definitions are the easy part
The free plan opens every tool with 500 credits and no card, so you can see these numbers on your own site rather than in an example.
Start free