Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which parts of the site they may crawl. It is read before crawling begins and is useful for keeping bots away from admin areas, cart and checkout steps, internal search results and endless parameter URLs.
- User-agent defines which crawler a rule applies to.
- Disallow blocks a path; a single slash blocks the whole site.
- Allow opens a specific path inside a blocked folder.
- Sitemap points crawlers to your XML sitemap.
Robots.txt controls crawling, not indexing. To remove a page from search results, use a noindex tag and let the page be crawled. The file is public, so it is not a security tool. The most damaging mistake is carrying a “block everything” rule from staging to the live site. Racoun configures robots.txt, sitemaps and noindex rules consistently on every site it builds and audits existing sites for conflicts.
Talk to Racoun: we are based in Istanbul and work with brands across Türkiye. Message us on WhatsApp or email ekip@racoun.com.
