robots.txt explained: what to allow and what to block

robots.txt sits at the root of your site and tells crawlers which paths they may visit.

User-agent: *
Disallow: /cart
Disallow: /account
Sitemap: https://shop.demo.pepploweb.com/sitemap.xml

What to block

• Private areas: carts, accounts, admin.
• Endless filter and search result pages.

What not to block

• CSS and JavaScript files search engines need to render pages.
• Pages you want to rank.

Important to know

robots.txt stops crawling, not indexing. A blocked page can still appear if other sites link to it. To keep a page out of results, allow crawling and use noindex.

Common mistakes

• Disallow: / left over from development, blocking the whole site.
• Blocking CSS or JS folders, so search engines cannot render pages properly.
• Using robots.txt to hide private pages. The file is public; anyone can read it and see the paths you list.
• Wrong location. It must be at the root: /robots.txt, not /folder/robots.txt.
• Typos in paths. Rules are case-sensitive.

Rules for specific crawlers

You can write a group for one crawler, such as an AI crawler, and a group for everyone else. A crawler follows the most specific group that matches its name, so be careful when you mix groups.

Test before you publish

Check a few important URLs against your rules before saving changes. One wrong line can hide your whole site from search.

Tools

Check yours with the robots.txt checker, build one with the robots.txt generator, and control AI crawlers with the AI robots.txt builder.