Robots.txt
Tester and Validator Free

What Is robots.txt and Why Every Site Needs a Robots.txt Tester

A robots.txt file is a plain text file located at the root of your website (for example, https://yoursite.com/robots.txt) that instructs web crawlers which parts of your site they are and are not permitted to crawl. It follows the Robots Exclusion Protocol, a standard developed in 1994 that all major search engines including Google, Bing, and Yandex support.

Every site with pages you want to control crawler access to needs a properly configured robots.txt file. But writing the file correctly is surprisingly easy to get wrong. A single misplaced slash, a missing User-agent line, or an accidental Disallow: / can block your entire site from Google in minutes. This is exactly why a robots.txt tester is not optional. It is a critical sanity check before you deploy any changes to your robots.txt file.

What robots.txt can and cannot do

Robots.txt controls crawling, not indexing. A page blocked by Disallow will not be crawled, but it can still appear in search results if Google discovers it through links pointing to it. In that case, Google may index the URL without being able to read its content, showing it with no title or description. If you want to prevent indexing, use a noindex meta tag instead. Our Noindex Checker can verify your noindex tags are set up correctly alongside your robots.txt rules.

Robots.txt also does not protect content from being accessed directly. If someone knows a URL, they can visit it regardless of what your robots.txt says. For private content, use authentication and server-side access controls.

Robots.txt Syntax Guide: Writing Rules That Actually Work

Correct robots.txt syntax is essential. Google's robots.txt parser is strict, and a poorly formatted file may be partially or entirely ignored. Here are the key syntax rules every robots.txt validator checks.

User-agent

The User-agent: line declares which crawler the following rules apply to. User-agent: * applies to all crawlers. User-agent: Googlebot applies only to Google's main crawler. User-agent: Bingbot applies only to Bing. You can have multiple User-agent lines before a set of rules to apply those rules to multiple bots. Each group of rules must begin with at least one User-agent line.

Disallow and Allow

Disallow: /path/ blocks crawlers from accessing that path and anything beneath it. Allow: /path/ explicitly permits access to a path, overriding a broader Disallow. When both match a URL, Google uses the most specific rule (the one with the longest matching path). An empty Disallow: or Allow: / means allow all.

Wildcards and anchors

Google supports two special characters in path patterns. The * wildcard matches any sequence of characters. Disallow: /search* blocks any URL starting with /search. The $ anchor matches end of URL. Disallow: /*.pdf$ blocks all URLs ending in .pdf. The $ must appear at the end of the pattern to work correctly. Our robots.txt tester flags patterns where $ appears in the wrong position.

Sitemap declaration

You can declare your sitemap location directly in robots.txt using Sitemap: https://example.com/sitemap.xml. This is not part of the original robots.txt specification but is widely supported by Google, Bing, and other crawlers. Always use absolute URLs for sitemap declarations. Pair your robots.txt sitemap declaration with our Sitemap Validator to ensure the sitemap itself is correctly formatted.

The Most Dangerous Robots.txt Error: Disallow: /

The single most catastrophic mistake in robots.txt is placing Disallow: / under User-agent: * or User-agent: Googlebot. This one line tells every crawler it may not access any page on your website, effectively removing your site from search engine indexes.

Why this happens

This error almost always originates from a staging or development environment. When developers build a new site or work on a redesign, they typically set robots.txt to Disallow: / to prevent the staging site from appearing in search results. The problem occurs when this staging robots.txt gets accidentally pushed to the production server. The result is that Google stops crawling your site, and within days or weeks, your pages drop out of search results entirely.

How to detect and fix it

Our robots.txt tester checks for this condition immediately and flags it as a critical error. To fix it, remove the Disallow: / line (or replace it with Disallow: to allow everything) and re-submit your site in Google Search Console to trigger a fresh crawl. Recovery typically takes days to weeks depending on your site's crawl frequency. Our Sitemap Validator can help you re-submit your sitemap to speed up re-indexing.

Prevention with a robots.txt validator

The best way to prevent accidental deployment of a blocking robots.txt is to run a robots.txt validator as part of every site deployment process. Use our tool to test your robots.txt file before publishing. The paste mode lets you validate content without needing to push it live first.

Crawl-Delay in robots.txt: Does Google Actually Use It?

The Crawl-delay directive was designed to tell crawlers how many seconds to wait between fetching pages on your site. A value of Crawl-delay: 10 asks the bot to pause 10 seconds between requests.

Googlebot ignores Crawl-delay

Googlebot does not honour the Crawl-delay directive. Google has stated this clearly in Search Central documentation. The reason is that Google manages crawl rate centrally and adapts it based on server response times rather than relying on site-declared delays. If your server is slow to respond, Googlebot naturally slows down. If you want to explicitly reduce Googlebot's crawl rate, the only supported method is to use the crawl rate setting inside Google Search Console.

Other bots do respect Crawl-delay

Bingbot, Yandex, and many other crawlers do respect Crawl-delay. If you are worried about non-Google bot traffic overwhelming your server, setting a reasonable Crawl-delay is still useful for those bots. Our robots.txt checker notes when Crawl-delay is present, informs you of Googlebot's non-compliance, and flags unreasonably high delay values.

How to Test robots.txt Rules Against Specific URLs

Understanding whether a specific URL is blocked by your robots.txt requires testing the path against each rule group in order. This is not always obvious when rules use wildcards or when Allow and Disallow directives overlap.

Using the URL path tester

After loading your robots.txt with our tool, use the URL Path Tester section to enter any path (for example, /admin/settings) and select the user-agent you want to test as (Googlebot, Bingbot, or custom). The robot checker shows you exactly which rule matched, which line it is on, and whether that bot is allowed or blocked from crawling that path.

How Google picks the matching rule

When multiple rules match a URL, Google applies the most specific rule based on the length of the matching pattern. For example, if you have Disallow: /blog/ and Allow: /blog/featured/, then /blog/featured/article is allowed because /blog/featured/ is more specific. If two rules have the same length match, the Allow takes precedence. Our robots.txt tester applies this exact Google algorithm when you test paths.

For a complete crawlability audit alongside your robots.txt check, also run our Canonical Tag Checker to ensure your canonical tags reinforce your crawl directives correctly.

Common Robots.txt Mistakes Caught by a Robots.txt Validator

Beyond the catastrophic Disallow: / error, there are many smaller robots.txt mistakes that silently harm crawlability or waste crawl budget. Our robots.txt validator catches all of them.

Blocking CSS and JavaScript files

Older SEO advice recommended blocking /wp-content/ or /assets/ folders. Modern Google needs to render your pages like a browser, which requires accessing CSS and JavaScript. Blocking these resources prevents Googlebot from fully rendering your pages, which can hurt rankings. Remove Disallow rules targeting CSS, JS, and image directories.

Redundant trailing wildcards

Disallow: /path/* is identical to Disallow: /path/ because paths already match everything beneath them. The trailing /* is unnecessary. Our robots.txt checker flags these for cleanup.

Missing the wildcard agent block

If your robots.txt only contains rules for specific bots (Googlebot, Bingbot) but has no User-agent: * block, then all other crawlers receive no guidance and are free to crawl everything. This is usually fine, but if you have a reason to restrict content scraping bots, the wildcard block is where to do it.

Relative sitemap URLs

A sitemap declaration like Sitemap: /sitemap.xml uses a relative path. All major crawlers require absolute sitemap URLs in robots.txt. The correct format is Sitemap: https://example.com/sitemap.xml. Our robots.txt validator detects all relative sitemap declarations.