Robots.txt Tester

Paste your robots.txt, choose a user-agent and test URL path to see if it's allowed or blocked. Includes syntax validation.

Robots.txt Content

Test a URL

Or type a custom user-agent below
Only the path part, e.g., /blog/post or /admin/login

Parsed Groups

Paste a robots.txt above to see parsed rule groups.

How Robots.txt Works

When a crawler visits your site, it first checks your robots.txt file to see which paths it's allowed to request. The parser evaluates rules in order, applying the longest matching path for the most specific user-agent match.

Common Syntax Rules

  • Each block starts with User-agent:
  • Disallow: blocks paths — Disallow: / blocks everything.
  • Allow: overrides a Disallow for more specific paths.
  • Empty Disallow: means "allow everything" for that user-agent.
  • Crawl-delay: sets delay between requests (Googlebot ignores this).
  • Sitemap: points to your XML sitemap location.
  • Lines starting with # are comments.
  • User-agent * acts as a wildcard matching all bots without specific rules.

Common Mistakes

  • Using noindex in robots.txt — this doesn't prevent indexing.
  • Blocking CSS/JS files — Google needs these for rendering.
  • Using relative paths — always use absolute paths starting with /.
  • Multiple Sitemap: directives — place sitemaps outside user-agent blocks.
  • Windows line endings (\r\n) — use Unix-style \n.