Robots.txt Checker

The Robots.txt Checker reads your robots.txt and reports which crawlers its rules allow, restrict, or block. Where your site sends visit data to SurfacedBy, it adds what verified crawler visits show: requests from each provider's published addresses that your site answered or refused. It checks indexing headers and meta tags, your llms.txt file, and declared AI usage preferences.

In the dashboard it is the Access checks tab of the Tools page: open a domain, then Utilities > Tools > Access checks.

Crawler purposes

AI platforms publish named request identities and content-use controls. Their documented purposes guide how to interpret an access rule:

• Retrieval identities fetch pages for user-requested assistant activity, such as ChatGPT-User, Claude-User and Perplexity-User. A verified request establishes a page read; it does not establish a citation or a human visit. Operators differ in whether these requests follow robots rules.
• Search indexing crawlers build indexes used by search and AI platforms, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Access supports discovery but does not guarantee a listing or citation.
• Training and content-use controls govern model training and other documented uses, such as GPTBot, ClaudeBot, Google-Extended and CCBot. Some controls also govern answer grounding. Review the operator's policy before changing access.

The check covers 32 documented identities and controls from OpenAI, Anthropic, Perplexity, Google, Microsoft, Common Crawl, Meta, Apple, Amazon, DuckDuckGo, You.com, Mistral AI, Moonshot AI, Liner, ProRata, and Yandex. Of these, 29 identify requests and 3 are robots content-use controls.

How to run a check

The tab opens on the most recent stored result for the domain. Click Run check (or Re-check once a result exists) to check again. View robots.txt opens your live file in a new tab.

An authenticated Re-check reads your current robots.txt and the verified crawler visits recorded for the last 14 days. SurfacedBy never sends requests under an AI crawler's name, so access evidence comes only from the providers' own crawlers. The public checker has its own cache and reads your published rules and homepage only.

What the result shows

Status. The headline separates firewall refusals from robots.txt blocks. An allowed robots rule alone makes no claim about live access. The card shows the date of the latest verified crawler visit, or says that none was recorded recently. If robots.txt could not be read, its rules are unknown. Platforms reachable counts the mapped search crawlers without a supported block; it is separate from Brand Coverage.

Access by crawler. Each row separates Website rules from Access evidence. Website rules are evaluated against your homepage and example content paths; they do not check every URL on your site. Content permitted can coexist with restrictions on other paths. Some paths restricted is a partial restriction. Content blocked means all checked content paths are disallowed.

Access evidence distinguishes Verified visits answered, Verified visits refused, Mixed results, No recent evidence, and No recent verified visits. A verified visit is a request from the crawler provider's published addresses, so it shows how the real crawler is treated. Zero verified visits proves neither a block nor access. Use Search & answers or Model training to focus on the crawler purpose you want to review. Google-Extended and Applebot-Extended are permission settings, not separate crawlers, and show as Permission setting.

Open Details for the crawler's purpose, the meaning of your rules, the evidence explanation, and any next steps. View robots.txt directives contains path-scoped context, not proof of a live refusal. A Disallow entry for an unrelated path is not presented as blocking the whole crawler, and more specific Allow directives can take precedence.

Page indexing. Page-level instructions on your homepage and on one inner page (a page AI platforms cite, when there is one). A global noindex applies to search engines that support it; a named instruction applies only to that crawler. The card distinguishes restrictions for Google from instructions for other crawlers. nofollow is shown separately because it concerns links. No indexing restriction found does not guarantee a search engine will index the page. An incomplete page inspection cannot establish that an instruction is absent. Open View page directives for crawler scope, X-Robots-Tag, meta robots, and Cache-Control. A security page shows Security challenge without a verdict about the site's indexing instructions.

llms.txt. An optional guide to your content, separate from crawler permission. AI platforms may not use it, and its presence does not guarantee citations. On Professional and Agency, Generate llms.txt builds a recommended file from the pages it finds through your robots.txt, sitemap, and internal links, using each page's title and description, for you to review before publishing. Generation is limited per day, resetting at midnight UTC: 10 on Professional, 25 on Agency. Generating again for a domain within 24 hours returns the file already built and does not count against the limit.

AI usage preferences. Any Content-Signal line in your robots.txt (search, AI answer input, model training). A missing line is neutral and does not change the allow or block result.

Access and AI visibility. For the mapped search-indexing bots behind ChatGPT, Claude, and Perplexity, access evidence next to the share of prompts that cited your site. Unmeasured coverage is separate from a measured zero. An access result does not establish why citations changed.

Suggested next steps. Suggested changes to your website rules contains edits for blocked search or on-request crawlers. Apply them only if you want those crawlers to access your content. Review refused requests contains one ordered set of steps covering every security layer that refused a crawler, ending with a single re-check. Both can be added to your Plan. Training blocks are a content-use choice and are not presented as required citation fixes.

What verified visits can establish

A visit counts as verified when it comes from an address the crawler's provider publishes for that crawler. An answered verified visit shows the real crawler reached your site. A refused verified visit confirms a refusal, which stays reported until a newer verified visit is answered. Visits answered and refused in the same hour are shown as mixed results, because their order is unknown. Visits older than 14 days no longer establish current access. Your firewall event log is where you find and change the rule behind a refusal.

When your site blocks our checks

Some hosts and CDNs run bot protection that answers automated requests with a security page (a challenge, a captcha, or a "403 Forbidden" page) instead of the site. When it answers our checker this way on your robots.txt, homepage, inner page or llms.txt, the check reports what it could not read rather than reading the security page as your site.

The dashboard then shows a notice at the top of every page for that domain, naming who answered when the response identifies it (for example SiteGround or Cloudflare) and how many pages were affected. View details opens a panel with:

• the pages the security layer answered instead of your site, and when the check ran;
• a ready-to-send request for your host, with everything they need to allowlist us;
• our crawler's IP addresses and user agent, each with a copy button.

Allowlist our crawler by IP address. Most requests identify themselves with the SurfacedBy user agent, but some pages are requested again with a standard browser user agent from the same addresses, so a user-agent rule alone misses part of our traffic. The current addresses are always published at surfacedby.com/bot, with a machine-readable list at surfacedby.com/bot.json for hosts that refresh allowlists automatically.

Once your host has allowlisted us, click Re-check on this tab. The notice follows the new result straight away. Otherwise it updates after the next daily check. You can dismiss the notice; it comes back only if the block changes (for example, a different page is refused).

The same protection can refuse the AI platforms' own crawlers. If you want AI assistants to read and cite your pages, ask your host to allow their crawlers too.

Automatic monitoring

On paid plans and the trial, SurfacedBy also runs this check on its own once a day for each domain that has completed a Full Check. When a retrieval or search-indexing identity becomes blocked, and the block shows up in two readable checks in a row, you get a site health alert that links to this tab. Homepage indexing alerts require a noindex that applies to Google, confirmed by a second readable homepage check at least 45 minutes later. Instructions for another named crawler and nofollow alone do not trigger that alert. An unreadable or incomplete check cannot resolve it. Monitoring also alerts when your robots.txt answers two checks in a row with a server error (crawlers treat that as a signal to stop crawling). You get a second alert when the issue is gone. The current alert policy excludes identities classified as training. That exclusion does not establish how an operator's content-use policy affects its answers.

Each fresh check, manual or automatic, is stored. The Tools History tab lists them alongside your page checks.

Only refused verified visits raise firewall alerts. The alert resolves once newer verified visits from those crawlers are answered.

Plan availability

The Robots.txt Checker is included in Starter, Professional, and Agency, and in the trial. The free plan does not include the Tools page. Running a check requires the Editor role or higher. llms.txt generation requires Professional or Agency.

A free public version at surfacedby.com/tools/robots-txt-checker is limited to 10 checks per 24 hours.