The problem
An AI assistant can only cite a page it can fetch and read. Those basics are invisible from the outside: you can't tell by looking whether robots.txt blocks ChatGPT's crawler, or whether a page's words only appear after JavaScript runs. I'm researching the GEO space, and this is where it starts.
What I did
- Read each page from its raw HTML, the way a crawler that doesn't run JavaScript sees it.
- Checked robots.txt for eleven crawlers, separating search and live fetching, where blocking hurts visibility, from model training, where blocking is a choice. Matching follows the robots.txt standard.
- Turned eleven checks into a score in three groups, with a "fix first" list ranked by the points each fix is worth.
- Reported a partial check when a site refuses unknown bots, instead of scoring it zero.
- Made it safe to run in public: only public addresses on every redirect, a 2 MB cap, and an 8-second timeout.
- Shipped it with no dependencies and ten tests, and made it pass its own check.
Results
11scored checks in three groups
11crawlers read from robots.txt
0dependencies
100its own score
Its first run on my own portfolio found two real gaps: no llms.txt, and a short address that names the longer one as its main URL.
What I'd do differently
Test on sites I don't control, early. My first version scored sites that refuse unknown bots as zero, which made a major news site look invisible when it had only refused my checker. Running it on real sites before launch caught that.
Artifacts
Try the checker ↗ · Source on GitHub ↗
