From a URL to the files that fix it.
crwl reads your site the way an AI crawler does, scores how legible it is to agents and answer engines, and writes the exact files that close the gaps. Here is the whole path, start to finish.
of all web traffic is machines, not people
In 2024, automated traffic passed human traffic for the first time in a decade. AI is the reason.
The web is read by agents and answer engines as much as by people now. If they cannot read your site, you are invisible to a growing half of the internet. crwl shows you exactly what they see, and what to change.
Source: Imperva 2025 Bad Bot Report (Thales), 2024 data.
Four steps, about ten seconds.
- 01
Point it at a URL
Give crwl any address. No login, no script to install, no tag to embed. It works on a site you own or any site you want to measure.
- 02
It reads the page like an agent
crwl fetches your pages the way GPTBot, ClaudeBot, and PerplexityBot do: raw HTML, no JavaScript, no waiting for a framework to fill the page in. A site can look finished to a person and be nearly blank to a machine. crwl reads what the crawler reads.
- 03
It scores the gaps
Every signal a machine relies on rolls up into one 0-100 score and a grade across three layers: can an agent find you, read you, and call you. crwl weights those for the kind of site it detects and skips the checks that do not apply, so a personal site is never marked down for lacking an API.
- 04
It writes the files that fix it
For each gap, crwl generates the actual file that closes it: an llms.txt, a robots.txt policy, a sitemap, JSON-LD, built from what the crawl really saw on your site, not a template. Drop them in, rescan, and watch the score move.
Found, read, called.
The score in step 03 is built from three layers, each asking a different question about whether an agent can actually use you.
Discovery
Can an agent find out you exist and what you offer, without guessing?
Structure
Once it has the page, can it read the content and its meaning?
Interface
For products and APIs: can an agent call you, not just read you?
Scan a site and read the report.
The report hands you the files, not homework. crwl scores the same way on the web and in the open-source CLI, so the result is identical wherever you read it.