# crwl.sh

> crwl scores any website or API on how well AI agents and answer engines (ChatGPT, Claude, Perplexity, GPTBot, ClaudeBot) can find, read, and call it, then generates the exact files to fix what is wrong. Free, open source, zero-dependency.

crwl reads a site the way a machine does: it crawls the pages, checks for the
signals agents rely on (llms.txt, structured data, an agent or MCP card, an
AI-crawler policy, a sitemap, a discoverable API spec), and returns a 0-100
score with a grade and a plain-language list of what to fix. The scanner is
free; hosted monitoring and answer-engine visibility tracking are the paid product.

## Core

- [Home](https://crwl.sh): what crwl is, scan any URL, see an example report.
- [Signals](https://crwl.sh/signals): the full method, the signals agents read, and how scoring works.
- [The field](https://crwl.sh/leaderboard): how sites rank on agent readiness, by category.
- [The CLI](https://crwl.sh/cli): the open-source command-line scanner, install and usage.
- [Example report](https://crwl.sh/s/atlas-docs.io): a sample scan (score, layers, what to fix).

## Reference

- [Pricing](https://crwl.sh/pricing): free in open beta; hosted monitoring later.
- [Contact](https://crwl.sh/contact): feedback, bugs, and questions.
- [Privacy](https://crwl.sh/privacy): what the scanner reads and stores.
- [Terms](https://crwl.sh/terms): terms of use.

## Optional

- [llms-full.txt](https://crwl.sh/llms-full.txt): the full text, for deeper grounding.