A full crawl of a large site can take a moment.
A full crawl of a large site can take a moment.
5 ready-to-commit files, generated from what the crawl actually saw. Drop each one at the path shown, edit the parts the notes call out, then rescan to watch the score climb.
llms.txtA map of your key pages for LLMs. Replace the description and prune the list to what matters.
# reddit.com > One or two sentences describing what reddit.com is and who it is for. (edit this) ## Pages
robots.txtExplicitly welcomes major AI crawlers and points to your sitemap. Merge with any existing robots.txt.
# Generated by agent-ready. Merge with your existing robots.txt. User-agent: * Allow: / # Welcome AI crawlers explicitly User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: Bytespider Allow: / Sitemap: https://reddit.com/sitemap.xml
schema.jsonldschema.org structured data. Paste inside a <script type="application/ld+json"> tag in your <head>, and fill in the blanks.
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"description": "(edit) what this is",
"name": "reddit.com",
"url": "https://reddit.com"
},
{
"@type": "WebSite",
"name": "reddit.com",
"url": "https://reddit.com"
}
]
}
.well-known/agent-card.jsonA starter agent card so MCP clients can discover you. Add your real tools.
{
"description": "(edit) what an agent can do with this service",
"name": "reddit.com",
"tools": [],
"url": "https://reddit.com"
}
openapi.jsonAn OpenAPI skeleton. Document your real endpoints under paths.
{
"info": {
"title": "reddit.com API",
"version": "0.1.0"
},
"openapi": "3.1.0",
"paths": {},
"servers": [
{
"url": "https://reddit.com"
}
]
}