Crawl a site, extract its key pages, and get a spec-compliant llms.txt in seconds.
# FastHTML> FastHTML is a Python library for building server-rendered hypermedia applications with HTMX and Starlette.## Docs- [Quick start](https://docs.fasthtml.com/quickstart.md) A brief overview of many FastHTML features- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md) All HTMX attributes, CSS classes, and events## Examples- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py) Complete CRUD walkthrough showing idiomatic FastHTML patterns## Optional- [Starlette docs](https://gist.githubusercontent.com/jph00/.../starlette-sml.md) Starlette subset useful for FastHTML development
How it works
We traverse the site via sitemap.xml, robots.txt, and link discovery — respecting robots.txt rules and prioritizing high-value pages like docs and API references.
An LLM classifies each page, assigns it to a meaningful section, scores its importance, and writes a concise description — all tuned to the site's domain.
A spec-compliant llms.txt is assembled with a generated preamble, importance-ordered sections, and an Optional section for supplementary pages.
The output is always a spec-compliant llms.txt. Learn more about the llms.txt spec here.
Sign in for more features
Every llms.txt page you request is saved to your dashboard.
Download a folder with the latest llms.txt versions in one click.