Definition

AI crawler

A bot that fetches public web content for AI search, retrieval, or model-facing indexes.

What it means

An AI crawler is a user agent from an AI search or assistant system that requests pages for retrieval, summarization, or index-building.

Why it matters

AI-search visibility depends on crawlable pages, clear entity data, useful answer-first content, and accurate source files such as llms.txt.

Evaluation questions

Are important pages server-rendered and indexable?

Do robots rules allow the crawlers you want?

Does llms.txt summarize only verified facts?