Discover the “ghost citation” issue in AI search, ...
Discover how AI-powered brand analysis helps busin...
Learn how entity optimization for Generative Engin...
If you still think of Googlebot as the main bot hitting your site, you are working with an outdated view of search. A large scale crawl study covering more than 24 million requests across commercial websites shows that OpenAI’s ChatGPT crawler is now significantly more active than Googlebot, and that AI search crawlers as a group are outpacing traditional search engine bots by a wide margin.
For search engine optimization in 2026, this is not a small technical curiosity. It is a sign that AI search, answer engine optimization and Google’s own evolving ranking systems now share the crawl stage with a new class of bots that decide when and how your content appears in AI generated answers.
The dataset that triggered this conversation was compiled by Alli AI and analyzed in a sponsored research article on Search Engine Journal. Over a 55 day period between January and March 2026, they captured 24,411,048 HTTP proxy requests hitting 69 client websites, covering more than 78,000 individual pages.
Most of the properties in the sample are WordPress based sites, which means the results are relevant to a very large share of the commercial web, from SaaS marketing pages and ecommerce catalogs to SEO driven content hubs. Request data was collected at the proxy or CDN layer rather than at the origin server, and crawler identity was determined by matching user agent strings and validating against published IP ranges.
This is not a complete map of the internet. It is, however, one of the largest open looks at how different crawlers behave right now in a world of AI search, AI overviews and aggressive model training.
When the team ranked all identified crawlers by raw request volume, the results clearly showed that AI crawlers now dominate the crawl mix across the measured sites.
The top ten crawlers by request count over the 55 day period looked like this (rounded numbers for readability):
ChatGPT-User (OpenAI) – about 133,000 requests
Googlebot – about 37,000 requests
Amazonbot – about 36,000 requests
Bingbot – about 18,000 requests
ClaudeBot (Anthropic) – about 14,000 requests
MetaBot – about 11,000 requests
GPTBot (OpenAI) – about 9,000 requests
Applebot – about 7,000 requests
Bytespider (ByteDance) – about 7,000 requests
PerplexityBot – about 6,000 requests
Across the measured network, the ChatGPT-User crawler alone hit these sites more often than Googlebot, Amazonbot and Bingbot combined. That is a dramatic reversal of the classic picture in which Google dominates crawl activity.
When crawlers are grouped by purpose, the split becomes even more striking. AI related crawlers, including ChatGPT-User, GPTBot, ClaudeBot, Amazonbot, Applebot, Bytespider, PerplexityBot and other AI engines, generated 213,477 requests during the study window. Traditional search crawlers like Googlebot, Bingbot and YandexBot generated 59,353 requests. In other words, AI crawlers made about 3.6 times as many requests as traditional search bots.
For anyone working on SEO, GEO or answer engine optimization, that 3.6:1 ratio should be treated as a leading indicator of where discovery is heading.
One of the most useful clarifications in the SEJ report is the distinction between OpenAI’s two main crawlers. Many site owners treat them as interchangeable or block one but not the other without understanding the consequences.
ChatGPT-User is the retrieval crawler. It fetches pages in real time when ChatGPT needs up to date web content to answer a user query, similar to how AI search or AI overviews fetch fresh data. If your content is not reachable by ChatGPT-User, it cannot appear as a cited source in ChatGPT answers.
GPTBot is the training crawler. It collects content that is used to train and refine OpenAI’s language models rather than to serve a specific AI search query in the moment.
The study found that combined, these two crawlers made 142,225 requests, which is roughly 3.8 times Googlebot’s request volume in the same window.
For AI search optimization, this matters a lot. Blocking GPTBot may control how your content is used in model training while still allowing retrieval for AI search. Blocking ChatGPT-User affects your visibility inside AI answers, even if GPTBot is allowed.
The raw numbers are only half the story. The study also compared performance characteristics like response times and success rates.
Across the sample, AI crawlers were significantly faster and more reliable than Googlebot. For example:
PerplexityBot saw an average response time of about 8 milliseconds with a 100 percent success rate.
ChatGPT-User averaged about 11 milliseconds with a 99.99 percent success rate.
GPTBot averaged about 12 milliseconds with a 99.9 percent success rate.
ClaudeBot averaged about 21 milliseconds with a 99.9 percent success rate.
Bingbot averaged about 42 milliseconds with around a 98.4 percent success rate.
Googlebot averaged about 84 milliseconds with roughly a 96.3 percent success rate.
Googlebot also received more error responses. Over the measured period it saw 624 blocked responses (HTTP 403) and 480 not found errors (HTTP 404), together accounting for around three percent of its total requests.
By contrast, AI search crawlers like ChatGPT-User and PerplexityBot encountered far fewer errors in this dataset. The report suggests that a mix of factors is driving this performance gap, including:
Younger, more up to date indexes that guide AI crawlers toward valid URLs.
Event driven crawling triggered by AI search queries rather than broad scheduled sweeps.
Architectures that favor pre rendered HTML and CDN edge caching, which can serve AI crawlers with minimal latency.
From an SEO and GEO perspective, the message is clear. AI search bots are not only more active than Googlebot in many environments, they are also extremely efficient, so technical improvements that help them tend to benefit user experience as well.
The Alli AI dataset is not the only signal that AI crawlers are rising. The SEJ article references additional analysis from Cloudflare and Akamai that supports the same conclusion.
Cloudflare’s 2025 data showed that ChatGPT-User requests grew 2,825 percent year over year, while AI user action crawling overall increased more than 15 times between 2024 and 2025. In the same period GPTBot requests alone grew about 305 percent, and its share of AI crawler traffic rose from roughly 2.2 percent to 7.7 percent.
Akamai’s measurements identified OpenAI as the largest AI bot operator on their network, accounting for about 42.4 percent of all AI bot requests.
Taken together with the new 24 million request dataset, these figures show that AI search engines and AI training systems are not a marginal traffic source. They are becoming structural consumers of web content at scale, with growth rates that far exceed those of classic search bots.
Traditional SEO has always focused on a specific pipeline. Googlebot crawls a page, Google indexes it, the page ranks within a Google algorithm driven ranking system, and users click from blue links to your site.
Answer engine optimization and AI search visibility work differently. AI crawlers such as ChatGPT-User, PerplexityBot, ClaudeBot and others fetch content in response to real time questions. The AI search engine then synthesizes an answer and may include zero, one or a small handful of citations. There is no concept of position three or page two. Either your site is cited in the AI answer or it is invisible in that interaction.
That leads to several important implications for modern SEO strategy:
Crawl volume is a leading indicator, not a vanity metric. AI search referrals are still smaller than Google Search clicks, but AI bots are already out crawling Googlebot in many environments. The 3.6:1 crawl ratio suggests that referral growth from AI search will likely follow.
Optimization targets are changing from rankings to citations. It is not enough to chase Google algorithm updates and core update recoveries. Brands now need to optimize content for selection inside AI answers, which can depend on clarity, topical authority, freshness and explicit licensing signals.
AI search and classic SEO are intertwined. Technical SEO improvements that help Googlebot also help AI crawlers. Structured data, clean HTML, strong information architecture and fast Core Web Vitals make it easier for both Google ranking systems and AI engines to interpret your site.
Thinking only in terms of “Google rankings” in 2026 underestimates how much of your future visibility will flow through AI search engines that operate on different mechanics.
One practical question many teams have is whether all this AI crawling will overwhelm their hosting or eat into their Googlebot crawl budget. The data suggests a nuanced answer.
On one hand, AI crawlers like ChatGPT-User send a much larger number of requests than Googlebot. ChatGPT-User alone sent more than 133,000 requests in 55 days across the Alliance AI sample, more than three times Googlebot’s count on the same sites. When you aggregate all AI search and AI training bots, the volume is clearly higher than traditional bots.
On the other hand, these requests are usually very light. Average response times in the 8 to 20 millisecond range indicate that AI crawlers are often hitting pre rendered HTML cached at the CDN edge rather than heavy dynamic pages. That means each individual request is inexpensive to serve, even if the total count is large. From a technical SEO and performance point of view, this leads to several recommendations:
Invest in CDN based architectures and static rendering for key SEO pages to absorb high levels of AI crawl activity without straining your origin servers.
Monitor server logs for spikes in AI crawler requests and verify that critical paths remain fast.
If necessary, use rate limiting at the CDN layer rather than blunt blocking in robots.txt, so you can manage load while preserving AI search visibility.
Well-engineered infrastructure will support both Google algorithm crawlers and AI search crawlers simultaneously, preserving crawl budget and user experience.
Technical access is only one part of the puzzle. To earn citations and traffic from AI search, Google organic and answer engines, your content still needs to deserve attention.
In the context of AI search and Google’s evolving ranking systems, useful content strategy principles include:
Create content that answers specific questions in depth. AI search engines look for pages that clearly and comprehensively address user intent. Detailed FAQs, how to guides and problem solution articles are natural fits for both SEO and AEO.
Use clear headings and semantic structure. Strong H1 and H2 headings, descriptive subheadings and consistent paragraph structure help both Google algorithms and AI models understand what each section is about.
Reinforce topical authority with internal linking. Cluster related articles and link them together using descriptive anchor text. This supports Google ranking systems, entity understanding and AI search engines that look for authoritative sources on a topic.
Keep key pages fresh and up to date. AI crawlers are tuned to fetch timely information. Updating statistics, examples and references on important pages gives both Googlebot and ChatGPT-User a reason to refresh their understanding of your content.
Ensure your content is accessible without heavy JavaScript. Several analyses, including the Alli AI study, note that major AI crawlers do not currently render JavaScript. Critical content and structured data should be present in the initial HTML wherever possible.
This is where SEO, GEO and AI search optimization converge. The same practices that help you survive Google core updates also set you up to be cited in AI answers.
To translate these insights into action, marketing and SEO teams can use a simple checklist:
Review server logs to quantify how often AI crawlers visit your site compared to Googlebot.
Audit robots.txt to ensure AI retrieval bots like ChatGPT-User are allowed where you want AI visibility.
Confirm that analytics and server performance monitoring can detect unusual AI crawl patterns.
Prioritize technical SEO projects that improve crawlability, Core Web Vitals and HTML clarity for both AI search and Google algorithms.
Build at least one content cluster that explicitly targets answer engine optimization, with structured FAQs and succinct, well structured answers.
Track mentions and citations in AI tools where possible to understand how AI search currently surfaces your brand.
Treating AI search bots as first class citizens in your SEO strategy does not mean abandoning traditional Google search optimization. It means accepting that crawl, indexing and discovery are now shared between classic ranking systems and a fast growing layer of AI engines that already generate 3.6 times more crawl activity than traditional search bots in some environments.
The brands that adapt their SEO and GEO strategies now will be better positioned when user behavior and referral traffic catch up with what these crawl numbers are already telling us.
