Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot
摘要
该页面是 Known Agents 的 “Agentic Web Index” 数据看板，基于 5000+ 网站的 Agent Analytics 与 AI Chat Referral Tracking 数据，展示 AI 代理、爬虫等机器流量如何重塑网络信息发现与使用。核心指标包括：机器流量占 35%（较前 90 天降 2%），其中 AI 相关流量占 28%（升 11%），AI 聊天引荐占人类访问的 0.1%（降 9%），robots.txt 被遵守率 98.5%。页面按代理类型（AI 助手、AI 编码代理、AI 数据抓取器、AI 搜索爬虫等）与运营商（Google、Microsoft、OpenAI、Anthropic 等）细分流量，并列出各类型下访问量最高的代理（如 ClaudeBot、ChatGPT-User、PetalBot）及最常被访问的网站类别。此外，页面分四类展示 AI 活动：AI 抓取（训练数据）、AI 实时获取（RAG 任务）、AI 搜索索引（引用来源）、AI 浏览（自主完成任务），各附代理与运营商排名。
荐读理由
这份 5000+ 站点的实测数据能直接校准你对 AI 爬虫格局的判断：ClaudeBot 占 AI 抓取 27%、OpenAI 占 AI 实时取用 86%，据此可决定自家站点 robots.txt 该拦谁、该给谁开白名单。
原文
The Agentic Web Index
The internet is rapidly evolving from an environment built primarily for humans, into one increasingly used by machines. See how AI agents, crawlers, scrapers, and other bots are reshaping the way information is discovered, accessed, and used across the web.
Overview
Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
Bot vs. Human Traffic
35%
↓ 2%
Compared to the previous 90 days
The amount of visits from bots vs. humans
Agentrification
28%
↑ 11%
Compared to the previous 90 days
The percentage of bot traffic that's AI-related
AI Chat Referral Volume
0.1%
↓ 9%
Compared to the previous 90 days
The percentage of human website visits that come from AI chat
See Referral Trends ↓
Robots.txt Effectiveness
98.5%
How often bots follow robots.txt rules
See Per Agent ↓
Traffic by Agent Type
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
Automated Agent
Automated Agent
Automates browser interactions programmatically without direct human supervision
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
Fetcher
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
Security Scanner
Security Scanner
Scans websites for security vulnerabilities, threats, and configuration weaknesses
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
Uncategorized
Uncategorized
Not yet assigned a type
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Hover over each agent type for more information about what they do
Top Agent Types
22.8%
19.7%
12.4%
11.0%
10.9%
7.0%
6.1%
3.8%
2.7%
1.0%
0.8%
0.8%
0.4%
0.3%
0.2%
Agent types with the most activity
Top Visiting Agents
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
8.2%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
7.9%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
6.2%
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
5.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.2%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.2%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
3.0%
FTCH
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
2.6%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.2%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.2%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.1%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.1%
Agents with the most activity
Top Operators
10.5%
8.9%
7.8%
6.8%
6.4%
5.9%
5.1%
4.8%
3.9%
3.4%
2.3%
2.3%
2.2%
2.2%
2.1%
Operators with the most activity
AI Scraping Activity
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers.
Traffic by Agent Type
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI scraping activity by agent type over time
Top Visited Website Categories
Law and Government
6.0%
Online Communities
5.9%
Home and Garden
5.5%
Computers and Electronics
5.3%
Shopping
5.2%
Jobs and Education
5.2%
Pets and Animals
5.1%
Business and Industrial
5.0%
Autos and Vehicles
4.8%
Games
4.8%
Hobbies and Leisure
4.6%
Internet and Telecom
4.5%
Beauty and Fitness
4.1%
Website categories with most activity
Top Agents
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
27.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.5%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.2%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
9.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
7.3%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
5.6%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.9%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.8%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
1.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.8%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.5%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.4%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.3%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.2%
Agents doing the most AI scraping
Top Operators
27.0%
19.5%
19.2%
9.4%
7.3%
5.6%
3.9%
2.3%
1.8%
1.4%
0.8%
0.5%
0.4%
0.3%
0.2%
Operators doing the most AI scraping
AI Fetching Activity
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents.
Traffic by Agent Type
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI fetching activity by agent type over time
Top Visited Website Categories
Reference
4.0%
Science
3.7%
Finance
3.4%
Law and Government
2.6%
Computers and Electronics
2.3%
Beauty and Fitness
2.1%
Travel and Transportation
1.9%
Business and Industrial
1.8%
Internet and Telecom
1.7%
Health
1.6%
Autos and Vehicles
1.3%
Sports
1.3%
Real Estate
1.3%
Website categories with most activity
Top Agents
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
86.4%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.3%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.7%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.1%
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
1.1%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.4%
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.4%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Agents doing the most AI fetching
Top Operators
86.4%
3.6%
3.3%
2.7%
2.1%
1.0%
0.4%
0.4%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
Operators doing the most AI fetching
AI Search Indexing Activity
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
Traffic by Agent Type
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
AI search indexing activity by agent type over time
Top Visited Website Categories
Travel and Transportation
6.8%
Reference
6.5%
Health
5.6%
Games
5.2%
Food and Drink
4.9%
Books and Literature
4.9%
Jobs and Education
4.7%
Computers and Electronics
4.6%
Arts and Entertainment
4.5%
People and Society
4.4%
Shopping
4.3%
Law and Government
4.2%
Hobbies and Leisure
4.2%
Website categories with most activity
Top Agents
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
25.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
17.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
16.7%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
11.9%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.8%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Agents doing the most AI search indexing
Top Operators
25.4%
17.3%
16.7%
11.9%
10.8%
10.4%
3.4%
3.4%
0.3%
0.2%
0.2%
0.0%
0.0%
0.0%
0.0%
Operators doing the most AI search indexing
AI Browsing Activity
These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents.
Average Session Duration
1 minute
↓ 5%
Compared to the previous 90 days
The average duration of a session
Average Pages per Session
7.6
↓ 1%
Compared to the previous 90 days
The average number of pages visited per session
Traffic by Agent Type
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI browsing activity by agent type over time
Top Visited Website Categories
Law and Government
0.0%
Autos and Vehicles
0.0%
Computers and Electronics
0.0%
Internet and Telecom
0.0%
Travel and Transportation
0.0%
Business and Industrial
0.0%
Shopping
0.0%
Beauty and Fitness
0.0%
Sports
0.0%
Books and Literature
0.0%
Finance
0.0%
Arts and Entertainment
0.0%
Reference
0.0%
Website categories with most activity
Top Agents
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
43.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
40.1%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
16.9%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
Agents doing the most AI browsing
Top Operators
43.0%
40.1%
16.9%
0.0%
0.0%
Operators doing the most AI browsing
Robots.txt & Compliance
See which robots.txt rules are set across the web and how well agents follow them. An agent's Robots.txt Effectiveness measures the effectiveness of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked.
Effectiveness (Overall)
98.5%
How often all agents across all agent types follow robots.txt rules
Effectiveness (AI Scrapers & Data Providers)
97.5%
How often AI Data Scrapers and AI Data Providers follow robots.txt rules
Top Rule-Following Agents
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
100.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
Agents with the best Robots.txt Effectiveness percentages
Top Rule-Breaking Agents
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
82.6%
UNC
Uncategorized
Not yet assigned a type
84.5%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
90.4%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
91.2%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
92.7%
UNC
Uncategorized
Not yet assigned a type
93.1%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
93.2%
SCRP
Scraper
Extracts large amounts of web data, often without explicit website permission
93.8%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
94.7%
AUTO
Automated Agent
Automates browser interactions programmatically without direct human supervision
95.4%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
95.9%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
96.1%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
96.3%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
96.8%
ARCH
Archiver
Captures and stores historical website snapshots for long-term digital preservation
96.9%
Agents with the worst Robots.txt Effectiveness percentages
Robots.txt Rules by Top Blocked Agent
The percentage of the top 1,000 websites blocking each agent in robots.txt over time
Top Blocked Agents
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
24.6%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
22.6%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
21.7%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
18.9%
●
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
17.9%
●
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
16.1%
●
UND
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
15.8%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
15.6%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
15.6%
●
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
14.0%
●
UND
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
14.0%
●
UND
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
14.0%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
13.5%
Agents blocked by the most top websites
Spoofing & Security
See which agents are most frequently impersonated, and how spoofing activity changes over time. A visit is considered spoofed when it claims a recognized agent identity but fails that agent's supported authentication method, such as verified IP or Web Bot Auth.
Active Threat: AI Bot Spoofing Campaign
We are observing a widespread campaign impersonating AI bots to scan websites for vulnerabilities. The attacker appears to be targeting credential and configuration paths used by AI coding tools. Contact us for more information.
Spoofed Traffic by Agent Identity
The percentage of impersonated website traffic for each agent identity over time
Top Spoofed Agent Identities
●
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
0.5%
●
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.1%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.1%
●
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.1%
●
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.1%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.0%
●
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
●
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
0.0%
●
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
●
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.0%
●
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
0.0%
●
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
●
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
●
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.0%
The most impersonated agent identities
Recent Top Targeted Paths
/.config/anthropic/credentials/default.json
/.claude/settings.json
/.claude.json
/.hermes/.env
/.openclaw/.env
/.codex/config.toml
/.continue/config.json
/.aider.conf.yml
/service-account.json
/serviceaccountkey.json
/service_account.json
/firebase-adminsdk.json
/firebase-service-account.json
/.aws/credentials
/.aws/config
/.s3cfg
/.boto
/.npmrc
/.env.example
/.env.local
/.env.production
/.env.backup
/.env.old
/backend/.env
/api/.env
/admin/.env
/dockerfile
/docker-compose.yaml
/.docker/config.json
/terraform.tfstate
/credentials.json
/secrets.json
/secrets.yml
/key.json
/rclone.conf
Examples of recent top targeted request paths
AI Chat Referrals
See which AI platforms like ChatGPT, Perplexity, and Gemini cite websites and send them human referral traffic. Citations are estimated. Google's guide explains how websites can optimize their content to be more visible in AI chat responses (GEO).
Traffic by AI Chat Platform
ChatGPT
Claude
Copilot
DeepSeek
Gemini
Meta AI
Mistral
Perplexity
Referral activity by AI chat platform over time
Top Mentioned (Cited) Website Categories
Reference
4.0%
Science
3.7%
Finance
3.1%
Law and Government
2.6%
Computers and Electronics
2.3%
Beauty and Fitness
2.1%
Travel and Transportation
1.9%
Business and Industrial
1.8%
Internet and Telecom
1.6%
Health
1.6%
Autos and Vehicles
1.3%
Sports
1.3%
Real Estate
1.3%
Games
1.2%
Jobs and Education
1.0%
Website categories most frequently cited in AI chat responses
Top Clicked (Referred) Website Categories
Travel and Transportation
0.3%
Autos and Vehicles
0.1%
Beauty and Fitness
0.1%
Finance
0.1%
Computers and Electronics
0.1%
Jobs and Education
0.1%
Business and Industrial
0.1%
Home and Garden
0.0%
Sports
0.0%
Health
0.0%
Internet and Telecom
0.0%
Real Estate
0.0%
Science
0.0%
Food and Drink
0.0%
Shopping
0.0%
Website categories receiving the most referrals from AI chat
Methodology
Data Scope
The Index is updated daily with completed days of traffic, security, and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. The current partial day is excluded. Percentage-change tags compare the current period with the preceding period of the same duration. Agent names, operators, and classifications come from the Agent Directory, which is updated as new agents are discovered or existing agents change. Website categories follow the taxonomy used by Google AdSense. Participating websites are not a random sample of the entire web, and the qualifying set can change as websites connect, disconnect, or cross activity thresholds. Results characterize the observed network and broader directional trends; they should not be interpreted as a precise census of global web traffic.
Qualification & Aggregation
Only websites meeting minimum activity and data-quality requirements are included. Internal, test, incomplete, or anomalous data is excluded. Bot traffic percentages use total server traffic as their denominator. AI chat referral percentages use estimated human traffic, calculated by excluding identified bot visits from total server traffic. Rates are calculated for each qualifying website first, then averaged across websites and completed days. This gives each website equal weight regardless of traffic volume and prevents a small number of high-traffic websites from dominating the results. Daily charts are not smoothed, allowing normal seasonality to remain visible.
Measuring Robots.txt Effectiveness
An agent's Robots.txt Effectiveness estimates the reduction in its request rate associated with a full disallow rule. For each completed day, Known Agents establishes an agent-specific baseline from qualifying websites where that agent is allowed, adjusts the baseline for the overall traffic of each website where the agent is disallowed, and compares the expected activity with the activity actually observed. Only website-day observations with sufficient site traffic, agent activity, cross-site coverage, and expected volume qualify. Scores also require repeated observations across multiple websites and days. When an agent publishes a supported authentication method, only verified traffic is attributed to it.
Each qualifying website-day contributes equally. Scores range from 0%, meaning no measurable reduction, to 100%, meaning no qualifying requests were observed where the agent was disallowed. The headline Robots.txt Effectiveness metric gives each qualifying agent equal weight. Because this is an observational estimate rather than a controlled experiment, it measures an association with robots.txt rules but does not claim that robots.txt caused every observed difference. Top Blocked Bots is calculated separately using daily robots.txt scans of Similarweb's top 1,000 websites.
Identifying Spoofed Bots
Spoofing statistics measure traffic from visits that claim the identity of a known agent but fail a supported authentication method, such as published IP verification or HTTP message signatures. Each agent's daily rate is calculated against total server traffic for every qualifying website, then averaged across websites. A failed check indicates that the visit was likely impersonating the named agent; it does not identify the software or operator that actually made the request. Agents without a supported authentication method are not included in these measurements.
AI Chat Citations & Referrals
AI chat referral statistics count directly observed human visits carrying a recognized AI platform in the referring URL or campaign source. Visits without usable referral information cannot be attributed to an AI platform. Citation statistics are estimates based on requests from agents known to retrieve content for AI platforms. Those requests indicate that content may have informed a response, but they do not confirm that a source appeared as a citation to a user. Because AI platforms do not provide a complete public record of their sources, citation results should be interpreted as directional patterns rather than exact citation counts.
Frequently Asked Questions
Can journalists and media organizations use this data?
Absolutely. You may cite The Agentic Web Index with attribution and a link to this page. For interviews, fact-checking, background context, or a more specific breakdown for a story, contact us and include your deadline.
Do you work with researchers?
Absolutely. We welcome thoughtful research into how agents and bots are changing the web. Tell us about your research question, timeframe, and intended use. Depending on the scope and data constraints, we may be able to provide additional context, compare approaches, or explore a joint analysis.
Can I request a specific analysis?
Yes. If you need a breakdown by agent, operator, activity type, website category, or time period that is not shown here, contact us. When the underlying data supports it, we can examine the question and provide a focused analysis.
How do I see these trends on my own website?
Agent Analytics shows which agents and bots visit your website, what they access, and how their activity changes over time. AI Chat Referral Tracking measures the human traffic arriving from AI chat platforms. Automatic Robots.txt helps manage which bots can access your content.
这条对你有帮助吗？