AKARUMI INSIGHTS: AI Bots Access 'robots.txt' and 'sitemap' Most Frequently
ipe inc. analyzed AI bot access to its website and discovered that site structure pages like robots.txt and sitemaps are accessed the most. Trends varied by bots such as Claude and GPT. The report highlights the growing importance of AI-friendly information architecture.
📋 Article Processing Timeline
- 📰 Published: May 19, 2026 at 01:01
- 🔍 Collected: May 18, 2026 at 16:31
- 🤖 AI Analyzed: May 18, 2026 at 23:40 (7h 8m after Collected)
ipe inc. (hereafter ipe) analyzed the AI bot access to the website of its proprietary AI writing tool 'DeepEditor' over the past 30 days, using 'AKARUMI', a visibility tool designed for the AI search era.
With information gathering increasingly originating from Generative AI platforms like ChatGPT and Gemini, it has become crucial for companies to understand how they are recognized and referenced by AI.
Here, we introduce excerpts from the report analyzing which pages AI bots were accessing.
Summary of Findings
In this analysis, the pages most frequently accessed by AI bots were not article or service pages, but site structure-related pages such as /robots.txt and /sitemap.xml.
Additionally, differences in access behavior were observed among ClaudeBot, GPTBot, and PerplexityBot. ClaudeBot predominantly accessed site structure pages, while GPTBot's access was more distributed, including pricing and article pages.
In the era of AI search, understanding exactly which pages AI bots are looking at has become essential.
'robots.txt' and Sitemap-related Pages Received the Most Overall AI Bot Access
Aggregating access from all AI bots, the most visited page was /robots.txt. This was followed by the root domain (/), /sitemap.xml, and /sitemap.rss.
According to the data, the total visits to /robots.txt, /sitemap.xml, and /sitemap.rss reached 249, accounting for approximately 61.5% of the access within the top 10 pages.
These results suggest that AI bots are checking not only the main text of the pages but also crawlability, URL structure, and update information.
Access Trends Vary by AI Bot
When looking at individual AI bots, distinct differences in access trends emerged.
ClaudeBot had a high ratio of access to /robots.txt, /sitemap.xml, and /sitemap.rss, focusing primarily on site structure pages.
On the other hand, GPTBot accessed sitemap-related pages but also visited pricing pages, article pages, case study pages, and news pages.
Meanwhile, PerplexityBot focused mostly on the top page and /robots.txt. These results indicate that different AI bots may be verifying different types of information and serving varying roles.
Website Improvement Points for the AI Search Era
This survey confirmed AI bot access to pricing pages, feature pages, solution pages, and article pages.
In the AI search era, it is increasingly important to organize pricing, feature differences, use cases, and implementation conditions to design information in a way that AI can easily understand.
Furthermore, clearly organizing the top page to define 'what the service is,' 'who it is for,' and 'what problems it solves' is considered critical for helping AI grasp the site's content.
'AKARUMI': Visualizing AI Bot Access
With AKARUMI, businesses can visualize access from AI bots like ChatGPT, Claude, and Perplexity, and analyze exactly which pages are being viewed by AI.
By confirming AI bot access, which is difficult to grasp using only GA4 or GSC, companies can better understand which pages to strengthen in the AI search era and prioritize improvements for LLMO.
About ipe inc.
ipe inc. is a digital marketing company that supports businesses through SEO and LLMO consulting to ensure they are correctly recognized, quoted, and selected by search engines and generative AI.
For over 13 years, the company has provided SEO support, primarily for large-scale and database-driven sites, assisting more than 300 companies. Currently, ipe leverages this expertise to enhance LLMO support while offering 'AKARUMI,' an analysis platform that visualizes brand exposure on AI search engines and LLMs.
Company Name: ipe inc.
Headquarters: Aoyama Tower Building 5F, 2-24-15 Minamiaoyama, Minato-ku, Tokyo 107-0062
Established: October 2013
Business Operations: Web consulting, media business, advertising business, etc.
Corporate Website: https://ipeinc.jp/
Phone: 03
With information gathering increasingly originating from Generative AI platforms like ChatGPT and Gemini, it has become crucial for companies to understand how they are recognized and referenced by AI.
Here, we introduce excerpts from the report analyzing which pages AI bots were accessing.
Summary of Findings
In this analysis, the pages most frequently accessed by AI bots were not article or service pages, but site structure-related pages such as /robots.txt and /sitemap.xml.
Additionally, differences in access behavior were observed among ClaudeBot, GPTBot, and PerplexityBot. ClaudeBot predominantly accessed site structure pages, while GPTBot's access was more distributed, including pricing and article pages.
In the era of AI search, understanding exactly which pages AI bots are looking at has become essential.
'robots.txt' and Sitemap-related Pages Received the Most Overall AI Bot Access
Aggregating access from all AI bots, the most visited page was /robots.txt. This was followed by the root domain (/), /sitemap.xml, and /sitemap.rss.
According to the data, the total visits to /robots.txt, /sitemap.xml, and /sitemap.rss reached 249, accounting for approximately 61.5% of the access within the top 10 pages.
These results suggest that AI bots are checking not only the main text of the pages but also crawlability, URL structure, and update information.
Access Trends Vary by AI Bot
When looking at individual AI bots, distinct differences in access trends emerged.
ClaudeBot had a high ratio of access to /robots.txt, /sitemap.xml, and /sitemap.rss, focusing primarily on site structure pages.
On the other hand, GPTBot accessed sitemap-related pages but also visited pricing pages, article pages, case study pages, and news pages.
Meanwhile, PerplexityBot focused mostly on the top page and /robots.txt. These results indicate that different AI bots may be verifying different types of information and serving varying roles.
Website Improvement Points for the AI Search Era
This survey confirmed AI bot access to pricing pages, feature pages, solution pages, and article pages.
In the AI search era, it is increasingly important to organize pricing, feature differences, use cases, and implementation conditions to design information in a way that AI can easily understand.
Furthermore, clearly organizing the top page to define 'what the service is,' 'who it is for,' and 'what problems it solves' is considered critical for helping AI grasp the site's content.
'AKARUMI': Visualizing AI Bot Access
With AKARUMI, businesses can visualize access from AI bots like ChatGPT, Claude, and Perplexity, and analyze exactly which pages are being viewed by AI.
By confirming AI bot access, which is difficult to grasp using only GA4 or GSC, companies can better understand which pages to strengthen in the AI search era and prioritize improvements for LLMO.
About ipe inc.
ipe inc. is a digital marketing company that supports businesses through SEO and LLMO consulting to ensure they are correctly recognized, quoted, and selected by search engines and generative AI.
For over 13 years, the company has provided SEO support, primarily for large-scale and database-driven sites, assisting more than 300 companies. Currently, ipe leverages this expertise to enhance LLMO support while offering 'AKARUMI,' an analysis platform that visualizes brand exposure on AI search engines and LLMs.
Company Name: ipe inc.
Headquarters: Aoyama Tower Building 5F, 2-24-15 Minamiaoyama, Minato-ku, Tokyo 107-0062
Established: October 2013
Business Operations: Web consulting, media business, advertising business, etc.
Corporate Website: https://ipeinc.jp/
Phone: 03
FAQ
Which pages do AI bots mainly access?
They primarily access site structure pages like /robots.txt and /sitemap.xml, rather than article or service pages.
What is important for LLMO?
Organizing information such as pricing, features, and usage in a way that is structured and easy for AI to understand.
What is AKARUMI?
It is an analysis tool provided by ipe inc. that visualizes AI bot access and brand exposure on AI searches and LLMs.