【Survey Release】First Large-Scale Survey on 'LLM Pretraining Data Passage Rate' Across 3,200 Major Domestic Media Outlets

Todoo Nada Inc., a PR tech company headquartered in Taito-ku, Tokyo (President and CEO: Yasuyuki Matsumoto, hereinafter "the Company"), has conducted the first large-scale quantitative survey on how many of Japan's major 3,166 media outlets can pass through the pretraining data pipeline of generative AI (LLM), using the LLM-Friendly Check feature built into its PR effectiveness measurement service, "Qlipper."

The results revealed that only 10.0% (317 outlets) of domestic media are expected to be included in LLM pretraining data, and even when including those with "conditional passage," the figure reaches only 33.6% (1,063 outlets). For the first time, a two-tiered structure was quantitatively confirmed: traditional media (national newspapers, regional newspapers, and news agencies) are completely blocked via robots.txt, while major portals and online news sites are excluded due to content structure issues. Additionally, it was revealed that about 30% of press release distribution services (wire services)—long relied upon by PR professionals—are classified as "immediately discarded," despite having fully open robots.txt files, exposing the structural barriers of distribution sites to LLM reach.

Survey Background: What Is the LLM Pretraining Data Pipeline?

Generative AIs such as ChatGPT, Claude, and Gemini acquire language capabilities by learning from vast amounts of text on the web. However, not all web content is used for training. AI vendors generally use a multi-stage filtering pipeline to select training data:

Stage

Content

Reasons for Rejection

1 Crawl Permission Check (robots.txt layer)

Check the site's robots.txt to see if AI crawlers are allowed. robots.txt is a "gentlemen's agreement" and not legally binding

Blocked by robots.txt for specific crawlers

2 Content Retrieval (UA check/WAF layer)

Crawlers send actual requests to websites. Servers, CDNs, or WAFs decide whether to respond based on User-Agent strings

AI crawlers rejected by UA check (returns 403/429), AI bots automatically blocked by WAFs like Cloudflare, empty content on JavaScript-rendered sites

3 Cleansing (main text extraction)

Remove navigation, ads, and scripts from HTML to extract main text

Extremely short main text, excessive ads or decorative elements, too much template-generated text

4 Quality Scoring

Determine if extracted text is of sufficient quality for training

Mass-produced short articles, repetitive boilerplate text, insufficient main text ratio

5 Deduplication and Final Selection

Remove duplicate content

Republished content from other sites, internal duplicates

Especially important is that stages 1 and 2 form two independent layers. Recently, there has been a sharp increase in cases where, even if robots.txt allows access, servers block AI crawlers based on User-Agent strings. For example, Cloudflare, a major CDN provider, began offering a feature in 2024 to block AI bots by default.

Many sites that allow access via robots.txt are still blocked at the server, CDN, or WAF layer

This survey measured stages 1 (robots.txt layer) and 3–4 (cleansing/quality layer). Additional dropouts due to stage 2 (UA check layer) exist beyond the scope of this survey's measurements. Therefore, the actual reach rate to generative AI is likely even lower than the 10.0% passage rate found in this survey.

In other words, "existing on the web" and "being accessible to LLMs as training material" are entirely different. The PR industry's traditional effectiveness metric—"publication = exposure"—no longer holds in the era of generative AI.

Survey Overview

Item

Details

Survey Tool

LLM-Friendly Check feature in PR measurement service "Qlipper"

Survey Targets

3,166 major domestic news and specialized media sites registered in Qlipper

Survey Content

1 Retrieve each site's robots.txt and determine permission status for five major LLM crawlers (GPTBot/CCBot/ClaudeBot/Google-Extended/PerplexityBot) 2 Retrieve up to three article URLs per site and calculate post-cleansing survival scores based on industry-standard cleansing pipelines (aligned with C4/Gopher/RedPajama)

Survey Date

June 28, 2026

Score Categories

Expected Passage (0.6 or higher) / Conditional Passage (0.4–0.6) / Difficult Passage (0.2–0.4) / Immediate Discard (below 0.2)

Measurement Scope Note

This survey measures stages 1 (robots.txt layer) and 3–4 (cleansing/quality layer) of the LLM pretraining pipeline. Stage 2 (User-Agent check layer by servers, CDNs, or WAFs) is outside the scope, meaning actual reach rates to generative AI may be even lower than the survey results

※ The "LLM-Friendly Check feature" is a proprietary tool by the Company that diagnoses how visible a company's or client's site is to LLMs. This press release presents the first large-scale application of this feature.

Key Survey Results

[Result 1] Only 10.0% of domestic media pass through LLM training data

Among 3,166 major domestic media outlets, only 317 (10.0%) are expected to pass through the LLM pretraining pipeline. Even including conditional passage, the total is only 1,063 (33.6%). For the first time, it was quantitatively revealed that the remaining 66.4% (2,103 outlets) are effectively excluded from LLM training data for various reasons.

[Result 2] The main barrier is content quality, not robots.txt blocking

When analyzing reasons for failure to pass:

Cause

Count

Percentage

Blocked or inaccessible via robots.txt

248

7.8%

Unable to retrieve article URLs or calculate scores

569

18.0%

Failed quality criteria during cleansing

2,032

64.2%

Contrary to common public discourse about "AI blocking media," the biggest barrier is actually that the content structure of websites themselves fails to meet LLM pipeline requirements.

[Result 3] Reality of robots.txt-based LLM crawler blocking

Among 2,847 sites with valid robots.txt, 171 (5.4%) completely block all crawlers, and 223 (7.0%) selectively block specific crawlers.

Crawler

Blocking Rate (among selectively blocking sites)

GPTBot (OpenAI)

68.2%

CCBot (Common Crawl)

64.1%

ClaudeBot (Anthropic)

47.1%

Google-Extended (Google)

35.4%

PerplexityBot (Perplexity)

14.3%

Site operators are not indiscriminately blocking AI. Instead, they show clear selectivity—distrusting OpenAI and Common Crawl, but more accepting of Perplexity. A clear map of "which AI platforms a site appears on or not" is already forming.

[Result 4] Category-specific characteristics—"Three-tiered structure of non-passage"

▼ Tier 1: Traditional media completely closed at the "entrance"

Category

n

robots.txt blocking rate

Expected passage rate

National newspapers

6

100.0%

0.0%

Regional newspapers

95

67.4%

2.1%

News agencies

6

66.7%

—

Sports newspapers

12

66.7%

0.0%

The more authoritative the traditional media with primary sources, the more strongly they reject inclusion in LLM training data. This clearly shows that even if PR professionals secure articles in these outlets, the impact is unlikely to reflect in AI-generated responses.

▼ Tier 2: Portals—"complex bottlenecks" as a critical blind spot

Category

n

robots.txt blocking rate

Immediate discard rate

Expected passage rate

Portal sites

17

29.4%

23.5%

5.9%

Portals, long considered successful for PR when articles are published, have only a 5.9% expected passage rate (1 out of 17). They face combined barriers from both robots.txt blocking and content quality—forming a "compound failure" distinct from traditional or online media.

▼ Tier 3: Online media—"allowed but failed due to quality"

Category

n

Immediate discard rate

Expected passage rate

News sites

218

74.3%

0.9%

Curation sites

24

58.3%

0.0%

Specialized journals

273

37.0%

11.7%

Magazines

297

33.0%

11.1%

▼ Specialized sites—40% of total, mixed quality

Category

n

Immediate discard rate

Expected passage rate

Average score

Specialized sites

1,277

24.5%

12.7%

0.360

The largest category (40% of total), specialized sites show a bimodal score distribution (bottom 25% score 0.098 or lower, top 75% score 0.555 or higher), splitting into two groups: those that pass through LLMs and those immediately discarded. When PR professionals believe their content is published on an industry-specific site, its actual effectiveness is nearly a coin toss.

[Result 5] Even PR's lifeline—press release distribution (wire services)—fails expectations

Category

n

robots.txt blocking rate

Failed URL retrieval

Immediate discard rate

Expected passage rate

Wire services

10

0.0%

30.0%

30.0%

30.0%

The 10 major wire services surveyed have a 0% robots.txt blocking rate—fully open to LLM crawlers. This is structurally logical, as press release distribution aims for wide dissemination, with no incentive to block AI crawlers.

Yet, despite full openness, 30.0% are immediately discarded and 30.0% fail URL retrieval, meaning only 3 out of 10 services achieve expected passage. This category most starkly illustrates the survey's core finding: being "robots.txt-permitted" and actually "reaching LLMs" are not the same.

In the era of generative AI, PR effectiveness must shift beyond distribution volume and frequency to consider whether the target site's structure allows passage through the LLM training pipeline—a new effectiveness metric is emerging.

Conclusion: PR Industry Faces "Media Portfolio Redesign"

This survey reveals that PR activities in the generative AI era must now accept the following realities:

Two out of every three domestic media outlets are effectively "invisible" to LLMs as training material

A paradoxical structure where authoritative traditional media are most absent from LLMs

Traditional PR relying on portals and online news does not contribute to AI-mediated awareness

Even press release distribution—PR's lifeline—sees 30% discarded immediately; robots.txt openness is merely a necessary condition, not sufficient without structural improvements to distribution sites

The primary cause of non-passage is not robots.txt blocking, but rather the content structure of the sites themselves—indicating significant room for improvement

The long-standing PR mindset of "which media to appear in" must evolve into "which media can reach LLMs" and "whether our own site has an LLM-friendly structure"

Todoo Nada Inc.'s Initiatives

The Company offers the "LLM-Friendly Check feature" used in this survey via its PR measurement service "Qlipper" to help PR professionals. It enables quantitative diagnosis and visualization of where a company's or client's content fails in the LLM pretraining pipeline and how to improve passage rates.

Additionally, the Company has developed "Digidigi," a chat-based diagnostic tool that assesses how LLMs perceive and cite a company or its clients. The Company aims to establish a new standard for "cognitive measurement in the LLM era" within the PR industry.

Inquiries Regarding This Release

Todoo Nada Inc. Contact: Qlipper Operations Office Email: qlipper@todo-o-nada.com Web: https://qlipper.jp/

Company Overview

Item

Details

Company Name

Todoo Nada Inc.

Representative

President and CEO Yasuyuki Matsumoto

Headquarters

Yahiko Building 302, 7-11-13 Ueno, Taito-ku, Tokyo

Business Activities

Development and operation of PR effectiveness measurement service "Qlipper," development of LLM cognition measurement tool "Digidigi"

URL

https://qlipper.jp/

※ All figures in this release are based on survey results as of June 28, 2026. Detailed information on survey targets and methodology is available upon request.