Channel sheet · CH-23 · gain 1 min · logged October 13, 2026

Content & SEO in the AI EraDirect input

Stradiji Audit: Most Websites Still Block AI Crawlers

Stradiji's AI visibility audit finds most websites still block LLM crawlers, leaving brands invisible in AI-generated answers. Operators should audit now.

By Marcus Bennett1 min read299 words

Signal notes

  1. Stradiji released an AI visibility study and audit covered by the Lansing State Journal
  2. The audit found most websites block access to LLM crawlers
  3. Blocked crawlers mean sites go uncited in AI-generated answers
  4. Many blocks stem from default CMS settings rather than deliberate policy

Stradiji's AI visibility study and audit has landed on a blunt verdict: most websites are closing their doors to LLMs. The audit, reported by the Lansing State Journal, found that the majority of sites it examined still block the crawlers that large language models use to read and index the web.

That's a quiet but consequential mismatch. Brands spend heavily on search visibility, yet many are effectively invisible to the AI systems that now answer customer questions directly. If a chatbot can't read your site, it can't cite you, recommend you, or describe you accurately.

What does this change for operators?

For site owners and marketing teams, the finding turns a technical question — what's in your robots.txt — into a business one. A site that blocks LLM crawlers doesn't just lose traffic in the traditional sense. It loses presence in the answers where more users now start their research.

The practical implication is simple to state and harder to act on: visibility work needs to cover both classic search and AI-mediated discovery. Auditing crawler access is the first step, and by Stradiji's numbers, most operators haven't taken it.

Why the door is closed

Blocks often happen by accident. Default CMS settings, inherited security rules, and blanket anti-bot policies frequently exclude AI crawlers alongside genuine threats. In other cases, the block is deliberate — publishers wary of their content feeding models they don't control.

Both positions are defensible. What the audit suggests is that the decision deserves to be a decision, made consciously with the trade-offs in view, rather than a leftover from an old config file.

The takeaway

Check whether the machines your customers ask can read your site — most can't, and that gap is now measurable.

via Google News — AI content and SEO (Source)

Filed under

  • ai-visibility
  • ai-crawlers
  • robots-txt
  • llm-seo
Share this article:

More from Marcus Bennett

Marcus Bennett

Show full bio

Senior reporter covering technology and product news in marketing professionals running AI in the stack at Mart Signal.

126 articles

Bus out

‹ Previous article