Exposure or Risk? How SMBs See AI Crawlers

While appearing in AI search results can elevate a business by helping them become more discoverable, many are worried about protecting their proprietary data. As a result, business leaders are split: while 72% express concern about AI crawlers visiting their sites, 62% also report that the impact has been positive.

It’s clear that small businesses owners and website managers are still figuring out how to handle AI crawlers: is brand visibility worth the risk to content ownership?

Key Findings:

• 72% of businesses are concerned about AI-powered crawlers coming to their site, with 57% concerned enough to block them.
• But SMBs are divided: 60% of businesses acknowledge that AI search and chatbots are positively impacting their revenue.
• AI crawlers are already impacting online performance, but not the way many people expected: 44% of respondents report that AI has positively impacted leads while 42% are reporting that it has positively impacted traffic.
• The majority (51%) of businesses say that brand visibility in AI search is the primary benefit of allowing AI crawlers.
• 43% of businesses are concerned most about how crawlers will impact content ownership and intellectual property.
• Cloudflare’s Pay Per Crawl is seen as a way to prevent unauthorized data scraping (68%) and to protect proprietary content (61%) without sacrificing brand visibility.
• Most think that their policy on AI crawlers needs to be reviewed regularly: 73% will reassess their business’s policy in the next 12 months.

To Crawl or Not to Crawl: Most SMBs Are Blocking AI Crawlers Despite Positive Impact

While many businesses report that they’ve been positively impacted by AI, 57% are still blocking AI crawlers.

Their top concern (43%) is how AI threatens their content ownership and intellectual property, and rightfully so — 16% of respondents also reported that AI bots have scraped their proprietary content in the last 12 months.

AI bots scrape content from websites to train LLMs, and as a result, content such as articles, data, research, and product documentation is frequently used without permission. Often, AI chatbots generate outputs that closely resemble these copyrighted materials without proper attribution or compensation.

Beyond copyright infringement, this can hurt a business’s revenue and brand image. If AI reduces the demand for a piece of original content, it threatens revenue streams tied to advertising, paywalls, or licensing. For instance, users may be less likely to click on a news article if they can find the same information through AI tools. As a result, ad placements on that site may decline.

This also means that businesses have less control over their brand and how content is distributed. That’s one reason 14% of respondents said brand misrepresentation was one of their biggest concerns with AI crawlers.

But small businesses that block AI crawlers altogether risk falling behind competitors who are discoverable in AI search.

Discoverability in AI Search is A Top Priority

The majority (51%) of respondents say that increased brand visibility in AI search is the primary benefit of allowing AI crawlers to their site, followed by indirect traffic and referrals (23%).

People are relying on generative AI chatbots to search more and more frequently. According to Semrush, ChatGPT hit a new record in July 2025 with a total of 5.24 billion web visits. That shift means consumers are increasingly encountering brands first through AI-generated content instead of traditional search engines.

If your competitors are being featured in AI search results, but you are not, they become the first option customers see. Over time, that visibility translates into greater trust, brand awareness, and sales opportunities.

Despite IP concerns, many companies are choosing to embrace AI search, knowing that as search evolves, they must too.

“I’m not blocking AI crawlers,” says Ali Dahmash, a marketing consultant. “I see them as another channel for visibility and discovery. The upside of having my content cited or summarized in AI results is greater than the downside at this point.”

By shutting out AI crawlers, businesses may safeguard their content in the short term, but they risk losing brand visibility and possibly impacting long-term growth.

Strategizing Beyond Just Blocking AI Bots

While 75% of businesses claim that it’s important to control whether AI crawlers can access their website, the decision to block (or not block) AI crawlers comes down to each business’s unique offerings and strategy.

Some may also choose which content AI crawlers can access, depending on the value it provides.

Website owners are most likely to block AI bot access to protect proprietary research, data, and reports (58%), customer reviews (48%), and pricing information (43%). These types of content are unique and can provide a lot of value to potential customers.

“We don’t block everything. The priority is protecting content that could hurt revenue or be misused to impact the business value,” says Mukul Singh, founder of the web and mobile app development company, Digital4design. “Most businesses protect content that drives revenue or is expensive to produce.”

For instance, reports often represent a significant investment of time, expertise, and resources. That, combined with customer reviews, can help differentiate a business from its competitors and influence purchasing decisions.

Pricing information, meanwhile, is closely tied to competitiveness and profitability; if freely accessible through AI crawlers, it can be easily compared, scraped, or even undercut by competitors in their industry.

In contrast, businesses are more likely to allow access to thought leadership content (77%), educational resources (78%), and user-generated content (86%).

Generally, this is considered top-of-funnel content that is designed to attract and engage potential buyers. This goal here is to drive awareness and generate interest and is perfect for companies looking to enhance visibility in AI search results.

Protecting IP in the Age of AI: Utilizing Cloudflare’s Pay Per Crawl & AI Crawl Control

In July 2025, Cloudflare, one of the largest global networks, announced that they would be offering a third option beyond just blocking or not blocking AI bots: pay per crawl. This feature leverages HTTP response code 402 to require payment for AI crawlers accessing content.

Roughly a month later, Cloudflare made AI Crawl Control generally available for paying customers. This feature also utilizes customizable HTTP 402 status codes to block specific bots and returns 402 Payment Required codes that provide licensing information.

Ultimately, this empowers content creators and gives them more control over their content. Instead of simply blocking bots or watching their work be scraped without consent, they can now decide who gets access, under what terms, and at what price.

That shift doesn’t just protect creators’ intellectual property — it also creates new opportunities to monetize valuable content and establish fair relationships with AI companies. In many ways, it represents a fundamental change in how creators safeguard, distribute, and profit from their IP in the age of AI.

It solves a lot of the biggest challenges businesses face with AI, too. In our survey, business leaders are planning to leverage pay per crawl to prevent unauthorized data scraping (68%), protect proprietary content (61%), and preserve SEO integrity (45%).

“Pay per crawl gives control and a revenue option,” says Singh. “It’s a tool I turn on if I need to monetize or limit high-volume AI training crawls.”

However, it's not the right fit for every business. “If your traffic is mostly driven by general discovery and you don’t face heavy crawling costs, the feature may add complexity without a clear upside,” says Singh.

How AI Search is Impacting the Online Performance of Small Businesses

There’s no doubt that AI is shaking up how people search online, and many business leaders worry about how it’s going to impact their business — can they still count on leads through their website if organic traffic continues to decline? How will this impact revenue? How should they adapt to remain competitive online?

Some may breathe a sigh of relief, though, thanks to our new data: 60% report that AI search has positively impacted their revenue. While many businesses are definitely challenged by the way search is evolving, the growth in revenue indicates that AI can be a powerful driver of growth for small businesses with the right strategy.

Beyond revenue, the majority (62%) of respondents said AI search/chatbots have positively impacted their business, with the biggest benefits being an increase in leads (44%), website traffic (42%), and inquiries (33%).

These findings suggest that AI search is opening new pathways for SMBs to be discovered, and it’s unquestionably impacting their business. It may also ease the minds of the 12% of respondents who said that loss of web traffic and engagement was one of their primary concerns with AI crawlers, too.

As more users shift to AI search, small and medium-sized businesses are increasingly seeing highly qualified and engaged visitors from AI referral traffic.

Too Good to Be True: Could AI Search Be Positively Impacting Traffic?

Nearly half (42%) of the participants in our survey reported that AI has positively impacted traffic, despite widespread data that web traffic is down likely because of zero-click searches.

The truth is that this is a nuanced data point: while total traffic is likely declining, traffic from AI chatbots continues to grow. A recent study of 391 SMB websites shows that ChatGPT referrals jumped by 123% between September 2024 and February 2025. The amount of AI referral traffic has doubled from .54% to 1.24%, too.

This sounds exciting, but it might not be the boon that people think it is. While AI chatbots are increasingly popular, they still trail behind traditional search engines and generate only a fraction of the daily traffic search engines like Google do.

That said, AI search continues to benefit businesses, particularly smaller, local ones, making it important to allow AI training bots access.

“Small businesses can benefit when they create helpful, local, or niche content that directly answers people’s questions. That’s because AI systems often rank clear, practical pages inside summaries or chat responses,” says Singh. “For example, small sites with strong local info (like service hours, pricing clarity, or FAQ pages) can be highlighted in AI answers and win new visitors.”  

Small businesses may be appearing more prominently in local-driven AI search results, which can translate into higher-quality leads and more inquiries as well.

Managing Bot Traffic is a Challenge for Most SMBs

The rise in AI also means an increase in AI training bots crawling the web, which can cause problems for web developers and website managers. In fact, 71% of businesses have experienced issues from bot traffic in the past 12 months, with their biggest challenge being performance and bandwidth strain (42%).

AI bots can generate thousands or even millions of requests in a short period, which consumes server resources and network bandwidth. If a site isn’t optimized for this load, it can slow down for real users or even cause outages. To combat this problem, some businesses have to block bots so their website doesn’t crash.

“Many businesses are starting to block or limit AI crawlers. Yes, I'm doing it too,” says Singh. “They cite high server loads, content scraping, or a lack of compensation. In one Reddit thread, a site owner shared that blocking bots was ‘the only way to prevent these bots from overloading my infrastructure.’”

Even if these website managers want to appear in AI results, those without the resources to handle increased bandwidth simply cannot.

How People Perceive AI Crawlers Likely to Change

How businesses are responding to AI crawlers will continue to evolve, with 73% of businesses planning to reconsider their policy towards AI crawlers in the next 12 months.

This highlights the fact that many leaders remain undecided about how best to balance the risks and rewards of allowing crawlers access.

As AI search continues to advance, many business leaders admit they don’t know what to expect in the coming year. On one hand, AI search could deliver even greater visibility and higher-quality leads, making the benefits of granting AI crawlers access outweigh potential costs to web performance or data exposure.

On the other hand, new models like pay per crawl may create entirely different incentives, giving businesses the chance to monetize their content and control how it’s used by AI systems.

In either case, the next 12 months are likely to be a testing ground, where companies experiment with different approaches to find the right balance between protection, discoverability, and revenue generation.

About the Author

Hannah Hicklen Content Marketing Manager at Clutch

Hannah Hicklen is a content marketing manager who focuses on creating newsworthy content around tech services, such as software and web development, AI, and cybersecurity. With a background in SEO and editorial content, she now specializes in creating multi-channel marketing strategies that drive engagement, build brand authority, and generate high-quality leads. Hannah leverages data-driven insights and industry trends to craft compelling narratives that resonate with technical and non-technical audiences alike. 

See full profile