As Artificial Intelligence continues to reshape how we consume and generate content, organizations are facing a new kind of challenge: AI bots and scrapers that crawl websites at scale, often without permission or compensation. These bots aren’t just a nuisance; they can inflate infrastructure costs, degrade performance, and potentially repurpose proprietary content to train large language models (LLMs).
In collaboration with our trusted partner GlobalDots, IBM has developed a smart, DNS-based solution using IBM NS1 Connect to help customers to regain control over who accesses their content and how.
The Growing Problem: AI Bots Are Not Your Typical Visitors
AI scrapers, deployed by companies to collect web data for training their models, now account for a growing share of internet traffic. Unlike organic human users, these bots don’t convert, don’t buy, and don’t engage with your content in a meaningful way. But they do consume bandwidth, inflate CDN bills, bypass terms of use and potentially harvest valuable content.
For content providers and e-commerce platforms, this is more than a nuisance—it’s a cost and copyright issue. As AI-generated summaries and search overviews become more common, original content creators risk being bypassed entirely.
An Autonomous System Number (ASN) is a unique identifier assigned to every network on the internet. Every internet service provider (ISP), cloud provider, or large organization that routes traffic independently is assigned an ASN.
When a DNS query is made, the ASN of the requester can often be determined—providing a powerful signal for traffic steering. This insight is critical in distinguishing between legitimate users and known AI bots that operate from specific ASNs.
The Solution: DNS-Level Traffic Steering with NS1 Connect
IBM NS1 Connect is a powerful authoritative DNS platform that offers advanced traffic steering capabilities. One of its standout features is Filter Chains, which allow DNS responses to be customized based on a variety of conditions—including ASN-based filtering. This allows you to apply a proactive defense at the very first step of a user’s journey.
1. Identify AI Bot ASNs: Many AI companies operate from known ASNs. These can be tracked and updated.
2. Configure NS1 Filter Chains: Using the ASN filter, you can instruct NS1 Connect to detect when a DNS query originates from a known AI bot ASN.
3. Serve Alternate DNS Responses: Instead of directing the bot to your main site, you can:
This approach is non-intrusive, highly scalable, and requires no changes to your web infrastructure. It is fast, invisible to users, and operates at the DNS layer – before bots even reach your infrastructure.
Real-World Impact: IBM, GlobalDots and a Leading European Organization
We recently implemented this solution for a major European organization in collaboration with our partner, GlobalDots. The result was immediate and significant, the customer achieved:
This success story highlights how DNS can be more than just a routing tool—it can be a strategic and proactive line of defense in the age of AI.
The AI landscape is evolving rapidly, and so must our infrastructure strategies. Content providers, eCommerce platforms and other digital businesses deserve tools that protect their assets and ensure fair use. IBM NS1 Connect offers a flexible, programmable DNS platform that empowers organizations to make smart, policy-driven decisions at the edge.
For Partners: If you are interested in co-selling this innovative solution and helping your customers combat unwanted AI traffic, please reach out to us today.
For Customers: To learn how this solution can help protect your digital assets and optimize your costs, request a demo today.
If you would like to contact GlobalDots directly, you can reach out to Francesco Altomare on francesco@globaldots.com
#Technical#CaseStudy#TechnicalBlog#BestPractices