AI bots are now reading the web for you, and skipping the clicks

Date: 13 June 2025     Read Time: 5 min

Retrieval bots aren’t just crawling — they’re thinking, responding, and rewriting the rules. Credit: Kittipong Jirasukhanont from PhonlamaiPhoto’s Images via Canva.com

First came the SEO, then social algorithms, and now it’s bots, and they’re not asking for permission. According to new data from analytics firm TollBit, a new type of AI is transforming how people access information online. They are unlike any traditional crawlers, and these retrieval bots don’t just scan pages; they respond to life prompts from AI tools, such as ChatGPT and Perplexity. It summarises web content instantly.

The change is already measurable, with bot traffic surging by 49%. In 2025, websites receive more visits from AI agents than from actual human readers. The public barely notices this while newsrooms, blogs, and online shops are currently being left in the dark literally.

How AI bots have evolved

We have had bots on the web for decades, such as Google’s indexer, which are trackers and SEO scrapers. However, the new wave isn’t here to just map the map; it’s here to utilise it.

• Old Bots scan pages slowly in order to build the database, such as Google’s index.
• AI Retrieval Bots visit pages in real-time, especially when someone asks a question of ChatGPT or Perplexity. They will, in turn, read, extract, and summarise instantly. 

This new behaviour is called retrieval augmented generation. Instead of relying on what an AI model already understands, it will send bots to find the answer live from those trusted sources.

TollBit tracks activity across the internet, stating that these retrieval bots are outnumbering numerous traditional crawlers by more than two to one. They are often going undetected. That’s the issue because while users get their answers, the websites that are providing them never see a visit from them. 

Great answers, no visits: The publisher problem 

This is the internet, and millions of users are now asking AI tools for definitions, recommendations, news, and getting polished answers without ever opening a tab. The sources are derived from real articles written by journalists, experts, and bloggers who are not credited, linked to, or even acknowledged.

TollBit’s data clearly shows:

• AI bought traffic is up 49% in 2025.
• The majority of bots do not follow robots. Txt instructions.
• Most of the retrieval events don’t result from a click through to their source.

For independent sites, as well as major publishers, they’re losing traffic, visibility, and ad revenue. Many of them are unaware that this is happening, while others watch as AI bots transform once-popular pages into invisible background material.

Unlike Google search, these bugs don’t even pretend to send traffic back, so the real question isn’t how AI currently finds information but who gets paid for it. 

The TollBit approach

TollBit is a kind of watchdog and traffic cop, tracking AI bots in real-time that show publishers which bots are currently accessing what. What content, how often, and whether they’re obeying the rules.

This goes even further; it’s all about sites’ meter AI usage, so they’re like paywalls for humans. Instead of letting bots help themselves to free content, a publisher says; You can read this, but it will cost you.

While deals with OpenAI, perplexity, and others are still largely private, major outlets like Time and The Washington Post are signing content licensing deals.

It’s the same shift that occurs in music and film, which has been accelerated by streaming, and the outcome may shape whether journalism and online knowledge survive in the age of AI.

The future of the web

For most people currently using the internet, nothing feels broken because the AI tools simply work. You ask a question, you get an answer with no ads, no cookies, and no clutter. But that has a cost to pay:

• It conceals the origin of the knowledge.
• It undercuts the people who create it.
• It reshapes the web without public debate.

If retrieval bolts have become the default, then the open web risks turning into a background issue where content exists not to be read, but mined. This doesn’t mean the end of writers or websites, but it means there’s a shift in what gets seen, what gets credited, and what survives.

The bots are already here, and the question is whether the web can remain our own or just become raw materials for machines to mine and spit back at you with no sources.

Luckily, tools like TollBit are giving publishers a chance to fight back. Time is running short because every day, more traffic is being redirected, more credit is being stripped away, and most users forget the real people still write, what the boss summarises. The internet doesn’t need just better answers; it requires a fair way to keep making them.

Go to the main website: https://euroweeklynews.com/2025/06/13/ai-bots-are-now-reading-the-web-for-you-and-skipping-the-clicks/?utm_source=chatgpt.com