Should publishers block the AI bots?

This was a panel of fighting’ talk. Publishers, after a couple of years of indecision, are aggressively blocking the AI training bots, and demeaning better deals and more data from the AI companies. And, in the meantime, exploring strategies that reduce their risk.

Is the industry getting its act together?

Panel

• Martin Ashplant, product development & operations director, PA Media

• Stuart Forrest, global audience director, Bauer Media

• Dan Rua, CEO Admiral, The Visitor Relationship Management Company

• Carly Steven, director of SEO and editorial e-commerce, Mail Online

• Chair: Charlotte Tobitt, UK editor, Press Gazette

Dan: He invested in Napster, and Grooveshark. It gave him a front-row seat on what happened to the music industry. Two decades later, music revenues are higher than they were before disruption. Why? He thinks it comes down to three things that the journalism world will have to adopt, too:

• Copyright Access Control
• Value Exchange
• Industry Solidarity

That’s how the music industry got its revenues back.

Carly: Clickthrough rates on AI overviews are low, but by virtue of being a big breaking news site, where AI doesn’t really compete yet, the impact on the Mail is negligible. “It matters, but we’re in the very fortunate position of over 50% of our traffic being direct, and much of their search traffic is branded search.” Her biggest plea to Google is to let us see the data, in the same way they let us see search data in Google Search Console.

Stewart: We have a very diverse portfolio across the UK and Germany. Zero click search has been a challenge for years, AI Overviews are just the next step in that. We’re seeing an impact on our Autos site — car specification data, for example. But our business isn’t built on that data, but on the insights we provide. The other thing we’e looking at is new information surfaces replacing Google — like ChatGPT. But that appears to be additive, rather than replacing Google.

Martin: What we’re seeing is a need to not reply on Google any more. For the last 20 years, it was largely seen that Google were the good guys out there. That’s clearly changed. This isn’t just about AI, it’s about how the Google interface has changed over the last few years.

Dan: Across the network, they’re not seeing overall drops in referral traffic – but look at the pattern by vertical, and you see patterns.

You are seeing escalating referrals from AI site — but it’s extremely small. The loss from search will not be compensated for by the rise from AI.

To block the AI training bots or not?

Stewart: There is no value exchange for us in AI scraping — and the bot scraping is costing us money as publishers. The sheer volume of bots… It’s now one in fifty web visits.

Carly: We block bots at robots.txt and at a web level. The referrals are so small, we don’t see the value. Plus, we’ve signed a deal with an equity stake in ProRata. They understand that there is a need for quality journalism to train the models on.

Dan: Step one is blocking the bots. Start there, but look for more creative ways of working with the ecosystem in future. It’s playing defence. And then start playing offence, working out how to offer AI features on the site. People are getting used to a chat UX.

Martin: Go and get the original content that can’t be indexed and trained on yet.

Give us the data, AI companies

Stewart Google needs to give us the ability to see what’s happening, and to chose to opt out of AI overviews, without losing search visibility. The bots are starting to disguise themselves by mimicking human behaviour, typically as a Chrome mobile user. We have to bring them under control. It’s not helpful to malign the bots — that leads to an arms race of detection and evasion. As publishers, we need to raise the price of scraping, so they come to us with a value exchange. We need regulation — it should be illegal for bots to imitate humans. And we have to work together as an industry. We require a common approach as an industry to bring the bots under control.

Carly: We need that data to differentiate AI Overviews and AI Mode in GA and Search Console. Google can do it. And we need more transparency around the algorithm. We need more transparency about what the big updates are targeting — more forewarning and insights would help. And better attribution and signposting when our content is used in AI. People don’t click on the links, and we think there needs to be better labelling and signposting.

Is there an AI upside?

Martin: Despite the negative take, I’m really bullish about AI. It enables us to do some interesting thing from a publishing point of view. We have an amazing archive, but we haven’t been able to do much with it. Could we go back into it and pull out the right content in the right way? Can we use AI to convert our content into other formats? Can we use Ai to create video out of still images on the fly? But I think we’ll see a premium placed on human-created content.

Carly: We want to do everything possible to give our engaged, loyal reader a great experience every time they come. Building on that brand and loyalty is our main priority.

Stewart: We’re reducing our dependence on Google search traffic. Authentic, real brands are becoming more important. Brand searches are becoming more important. We’re doing a lot of work around agentic workflows — can we put AI around people to help them make decisions? We’re also doing a lot of work on content lifecycle for SEO freshness reasons. That could free up valuable time to do new journalism.

Dan: The days of tracking people around the internet and shooting ads at them are gone. Publishers need to build relationships with their audience, and take an anonymous visitor and take them to maximum relationship, be it a newsletter relationship or a subscription relationship. If you want a conversation about sports, do you want to have that with an LLM or a sports expert?