(Animals strike curious poses. Imagen 4.)
Matthew Prince of Cloudflare recently described an alleged imbalance affecting content creators, and what Cloudflare and others are doing about it. It turns out that today’s AI web crawlers behave differently than yesterday’s search web crawlers.
The revolution
Prince began his article by describing a win-win deal facilitated by a content-gathering company known as Google. Google’s web crawlers would acquire site content, but the content creators would win also.
“The deal that Google made with content creators was simple: let us copy your content for search, and we’ll send you traffic. You, as a content creator, could then derive value from that traffic in one of three ways: running ads against it, selling subscriptions for it, or just getting the pleasure of knowing that someone was consuming your stuff.”
Sounds like a win-win to me.
The new power generation
What Prince didn’t say is that not everyone was thrilled with the arrangement.
Let’s start with Spain, and the relationship between Spanish online publications and Google Noticias (Google News).
The publishers thought they were getting the raw end of the deal, since Google would present summaries of the publishers’ content on Google pages, but no one would go to the publishers’ pages. Why bother? Google had shared the important stuff.
So Spain passed a law in 2014 requiring Google Noticias to pay…and Google Noticias shut down in Spain in December of that year.
“Reacting to a law that requires news sites in Spain to charge for their content, Google shut down its Google News service in the country….The tech company and other news aggregators would face steep fines if they publish headlines and abstracts without paying.”
At the time, I cast this as a battle between the nations and plucky individuals fighting for freedom…ignoring the fact that Google (cited twice below) was more powerful than some nations.
“So it’s possible for individuals to flout the laws of nations. The nations, however, are fighting back. Spain has passed content laws that are forcing Google to shut down Google Noticias in Spain. Swedish laws have brought the Pirate Bay offline. Russia is enacting laws that are forcing Google (again) to take its engineers out of Russia.”
As an aside, it’s worth noting that several nations subsequently banded together to implement GDPR, shifting more power to the governments.
Oh, and the Spanish law was changed to conform with European Union copyright law. As a result, Google Noticias came back online in Spain in 2022, eight years later.
3rdeyegirl (bear with me here)
Back to Cloudflare’s Matthew Prince, who talked about a brand new voracious web crawler that didn’t feel like a win-win. Rather than presenting links to outside content, or summaries of content accompanied by prominent links, AI tools (including Google’s own) would simply present the summaries, burying the links.
“Google itself has changed. While ten years ago they presented a list of links and said that success was getting you off their site as quickly as possible, today they’ve added an answer box and more recently AI Overviews which answer users’ questions without them having to leave Google.com. With the answer box they reported that 75 percent of queries were answered without users leaving Google. With the more recent launch of AI Overviews it’s even higher.”
So the new AI-sponsored web crawlers and their implementation effectively serve to keep readers in the walled gardens of OpenAI, Google, Microsoft, and the rest.
Walled gardens again? It’s just human nature. Having to click on a link to go somewhere else causes friction. This very post links to Cloudflare’s article, my old Empoprise-BI blog, and a multitude of other sources. And I bet you won’t click on ANY of those links to view the other content. I know. WordPress tells me.
As Prince himself puts it:
“…increasingly we aren’t consuming originals, we’re consuming derivatives.”
Ran out of band names
Back in 2023 I already noted a move to block AI web crawlers.
So what’s Cloudflare doing about the AI web crawlers that are sucking information away with little or no return to the content creators?
Blocking them by default.
“That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”
Cloudflare envisions a marketplace in which AI companies will pay creators for high quality content.
However, today’s content creators may face the same challenges that Spanish periodicals faced from 2014 to 2022. They may prevent their content from being ripped off…but no one will ever know because the people who go to ChatGPT will never learn about them.
Because in the end, most people are happy with derived content.
But your hungry people want to hear from you.
If you are a tech marketer who needs help creating content, talk to Bredemarket.
Bredemarket 