Voracious Crawlers: Cloudflare and AI

(Animals strike curious poses. Imagen 4.)

Matthew Prince of Cloudflare recently described an alleged imbalance affecting content creators, and what Cloudflare and others are doing about it. It turns out that today’s AI web crawlers behave differently than yesterday’s search web crawlers.

The revolution 

Prince began his article by describing a win-win deal facilitated by a content-gathering company known as Google. Google’s web crawlers would acquire site content, but the content creators would win also.

“The deal that Google made with content creators was simple: let us copy your content for search, and we’ll send you traffic. You, as a content creator, could then derive value from that traffic in one of three ways: running ads against it, selling subscriptions for it, or just getting the pleasure of knowing that someone was consuming your stuff.”

Sounds like a win-win to me.

The new power generation

What Prince didn’t say is that not everyone was thrilled with the arrangement.

Let’s start with Spain, and the relationship between Spanish online publications and Google Noticias (Google News). 

Imagen 4.

The publishers thought they were getting the raw end of the deal, since Google would present summaries of the publishers’ content on Google pages, but no one would go to the publishers’ pages. Why bother? Google had shared the important stuff.

So Spain passed a law in 2014 requiring Google Noticias to pay…and Google Noticias shut down in Spain in December of that year.

“Reacting to a law that requires news sites in Spain to charge for their content, Google shut down its Google News service in the country….The tech company and other news aggregators would face steep fines if they publish headlines and abstracts without paying.”

At the time, I cast this as a battle between the nations and plucky individuals fighting for freedom…ignoring the fact that Google (cited twice below) was more powerful than some nations.

“So it’s possible for individuals to flout the laws of nations. The nations, however, are fighting back. Spain has passed content laws that are forcing Google to shut down Google Noticias in Spain. Swedish laws have brought the Pirate Bay offline. Russia is enacting laws that are forcing Google (again) to take its engineers out of Russia.”

As an aside, it’s worth noting that several nations subsequently banded together to implement GDPR, shifting more power to the governments.

Oh, and the Spanish law was changed to conform with European Union copyright law. As a result, Google Noticias came back online in Spain in 2022, eight years later.

3rdeyegirl (bear with me here)

Back to Cloudflare’s Matthew Prince, who talked about a brand new voracious web crawler that didn’t feel like a win-win. Rather than presenting links to outside content, or summaries of content accompanied by prominent links, AI tools (including Google’s own) would simply present the summaries, burying the links.

“Google itself has changed. While ten years ago they presented a list of links and said that success was getting you off their site as quickly as possible, today they’ve added an answer box and more recently AI Overviews which answer users’ questions without them having to leave Google.com. With the answer box they reported that 75 percent of queries were answered without users leaving Google. With the more recent launch of AI Overviews it’s even higher.”

Imagen 4.

So the new AI-sponsored web crawlers and their implementation effectively serve to keep readers in the walled gardens of OpenAI, Google, Microsoft, and the rest.

Walled gardens again? It’s just human nature. Having to click on a link to go somewhere else causes friction. This very post links to Cloudflare’s article, my old Empoprise-BI blog, and a multitude of other sources. And I bet you won’t click on ANY of those links to view the other content. I know. WordPress tells me.

As Prince himself puts it: 

“…increasingly we aren’t consuming originals, we’re consuming derivatives.”

Ran out of band names

Back in 2023 I already noted a move to block AI web crawlers.

So what’s Cloudflare doing about the AI web crawlers that are sucking information away with little or no return to the content creators?

Blocking them by default.

“That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”

Cloudflare envisions a marketplace in which AI companies will pay creators for high quality content.

However, today’s content creators may face the same challenges that Spanish periodicals faced from 2014 to 2022. They may prevent their content from being ripped off…but no one will ever know because the people who go to ChatGPT will never learn about them.

Because in the end, most people are happy with derived content.

But your hungry people want to hear from you.

If you are a tech marketer who needs help creating content, talk to Bredemarket.

Stop losing prospects! Use Bredemarket content for tech marketers
Content for tech marketers.

July 4 Parade, Post Assembly

In addition to the security and pre-assembly videos I filmed before today’s July 4 Independence Day parade in Ontario, California, I also filmed a little bit of video during the parade itself.

But not the bagpipes. (I heard enough of them when I lived near Upland High years ago.)

The video below includes the Vietnam veterans and Chaffey High School. (Not that any current Chaffey students are Vietnam veterans.)

There is no video of horses, because I saw no horses during the parade. Don’t know why.

July 4 parade.

For comparison, here is my July 4, 2023 post.

Back to promoting Bredemarket content services (in Ontario and elsewhere) later.

The Seven-Year Long Conversion Funnel For One Well-known Piece of Written Content

(Imagen 4)

As we think about the single piece of written content that received approval 249 years ago today, I want to revisit my earlier statement about how long it takes content to convert. 4 months? 17 months?

“It takes longer than three days for content marketing to yield results. One source estimates four to five months. Another source says six to twelve months. Joe Pulizzi (quoted by Neil Patel) estimates 15 to 17 months. And all the sources say that their estimates may not apply to your particular case.”

The estimates certainly didn’t apply to the United States of America.

Put your John Hancock here.

The content approved by the Continental Congress on July 4, 1776—namely, the Declaration of Independence—had a strong call to action.

“That these United Colonies are, and of Right ought to be Free and Independent States; that they are Absolved from all Allegiance to the British Crown, and that all political connection between them and the State of Great Britain, is and ought to be totally dissolved; and that as Free and Independent States, they have full Power to levy War, conclude Peace, contract Alliances, establish Commerce, and to do all other Acts and Things which Independent States may of right do.”

So how long did it take for this statement to become reality?

More than 17 months, that’s for sure.

Actually, it took over 7 years. The Treaty of Paris was signed on September 3, 1783.

You want to talk about turnover? During the time from content approval to conversion, the United States went through seven (or eight) Presidents:

  • John Hancock: May 24, 1775 – Oct. 31, 1777
  • Henry Laurens: Nov. 1, 1777 – Dec. 9, 1778
  • John Jay: Dec. 10, 1778 – Sep. 27, 1779
  • Samuel Huntington: Sep. 28, 1779 – Mar. 1, 1781
  • Samuel Huntington: Mar. 2 – July 6, 1781
  • Thomas McKean: July 10 – Oct. 23, 1781
  • John Hanson: Nov. 5, 1781 – Nov. 3, 1782
  • Elias Boudinot: Nov. 4, 1782–Nov. 3, 1783

And I didn’t even try to figure out how many “Chief Revenue Officers” we had during that time. CROs are often fired when they don’t bring in enough money, and during the 1770s and 1780s the U.S. Government was spectacularly bad at bringing in money.

But that topic is better suited for Constitution Day, which we celebrate on September 17.

But for now, enjoy the day.

Tommy.

Apple pie and Chevrolet not included.

But if you have your own content needs and Thomas Jefferson isn’t your employee, Bredemarket can provide content for tech marketers.

Stop losing prospects! Use Bredemarket content for tech marketers
Content for tech marketers.

Expanding Internal Content

(Imagen 4)

This week I’ve been expanding an internal document for a Bredemarket client.

I guess I could call it a Frequently Asked Questions (FAQ) document for salespeople, although it contains more than just the FAQs.

Jack Webb (Joe “Just the Facts” Friday). Public Domain.

Why expand it? Because we added new FAQ categories.

  • Easy enough to expand if your document is designed for expansion from the start.
  • And if you include a regular checkpoint (say, quarterly or monthly) to revisit your internal and external content.
  • It’s a lot of maintenance, but it’s worth it in the long run. Do you really want to head into 2026 with Windows 10 installation instructions?

(Right now a lot of you are making notes to scrub Windows 10 from your marketing collateral. Good for you.)

Of course, FAQs aren’t the only content that product marketers create. There are others. 

If you need help creating or maintaining your content, Bredemarket can help.

Stop losing prospects! Use Bredemarket content for tech marketers
Content for tech marketers.

The Difference Between Busy and Too Busy

Regarding your hungry people (target audience), Kristina God wrote:

“You’re not annoying people when you tell them about your offer (behind the paywall), you’re reminding busy humans.”

As someone who sells to people who sell, I reflected on what Kristina said:

“Reading Kristina reminds me that there is a difference between BUSY and TOO BUSY.

“Prospects who are too busy won’t convert.

“Those who are busy may.”

And reminded myself that while “too busy” people aren’t hungry, “busy” people still need to stop to eat to fill that gaping hole in their stomachs.

And possibly in their content.

In space, you cannot wash away the rain.

Do YOU have a content black hole?

Let Bredemarket help you take the blindfolds off. We can work together to fill your content black hole with blogs, articles, case studies, white papers, and other written words that make your prospects stop and eat.

Get started here.

(All pictures Imagen 4)

On Communities

My written content usually targets a PRIMARY channel:

This content has a new target: my Substack “subscriber chat” https://open.substack.com/pub/johnebredehoft/chat

Because unlike the others, Substack subscriber chat is DESIGNED as a community.

A community that I’m not currently utilizing, but one that I should in the future.

By the way, if you want to read my Substack, visit https://substack.com/@johnebredehoft

Authenticity, Generative Artificial Intelligence, and Product Marketing: Two Versions

When you’re…um…surfing the web, do you say to yourself, “I really hope I encounter something written by ChatGPT”?

Maybe you want to learn about a particular product and you’re hoping to find a page that begins with the words “in the ever-changing landscape of handheld gaming advances…”

Or you’re dying to read text that could be written by anybody…or by nobody.

Generative AI text is, after all, the “professional” tone.

When the CEO asks the product marketers to write just like the competitors, it’s possible that you could write exactly like the competitors by using the exact same generative AI tool (ChatGPT, because everybody uses it) and the exact same prompts to get the exact same content. This brands you as a respected member of the industry.

But whatever you do, act professionally.

Don’t randomly blow off steam like I just did.

Don’t loudly proclaim silly antiquated thoughts like “a bot should never write the first draft.”

In fact, forget that you read any of this.

Because I’m going to have it rewritten with a professional tone.

Because while your readers claim that they want to read authentic text rather than AI-generated text, they don’t know what’s good for them.

So I’m going to rewrite this the way it’s supposed to be done…with one exception. I’m going to use Google Gemini rather than ChatGPT. I know I’ll get in trouble for doing it this way.

But the text below is what readers WANT to read…right?

Navigating AI in Product Marketing: Authenticity vs. Efficiency

The increasing prevalence of generative AI in content creation presents a critical challenge for product marketers: balancing the pursuit of efficiency with the imperative of authenticity. While AI tools offer significant advantages in content generation, their widespread adoption raises questions about brand differentiation and genuine engagement.

A common misconception is that a “professional tone” inherently aligns with the output of generative AI. However, relying solely on AI, particularly popular platforms like ChatGPT, can lead to generic content that lacks unique brand voice and fails to resonate deeply with target audiences. When product marketing teams are tasked with mirroring competitor messaging, using the same AI tools and prompts as competitors can result in indistinguishable content, ultimately eroding a brand’s distinct identity within the industry.

This scenario underscores a fundamental tension: while consumers often express a preference for authentic, human-generated content, the demand for high-volume, rapid content creation pushes marketers towards AI solutions. The challenge lies in strategically integrating AI without compromising the unique elements that define a brand’s communication.

Product marketers are encouraged to consider the following:

  • Strategic AI Implementation: Instead of viewing AI as a complete content solution, leverage it for specific tasks that enhance efficiency without sacrificing originality. This could include initial research, data synthesis, or generating diverse content formats that human marketers can then refine and personalize.
  • Prioritizing Distinctive Brand Voice: Invest in developing a clear and consistent brand voice that transcends generic AI output. This involves defining unique terminology, storytelling approaches, and a tone that reflects the brand’s values and personality.
  • Authenticity and Connection: While AI can streamline content production, fostering genuine connections with audiences still requires human insight and empathy. Focus on creating content that addresses real customer needs, shares authentic brand stories, and encourages meaningful interaction.

The objective for product marketers is not to abandon AI, but to wield it judiciously. By understanding its capabilities and limitations, and by consciously prioritizing authenticity and unique brand expression, marketers can leverage AI as a powerful tool to enhance their efforts without sacrificing the crucial human element that drives true engagement.

Do you believe the future of product marketing lies in a complete embrace of AI, or a more nuanced integration that prioritizes human oversight and authentic brand voice?