Logo
The Rebooting
Search
Sign In
Subscribe
Newsletter
Podcast
Logo
The Rebooting
A Spotify model for AI

Oct 13, 2025

A Spotify model for AI

A Spotify model for AI
00:00
51:17

Bots are taking over the internet, previewing a new world where agents interact and humans consume bot-created content. Cloudflare, a leading content delivery network, saw this coming.

Cloudflare Chief Strategy Officer Stephanie Cohen told me that publishers reported less traffic to their sites and less human traffic. “Those numbers were 10 times worse than a decade ago.”

This has broken the open web trade of the crawling right for traffic promise. Cohen said, “For some bots, the crawl-to-traffic ratio is tens of thousands of times worse than it used to be.”

Five takeaways from our conversation:

  1. Scarcity is required. Cloudflare’s pay-per-crawl model aims to restore balance by creating enforceable scarcity. “If you don’t have scarcity, there’s no market. Changing the defaults was the only way for a market to develop.”

  2. AI engines need publishers. Models rely on fresh, high-quality publisher content to stay accurate. “If you got asked a question and didn’t know what happened over the last two weeks, your answer is going to be stupider.”

  3. Pay-per-crawl is a first step, not the finish line. Cloudflare’s system lets publishers decide whether to block, allow, or charge AI crawlers. It’s a way to test how a content economy might form. As Cohen put it, “It’s the simplest thing you can do because you can count a crawl.”

  4. The goal is the Spotify model. Cohen pointed to Spotify’s emergence after Napster as a precedent for compromise. “There’s lots of money going to creators in that model… it took a while for that market to develop.”

  5. The problem is Google. Publishers face a lose-lose choice: block Google entirely or allow AI Overviews to rewrite their work. “If they want search, they’re also in AI overviews… They can’t both crawl and pay nothing.”

Transcript

The Rebooting Show

Listen on

Apple PodcastsApple PodcastsSpotifySpotifyPocket CastsPocket CastsCastroCastro

Recent episodes

Penske's Ryan Pauley on the end of anonymous scale

Penske's Ryan Pauley on the end of anonymous scale

Nov 17, 2026

Find your superusers

Find your superusers

Nov 12, 2026

Sara Fischer: The open web will be owned by agents

Sara Fischer: The open web will be owned by agents

Nov 10, 2026

Inside the Anthropic IPO filing with CJ Gustafson

Inside the Anthropic IPO filing with CJ Gustafson

Nov 3, 2026

Google Zero by the numbers

Google Zero by the numbers

Oct 22, 2026

Reuters' Paul Bascobert on the AI production shock

Reuters' Paul Bascobert on the AI production shock

Oct 20, 2026

The anti-AI bet on print

The anti-AI bet on print

Oct 15, 2026

Inside the knowledge creator business

Inside the knowledge creator business

Oct 13, 2026

The Lever's David Sirota on challenging 'corporate media'

The Lever's David Sirota on challenging 'corporate media'

Oct 6, 2026

The Atlantic’s Nick Thompson on preparing for a post-web world

The Atlantic’s Nick Thompson on preparing for a post-web world

Sep 29, 2026

The Rebooting

Exploring the reinvention of media in a time of great change.

© 2026 The Rebooting.
beehiivPowered by beehiiv