The internet met travel back in 1995. Early pioneers like PC Travel and Internet Travel Network (later GetThere) promised a golden era of self-service, transparency, and seamless search. Expedia and Travelocity followed in 1996, offering consumers direct access to flights and hotels. The dream? A truly open digital marketplace. The reality? We’re still chasing it. Nearly 20 years ago, AC/DC nailed it with Dirty Deeds Done Dirt Cheap.
Cloudflare’s Move:
Fast forward to today. Cloudflare, one of the internet’s most important sentries, recently blocked all AI crawlers. No more free-for-all bot scraping. No more silent vacuuming of the world’s data to feed hungry models. For travel, this is both overdue and complicated.
Back in 1995, travel data was designed for humans: readable web pages, meant to be compared and pondered. Data structures were messy, inconsistent, and unstandardized across agencies and aggregators. Now, in the age of AI bots, that same human-focused data is cannon fodder. Bots misread, mis-categorize, and misrepresent it. The result? Consumers face more searches and less clarity, while the industry bears spiraling costs trying to protect or parse whatever trustworthy data remains.
Cloudflare’s AI block should serve as a wake-up call. If even the defenders of open, efficient web infrastructure see AI crawling as a net harm, what does that tell us about the state of the internet? Travel’s proprietary content makes this issue even trickier. On the one hand, content owners gain protection. On the other, there’s still no ethical, efficient way to consolidate data for legitimate comparison.
Messy Travel Data
The core issue? The travel industry never built machine-readable standards. There’s no unified framework for fares, schedules, or ancillary services. Governance on data access and cost is minimal, and big players often profit from the chaos. Attempts like New Distribution Capability, One Order, and XML schemas often felt more like turf protection than real solutions.
AI promises the next big leap, but don’t bet on it yet. Expect exponential query growth without better outcomes, rising costs for redundant crawls, and persistent consumer frustration. An insider revealed: agentic AI shopping tests increased infrastructure costs tenfold without moving the needle on conversions.
Cloudflare took a stand. The travel industry? Silence. Where’s the initiative for true AI-era standards, ethical search models, and leadership saying, Enough chaos let’s build trust? The status quo favors the powerful, but it fails travelers. Without urgent action, messy travel data will continue to hamper AI progress regardless of technological hype.
Side notes to consider: The first web crawler indexed about 110,000 sites in 1993. Today, a single AI bot can scrape that many pages in under a minute if Cloudflare allows it. The original Googlebot crawled at one page per second. Modern AI scrapers hit hundreds per second unless throttled.
Schema Selected:
