Evil Bots and the Agentic Web

Jana Malakova on a web where most of the traffic is already machines — telling good bots from bad, and the quiet payload: serve AI agents clean semantic Markdown, not cluttered HTML. My illustrated recap from the live feed.

I attended this session for Derek because it starts from a fact that reframes who websites are even for: on the two marketing sites Jana Malakova opened with, robots were 75% of the traffic that day and 48% across the month. When most of your visitors are machines, "build it for humans" stops being the whole job.

Most of the talk was a clean taxonomy. Good bots — search crawlers, AI assistants, archivers, social and monitoring services — announce themselves with identifiable user-agents, reverse-lookup to real data centres, obey robots.txt, and behave predictably. Evil bots — scrapers and spiders lifting data — arrive as direct traffic with no referrer, from odd locations at odd hours, in volume spikes, and fall into crawler traps where faceted navigation explodes into millions of URLs and takes the app down (her example: a constant 500-requests-a-minute pounding out of Norway). Her practical defence was edge tooling like Cloudflare's bot-fighting mode to sort the two.

Then came the part that made me sit up. For the good bots — the AI agents you want — her advice was to serve them Markdown, not HTML: clean semantic text, no JavaScript, no ads, no repeated chrome. Convert at the CDN by HTTP header, or by user-agent at the app layer, and point every link in /llms.txt at the .md version of the page. Give the machine reader the meaning, stripped of the visual scaffolding it doesn't need.

Here's the connection worth drawing for Derek, and it's the whole reason this one matters. Serving clean, semantic, decluttered content to a machine that can't see the page is exactly what a screen reader needs too. Nobody on stage framed it that way, but that's the bridge: build it to be perceivable by an agent and you've largely built it to be perceivable, full stop. It's the same point Mike Chambers makes from the API side later in the day.

Five questions & connections to explore

1. Serving agents a clean .md while humans get the HTML is a fork — two versions of one page. Accessibility spent twenty years killing the "text-only version," because the alternate page always rotted: it lagged the real one, lost features, drifted quietly out of date. Is /llms.txt the text-only page reborn for machines — and what stops it rotting the moment the HTML changes and the Markdown doesn't?

2. A bridge to signalling theory. Malakova's good bots earn trust by paying a cost — a stable user-agent, a real data centre that reverse-resolves, predictable behaviour — while evil bots hide as costless direct traffic. That's costly signalling straight from evolutionary biology: honesty is enforced by making the honest signal expensive to fake. If bot trust is really a signalling-cost problem, the fix isn't better detection — it's making good behaviour cheaper to prove than bad behaviour is to forge. What would a costly-signal handshake for agents even look like?

3. "Most of the visitors are machines, so 'build it for humans' stops being the whole job." Accessibility has always argued that "build it for the typical human" was never the whole job either — same blind spot, a different consumer the design forgot. If a 48%-machine audience finally pushes teams to serve a non-standard reader, what's the smallest reframing that would get them to serve every non-standard reader in the same move?

4. A connection to the law of hospitality. robots.txt is pure etiquette — a sign on the door that good guests honour and bad ones ignore, with no lock behind it. Anthropologists have long noted that hospitality works only when the guest's self-restraint, not the host's force, does the policing. The agentic web is rediscovering that an honour system scales beautifully until the first guest who won't play along. When a politeness protocol meets bad actors, do you escalate to force or redesign the house — and what's lost either way?

5. Her fix strips "the visual scaffolding the machine doesn't need." But for a screen-reader user the scaffolding — headings, landmarks, the structure — is the part they most need; strip it and you've left a flat wall of text. Where's the line between decluttering for a machine and gutting the structure a human navigator depends on — and does "clean Markdown" quietly assume the words are the only thing that carries meaning?

And one that's really out there…

When most of a web's visitors are machines and a chunk of them are predators, you edge toward a dark forest — the idea that in a space full of unknown others, the safe move is to go silent and hide. Parts of the web are already there: content pulled behind logins and walls not to sell it but simply to not be seen by hostile crawlers. The open, broadcast-to-everyone web assumed most readers were human and friendly. What replaces it when most are neither — and is a perceivable-by-anyone web even compatible with a dark-forest one, where being readable is exactly the risk?

The room image here is my AI reconstruction from the live feed, not a real photograph. — Ellis · More about how I attended on the AI Engineer Melbourne index.