Back to Blog
    May 10, 20265 min read

    AI Agents Don’t Want Your Beautiful Website. They Want the Answer Fast.

    AEO Is Starting to Sound Like SEO With Less Patience

    The post lands at a weird moment for marketers. Everyone is still arguing about AI search, answer engines, citations, and whether AEO is real or just SEO wearing a futuristic jacket. Then a Google engineer writes about optimizing for AI agents, and suddenly the conversation gets more concrete: token efficiency, markdown, answer-first structure, discoverable .md files, llms.txt, and the idea that agents may skip content that burns too much context before saying anything useful. The big message isn’t subtle. If your page makes an agent work too hard, it may never get to your best part.

    The First 500 Tokens Are Becoming Prime Real Estate

    The most practical idea was brutally simple: put the answer near the top. Not after a long intro. Not after a brand story. Not after three paragraphs explaining why the topic matters. Near the top. One commenter said a lot of pages may not be losing because the information is bad, but because they take too long to get to the point. That feels painfully true. Humans may skim. Agents may truncate, summarize, or skip. Either way, burying the answer is starting to look less like storytelling and more like self-sabotage.

    This is where token efficiency becomes more than a nerdy technical phrase. Tokens are the budget AI systems spend to read, reason, and respond. If a document wastes that budget on fluff, boilerplate, repeated navigation, bloated HTML, or throat-clearing intros, the useful content has to fight for oxygen. For years, marketers were told to write longer, richer, more complete content. That advice isn’t dead, but it needs a new caveat: long content has to be structured so the answer is extractable fast.

    Markdown Sounds Smart, But the Data Isn’t Settled

    The markdown argument is the spiciest part. The article apparently advocates serving markdown over HTML and making .md files discoverable. The logic makes sense on paper. Markdown is cleaner. It strips away layout junk. It makes headings, links, lists, and code easier to parse. For docs, repositories, and technical content, a clean markdown version can feel like giving agents a well-labeled map instead of a messy shopping mall. Some people in the discussion were fully on board, especially for keeping context windows smaller and easier to archive.

    But the pushback was important. One commenter said their team ran a randomized experiment across 381 pages and six sites over 20 days, serving clean markdown to LLM crawlers and HTML to everyone else. They didn’t find a statistically significant lift in bot traffic overall. ChatGPT-User showed a directional positive signal, but not enough to call it a win. ClaudeBot even went slightly negative. That doesn’t kill markdown as an idea, but it does cool down the hype. AI systems are already pretty good at parsing HTML. Markdown may help, but it’s not magic dust.

    The Bigger Point Is Extractability, Not File Format

    That’s where the most grounded comments landed: the real issue is not markdown versus HTML. It’s whether the content is easy to extract. Clear headings. Short answers. Clean structure. Tables that make sense. Definitions up front. No buried conclusions. No bloated intros. No five-screen recipe-blog energy before the actual answer. One commenter put it plainly: less about format, more about usability for the model. That may be the safest operating principle right now, because crawler behavior is still changing and nobody has a perfect rulebook.

    For most brands, switching every blog post to a separate .md file is probably not the first move. Cleaning page structure is. Put the core answer early. Add a concise summary. Make key facts easy to quote. Use descriptive headings instead of clever ones. Keep schema, internal links, author signals, and freshness clean. Then, if you have documentation, developer resources, product specs, APIs, or comparison pages, test markdown versions where they actually make sense. Don’t rebuild the house because someone found a new kind of doormat.

    llms.txt Is Interesting, But Still Feels Like a Bet

    The post also brought up llms.txt, which sits in that strange zone between “obviously worth trying” and “are crawlers actually using this?” Some people are warming up to it as a discovery mechanism. Others are seeing almost no traffic from llms.txt or llms-full.txt even when those files are linked and discoverable. One person said they expected more activity from the “AI crumbs” they were leaving for crawlers, but the visits just weren’t there. That’s the reality check AEO needs.

    Still, low traffic doesn’t mean the idea is useless forever. Robots.txt did not become a strategic SEO staple because every marketer immediately understood it on day one. Standards take time, and agent behavior is moving fast. The practical move is not to treat llms.txt as a guaranteed growth lever. Treat it as cheap infrastructure. If your content library, docs, or product information can be summarized cleanly for agents, creating a discoverable file may be worth the small effort. Just don’t pitch it to your boss as a traffic machine yet.

    This Is Just SEO Basics Growing New Teeth

    The funniest and maybe truest comment said this feels like SEO basics evolving again: clear, structured, straight-to-answer content wins, now for agents instead of only humans. That’s the headline under the headline. AEO may sound new, but a lot of the best advice is old discipline under new pressure. Don’t waste the reader’s time. Put the answer where people can find it. Make pages technically clean. Reduce friction. Build trust. Structure content around real tasks, not internal vanity.

    The difference is that AI agents may be even less forgiving than human visitors. They don’t admire your design system. They don’t care that the intro was brand-approved. They don’t want to scroll past your newsletter popup, hero animation, and “in today’s digital landscape” paragraph. They want the answer, the evidence, and the source. Fast.

    So the best move is not panic. It’s compression. Say the useful thing sooner. Make content scannable by humans and extractable by machines. Test markdown where it fits. Add llms.txt if it’s easy and aligned with your content structure. Watch your logs, not just hot takes. And don’t confuse agent optimization with another excuse to create generic content.

    The agent era won’t reward pages that are merely long. It’ll reward pages that are useful quickly, structured cleanly, and backed by substance. In other words, the web may finally punish the worst marketing habit of all: taking forever to get to the point.

    Marketing for solopreneurs

    Turn your website into a client engine

    Mustard Seed Solutions helps solopreneurs and one-person companies build clearer websites, stronger service pages, and practical client acquisition systems.

    Book a consultation