Google’s John Mueller responded to a submit on Bluesky that lamented all the trouble being wasted on AI agent accessibility as a substitute of specializing in the extra productive exercise of creating websites accessible for people.
Tactical search engine marketing
One of many constant points of the historical past of search engine marketing is that practitioners are likely to observe tactical developments if it’s obvious that everybody else is doing it. I suppose it’s human nature.
Again within the early days there was a man who constructed a listing and bought sufficient excessive PageRank hyperlinks to the house web page to get it to a couple of PageRank of seven (on a scale of 1-10). SEOs had been virtually pushing one another out of the way in which handy this man their cash. He offered 1000’s of hyperlinks from his listing, and no one thought to examine if any of the hyperlinks truly made a rating distinction. It was straightforward to examine, however no one did.
The newest tactic is LLMs.txt and creating markdown pages for AI agent consumption. Cloudflare introduced in February that their infrastructure can robotically create markdown information, largely as a means for builders to avoid wasting LLM token consumption. Their announcement acknowledged that OAI-SearchBot, OpenAI’s search crawler, was consuming markdown pages.
OpenAI’s official documentation explicitly positions its OAI-SearchBot as crawling web sites and doesn’t point out or advocate markdown information.
“OAI-SearchBot is for search. OAI-SearchBot is used to floor web sites in search ends in ChatGPT’s search options. Websites which might be opted out of OAI-SearchBot is not going to be proven in ChatGPT search solutions, although can nonetheless seem as navigational hyperlinks. To assist guarantee your website seems in search outcomes, we advocate permitting OAI-SearchBot in your website’s robots.txt file and permitting requests from our printed IP ranges under.”
Sure, it will possibly crawl markdown if it’s pointed to it however that’s not what it’s on the market for.
Stephanie Walter posted:
“Unhappy reality: we’re making the online accessible for AIs, not for individuals.
Some websites now supply a textual content model for LLMs, however nonetheless skip actual accessibility wants like correct heading construction, landmarks that display screen reader customers want.”
Google’s John Mueller responded:
“A correctly made web site works properly for AI brokers … and search engines like google and yahoo, and LLMs, and above all, for precise individuals.
In case you’re making an attempt to repair accessibility points by making a separate “agent-friendly” model, you might be simply constructing technical debt. You’ll need to redo it a number of occasions. Simply repair it.”
AI Brokers Crawl HTML
It’s a comparatively trivial factor to crawl HTML. Search engines like google have been doing it for over thirty years. Making websites accessible for everybody is sensible, particularly because it’s consumer habits indicators which have all the time pushed search engine rankings. From hyperlinks to customers looking out with a model identify, Google has persistently used consumer indicators for rating functions.
So, from the angle of search engine marketing, it makes a load of sense to make web sites accessible for everybody.
Featured Picture by Shutterstock/DETHAL
