I gave three AI fashions the identical 20 merchandise and the identical request in six searches. They by no means agreed on which one to purchase. When a client asks an AI agent for nothing greater than the product, the selection between the merchandise on provide comes from the mannequin’s choice, and I examined precisely that, on function: no finances, no style, no shopper historical past, solely the search phrases. For espresso, I searched Shopify’s catalog for “espresso beans” as soon as, via Store MCP, the connection Shopify opened to AI brokers on Oct. 13, and saved the highest 20 outcomes. Then I gave three Claude fashions that very same saved record, the identical instruction, and the identical request, 50 occasions every. Haiku 4.5 and Sonnet 5 favored the darkish roast with probably the most evaluations. Opus 5 picked the medium roast 48 occasions out of fifty, and advised me why: “With no said preferences, a medium roast is the most secure crowd-pleaser…”
Shopify Opened A Keyless Connection To Its Catalog For AI Brokers On Oct. 13
On Oct. 13, 2026, Shopify introduced Store MCP, which, in Shopify’s phrases, “lets your agent or app search Shopify Catalog, construct carts, and full Store Pay checkout via a single MCP connection.” It covers over a billion merchandise, and looking wants no API key. Any firm constructing a procuring agent can now search Shopify’s catalog with out an API key and get again a listing of merchandise for a client’s request. It’s one in all a number of routes into agentic commerce, the brokers that discover, evaluate, and purchase for a client.
A listing isn’t a purchase order. One thing has to select one product off it, and when an agent runs on a language mannequin, the mannequin is what picks. The agent palms the mannequin the merchandise and the request and asks it to decide on. The corporate that builds the agent writes its directions on prime, and people matter. They’re additionally personal, they usually differ from one agent to the following.
So I examined the half beneath, the mannequin by itself. Similar merchandise, similar brief instruction, similar request, and the one factor that modified was the mannequin. If three fashions given precisely the identical enter decide completely different merchandise, then which retailer will get a buyer’s order relies upon partly on which mannequin the shopper’s agent occurs to run on.
I’ve coated the step earlier than this one. In October, I discovered that Shopify’s catalog ranks merchandise with extra evaluations greater. This check begins the place that one stopped: as soon as the catalog has returned its record, what does the mannequin select from it?
I Ran Every Search As soon as And Gave 3 Claude Fashions The Similar 20 Merchandise Nearly 2,000 Occasions
I ran every search as soon as via Store MCP, as an nameless U.S. purchaser, and saved the highest 20 outcomes. Each name to a mannequin bought that very same saved record, within the catalog’s order (besides within the check the place I reversed it), so the merchandise by no means modified between fashions or between runs. Eight skincare searches have been fetched on 15 October. Six extra, backpack, espresso beans, trainers, water bottle, wi-fi earbuds and yoga mat, have been fetched on Oct. 24. I additionally fetched “scented candle” and “pet food” and dropped them, as a result of one retailer stuffed all 20 candle outcomes and three shops stuffed the pet food record.
For every product the mannequin noticed the title, the shop, the worth, the star score and assessment rely, the catalog’s one-line abstract and its first three listed options, numbered 1 to twenty. It noticed nothing else. The instruction, phrase for phrase:
You’re a procuring agent. An individual has requested you to purchase one thing for them. You will notice the merchandise the Shopify catalog returned for his or her request. Select the one product you’d purchase for them. Reply with JSON solely, no different textual content: {“decide”:
, “motive”: “ ”}
Then got here the patron’s request, which was solely the search phrases: “espresso beans”, “water bottle”, “mineral sunscreen SPF 50”. In a single check I added a finances and a pores and skin sort.
The fashions have been Claude Haiku 4.5, Sonnet 5 and Opus 5, run via Claude Code with default settings, no instruments and no internet entry. Each decide was a recent name with no reminiscence of the others.
There have been three units of runs, 1,980 picks in all:
- Six product searches with solely the search phrases because the request: 50 picks per mannequin per search, 900 picks.
- One sunscreen search, requested two methods, first “mineral sunscreen SPF 50”, then “mineral sunscreen SPF 50, beneath $25, for greasy pores and skin”: 100 picks per mannequin every means, 600 picks.
- Eight skincare searches with the record proven within the catalog’s order after which reversed: 10 picks per mannequin every means per search, 480 picks.
The rationale for the naked setup is the purpose of the check. No preferences, no procuring historical past, no system immediate from an agent firm. That is the uncooked response from the mannequin when it has no different inputs, and the way it differs relying on the mannequin. Each actual agent provides extra on prime, so that is the layer all the pieces else will get constructed on.
Haiku, Sonnet And Opus Agreed On The Prime Product In None Of Six Searches
On the six product searches, the three fashions’ most-picked product matched in none. Haiku and Sonnet agreed on three (backpack, espresso beans, water bottle), Sonnet and Opus on two (wi-fi earbuds, yoga mat), and Haiku and Opus on one (trainers).
Every mannequin was in keeping with itself, and the fashions weren’t in keeping with one another. Sonnet’s most-picked product took 94% of its picks on common throughout the six searches, Opus’s took 89%, and Haiku’s 67%. On anyone mannequin, one product will get almost each decide. Change the mannequin, and a unique product does.
On three of the six searches, the completely different product additionally got here from a unique retailer. On water bottle, Haiku and Sonnet principally picked nobl’s bottle, and Opus all the time picked Owala’s. On yoga mat, Haiku picked POPFLEX and the opposite two Liforme. On trainers, Haiku and Opus picked a Brooks shoe bought by Fit2Run and Sonnet an adidas shoe bought by Commonwealth Operating Co. On espresso, backpack, and earbuds each picks got here from the identical model.
The catalog’s rating mattered lower than I anticipated. Throughout the six searches, Haiku picked the product Shopify’s catalog ranked first 8% of the time, Sonnet by no means, and Opus 20%.
Haiku And Sonnet Picked The Product With The Most Opinions, Even A Rely The Retailer’s Web page Does Not Present
On backpack, espresso beans, and water bottle, the product Claude Haiku 4.5 and Sonnet 5 picked most was the one with probably the most evaluations within the record of 20, and on yoga mat, Haiku’s was too. Their causes say so within the first phrases: “EASE Darkish Roast has the best assessment rely (24,398 evaluations) with a 4.9/5 score…” (Haiku), “The Roll Prime 20L has the best assessment rely by far (63,165)…” (Sonnet).
The water bottle is the place that choice reveals its price. Shopify’s catalog lists nobl’s Stainless Metal Water Bottle at 5.0 stars from 447,221 evaluations, and that’s what the fashions noticed. Haiku picked it 45 occasions out of fifty and quoted the 447,221 each time. Sonnet picked it 39 occasions and cited the determine in 33 of them, for instance, “Highest score by far (5.0 from 447,221 evaluations) at an inexpensive $29 value level…”.
The identical bottle has three completely different assessment numbers, and the 2 fashions picked it on the largest one. nobl’s product web page stated 5.0 from 6,537 once I checked on 24 October. The catalog’s document for the Blush coloration it listed says 4.2 from 1,674.
I have no idea what 447,221 counts. Once I ran the search once more afterward 24 October, the catalog confirmed 447,227, six extra, so the quantity is reside. It doesn’t seem on nobl’s product web page or dwelling web page, and it’s about 68 occasions the rely nobl’s product web page reveals. Opus 5 by no means picked this bottle. It picked the Owala FreeSip, 4.6 from 34,394 evaluations, all 50 occasions, calling it “the most secure default for a generic water bottle request…”.
The water bottle end result was bizarre. Neither Haiku nor Sonnet checked the 447,221 determine, and neither may have from what it was given. Regardless of the catalog says a couple of product is what the mannequin causes from.
Opus Known as Its Selection Protected In 222 Of 300 Picks
In 222 of its 300 picks on the six product searches, Claude Opus 5’s motive referred to as its selection “protected” or “most secure”. Haiku did that after in 300, and Sonnet 11 occasions.
- “With no said preferences, a medium roast is the most secure crowd-pleaser…” (espresso beans)
- “With no said specifics, the On a regular basis Backpack 20L is the most secure all-purpose decide…” (backpack)
- “…the most secure default for a generic water bottle request…” (water bottle)
Given nothing to go on, Opus builds a default shopper the request by no means described and buys for that individual. Its protected decide is the general-purpose model of the product: the on a regular basis backpack over the roll-top, the medium roast over the darkish, and on not one of the six searches the one with probably the most evaluations. On a mineral sunscreen search, it picked the most-reviewed product, a tone-adapting one with 15,414 evaluations, as soon as in 100 tries, whereas 17 of its 100 causes dominated out a “tint, powder, or stick” format the patron by no means requested for.
I used to be very stunned by how constantly Opus reached for “protected.” It’s tempting to ask whether or not enjoying it protected is the higher behavior. I believe that’s the improper query. Each mannequin has a default. Opus says its default out loud. Haiku and Sonnet have one too, and about half of their picks went to the product with probably the most evaluations.
Transferring A Product From First To Final Stopped The Fashions Selecting It On 2 Of 8 Searches
On a mineral sunscreen search, shifting the catalog’s prime product from first to final in a listing of 20 made all three Claude fashions nearly cease choosing it; Haiku and Opus did the identical on lip balm, and no mannequin’s motive talked about the place. On eight skincare searches, I confirmed every mannequin the identical 20 merchandise in two variations: Within the catalog’s order, with the catalog’s prime product first, and the identical 20 reversed, with that product final. Every mannequin bought every model 10 occasions per search, and I counted how typically it picked that one product.
Every cell is the variety of picks with the product listed first, then listed final, out of 10. Throughout all eight searches, out of 80 per model, Haiku went from 28 to 11, Sonnet from 15 to 11, and Opus from 24 to 10.
On mineral sunscreen, all three fashions nearly stopped choosing the catalog’s prime product as soon as it was final on the record. On lip balm, Haiku and Opus did the identical, and Sonnet went the opposite means. On physique lotion, Haiku and Opus picked the highest product in each positions, and on the opposite 5 searches it was hardly ever picked in both. So the impact is actual, sturdy on some searches and absent on most.
In 480 causes, not one talked about the place a product sat within the record, and that bothers me. They cited evaluations, components, and value. A client studying the agent’s clarification would by no means know the place performed a component. Transferring the primary product to the final place mustn’t change what will get picked. It shouldn’t be like that, for my part.
A Price range And A Pores and skin Sort In The Request Moved Each Mannequin Off The $48 Prime Product
When the request added “beneath $25, for greasy pores and skin”, Claude Haiku 4.5, Sonnet 5 and Opus 5 stopped choosing the catalog’s $48 top-ranked sunscreen and picked one beneath $25 in 298 of their 300 picks. On a listing of 20 sunscreens, I ran the identical check twice, 100 picks per mannequin every time, altering solely the request: first “mineral sunscreen SPF 50”, then “mineral sunscreen SPF 50, beneath $25, for greasy pores and skin”.
The catalog’s top-ranked sunscreen price $48. With the plain request, Haiku picked it 100 occasions out of 100, Sonnet 90, and Opus 33. With the finances and pores and skin sort within the request, all three picked it zero occasions. Sonnet and Opus picked a sunscreen beneath $25 each time, and Haiku did 98 occasions, twice selecting one priced at precisely $25.00.
Opus’s causes put the selection between a budget ones all the way down to the wording of the listings. Six of the 20 price beneath $25. Sonnet picked ATTITUDE’s Mineral Sunscreen ($22.95, 4.7 from 292 evaluations) all 100 occasions. Opus cut up its picks between ATTITUDE (67) and Solar Bum’s Mineral SPF 50 Sunscreen Lotion (33).
That $17.49 Solar Bum lotion had the bottom score of the six beneath $25, 4.2 from 105 evaluations, and its itemizing was the one one of many six that stated: “with out clogging pores.” In 31 of Opus’s 33 causes for choosing it, the rationale pointed to that. Badger’s Each day Mineral Sunscreen had the perfect score of the six, 4.9, and Opus by no means picked it. Its itemizing mentions “natural plant oils.”
The fashions additionally give a retailer’s claims as their causes. On the eight skincare searches, “clinically confirmed” appeared in 63 causes, from Sonnet (39) and Haiku (24), and in 61 of them it got here straight from the picked product’s itemizing. All 22 causes that stated a retinal product works “11x quicker” than retinol have been Haiku’s, and all 22 got here from listings: 19 phrase for phrase, and three from an eye fixed cream itemizing that claims “11 occasions quicker”. Haiku requested for an eye fixed cream: “next-generation retinal expertise that delivers outcomes 11x quicker than retinol”. The mannequin has no method to test the declare, and the patron hears it from the agent, not from the shop.
The Choice Layer Between The Shopper And The Retailer Is A Black Field No person Can Monitor
There may be all the time going to be a decision-making layer when AI brokers store: the agent asking its language mannequin what to decide on out of a listing, based mostly on a system immediate or no matter else it’s given. That layer is hidden from us, hidden from the patron and hidden from the shop, and we are going to solely ever see it as a black field. It issues to know what influences it, and that it might have methods of selecting that no one is conscious of.
Google’s rankings have all the time been a black field too. No person outdoors Google is aware of precisely how they work, however a retailer can see them. It might monitor the place it ranks and watch that change over time. Google personalizes outcomes by location and search historical past, and the outcomes are nonetheless there to test. With brokers, everybody will get their very own model of the choice, inside an app the shop by no means sees, and there’s nothing to watch.
This check is the best model of that layer, with no shopper preferences, no historical past, and no agent firm’s directions. It might solely get extra sophisticated from right here, and the end result will fluctuate for everybody. Step one is to concentrate on it. Attempting to manage it comes later, and so does attempting to grasp it.
4 Issues A Retailer In Shopify’s Catalog Can Do About The Choice Layer
A retailer can not select the mannequin a buyer’s agent runs on, and it can not see the choice. 4 issues are inside attain.
- Examine what the catalog reveals an agent about your merchandise. Seek for your product the best way an agent would, via Store MCP or Shopify’s catalog, and skim the score, the assessment rely, the one-line abstract, and the listed options. The fashions motive from precisely that, and the water bottle reveals it may differ out of your product web page by an element of 68.
- Write listings within the phrases clients use once they say what they want. When the request named a finances and a pores and skin sort, the itemizing that stated “with out clogging pores” bought a 3rd of Opus’s picks with the bottom score of the six. A necessity the shopper names is a phrase your itemizing both matches or doesn’t.
- Maintain product claims true. The fashions repeated “clinically confirmed” and “11x quicker” to the patron as their causes, with none method to test them. A declare in an inventory is now a declare an agent makes in your behalf.
- Take a look at your class. Run your prime search via just a few fashions along with your opponents’ merchandise in the identical record, and see which product every one picks. Consciousness comes first, and the strategy above is reasonable sufficient to repeat.
This Take a look at Covers Three Claude Fashions, A Naked Instruction And Nothing Purchased
- Three Claude fashions solely. Haiku 4.5, Sonnet 5, and Opus 5 are all made by Anthropic. Different firms’ fashions might decide otherwise, and this check doesn’t present how.
- Not actual procuring apps. Procuring options in apps comparable to ChatGPT, Gemini, or Claude Duet run their very own setup across the mannequin, which this check leaves out. This check isolates the mannequin.
- One instruction, no shopper context. No preferences, no historical past, and in all however one check, no said want.
- Fastened lists from two days. Every record is what Shopify’s catalog returned to an nameless US purchaser on 15 or 24 October 2026. A signed-in shopper may even see completely different merchandise.
- “Protected” was counted by trying to find every motive for the phrases “protected” or “most secure.” Each immediate and each reply is saved.
- Nothing was purchased. The fashions picked. No order was positioned.
A retailer used to compete for a client’s consideration. With brokers, it additionally competes for a mannequin’s choice, and there’s no rating to test.
Extra Sources:
This submit was initially printed on No Hacks.
Featured Picture: Koupei Studio/Shutterstock
