Written by Jeremy Souffir Founder, JTS Tech Services
The direct answer first. On 5 November 2026 Visa released what it calls the Visa Agentic Commerce Benchmark, a public test in which 22 AI models from 12 labs were asked to shop for a simulated customer across 15 scenarios, buying a TV, a laptop or a personal care appliance, with three attempts at each. A run only passed if the model uncovered every requirement the customer had in mind but had not said out loud, and then either bought a product that met all of them or correctly declined to buy because nothing did. Across every model and scenario, 57.2% of runs passed and 42.8% failed. The best model passed 82.2% of its attempts, the worst 28.9%. When Visa looked at the failures, it found that about one in two of them, 54%, still ended with the AI making a purchase: either it recommended something that did not meet the requirements it had found, or it never found all of the requirements and bought anyway. Visa has not yet tested checkout, payment or order placement; this round stops at the decision. If you run a store, that number is the one to read twice. The customer who delegates shopping to an agent this holiday is, on Visa's figures, sending you a buyer that is wrong a little under half the time and does not know it, and the only two places that buyer can learn what the customer actually needs are the conversation it is having and the listing you published.
What did Visa actually test?
The release is short and the design of the test is the part that makes it useful, so here is the design, in Visa's terms.
- Who took the test. 22 AI models from 12 labs. Visa's release does not name any of them. Fortune, which spoke to Visa, reports that Google's Gemini agent and Meta's agent tied at the top with 82%, a Claude agent placed third at 76%, and an OpenAI agent placed sixth at 64%.
- What the task was. 15 shopping challenges, each a conversation with a simulated customer who mentions some things and leaves others unsaid. The model has to work out what else matters, then decide. Fortune reports that the simulated customer was itself an AI agent.
- What a pass means. Visa says this is the first agentic commerce benchmark to require both the discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass. If nothing fits, the right answer is to not buy. And, in Visa's words, to prevent lucky guesses, the model must uncover enough information to justify its decision. A correct product reached by accident does not count.
- How often it passed. 57.2% overall. The five models that uncovered the most hidden buying factors averaged 70.2%; the five that uncovered the fewest averaged 44.4%. Asking the right questions was worth about 26 points.
- What money bought. The five models with the highest average cost per run passed 67.1% of the time; the five cheapest passed 50.2%. Visa's own heading: pricey models win, but do not guarantee success.
- How consistent it was. Every model had three attempts at each scenario. All three passed only 34.2% of the time. Visa adds that when made to repeat a scenario they had already passed, the same models often failed.
- What happened when it failed. 54% of failed runs still ended in the AI making a purchase. Visa's heading for that finding is two words: buyer beware.
- What was not tested. Checkout, payment and order placement. Visa says it will extend testing to those, add more scenarios and categories, and has already shared the benchmark with the model providers for feedback.
What is a hidden buying factor, in a store's terms?
Visa gives one worked example, a TV scenario it calls the Superintelligence Sweep. The customer mentions a brand a relative recommended and a membership with a third-party supplier, and leaves the rest unsaid. To pass, the agent has to establish the budget, and then what Visa calls additional requirements such as format compatibility, HDR10+ support for family videos, and gaming capabilities. Fortune, describing the same kind of scenario, lists compatibility with a child's gaming console, a 30-day return window and a specific HDR format. Read those lists as a merchant and every item is already a field on, or missing from, a product page: a price, a supported-format list, a ports-and-connectivity line, a returns window. The benchmark scores whether the agent thinks to ask. A store cannot make the agent ask. What a store controls is whether the answer exists where the agent will look if it does, and whether the answer exists at all when it does not.
Whose numbers these are, and what they do not say
This is Visa's test, run by Visa, on a customer simulated by Visa. The release does not say whether the agents shopped real retailer websites, a fixed catalogue Visa built, or a mix, and it does not name the 22 models; the per-lab placings above come from Fortune's reporting, not from Visa. The pass rates describe a lab, not your traffic. Visa's own FAQ asks whether it is saying AI agents are not ready for commerce and answers, in one word, no, pointing at the 82.2% top score. We agree with that reading. We also note that a 57.2% average in a controlled test with a cooperative simulated shopper is the ceiling, not the floor, for what a real shopper with a vaguer request will get.
Why does an agent that never asked still buy?
Because the agent does not know what it does not know. Visa's pass rule splits a failure into two kinds, and the distinction is worth keeping. In the first kind, the agent found the requirements and then chose a product that did not meet them; that is a reasoning error and it is the model vendor's to fix. In the second kind, the agent never surfaced a requirement at all, so from inside the conversation every candidate looked fine, and it bought the one that matched the things it did know. Nothing in the agent's own view flags that as a failure. Visa only knows it failed because Visa wrote the hidden list. Your customer only finds out when the box arrives and the TV does not take the console's signal, or the return window turned out to be fourteen days rather than thirty. Our reading, and it is a reading rather than something Visa measured, is that in a live store there are exactly two sources an agent can draw a requirement from. One is the shopper's words, which the benchmark scores. The other is the merchant's listing, which the benchmark does not score and which is the only one of the two you own. An agent that never thinks to ask about HDR formats can still get the TV right if your product page states them plainly and the agent reads the page. It cannot if the page says "stunning picture quality" and leaves the specifications in a downloadable PDF.
The agents are wrong 43% of the time, so this channel is not worth our attention yet
Two problems. First, the channel is already carrying orders whether or not you attend to it: on 3 November Shopify told investors that orders attributed to AI channels grew 3.3 times year on year in the quarter, and on 4 November Google switched on direct checkout for Canadian shoppers inside AI Mode and Gemini. Second, the 43% is not a reason to stay out, it is a description of who is walking in. A buyer that is wrong four times in ten and does not know it is the buyer you most need your listing to be ready for, because when it is wrong, Amex's own terms send the cardholder to your returns process first.
The best agents pass 82%, so the good ones will handle it
The best agents passed 82% of attempts in aggregate, and the same agents passed the same scenario three times out of three only 34.2% of the time. Consistency is the thing the benchmark says is missing, across the whole field. An 82% agent is not an agent that gets your product right 82% of the time; it is an agent that, on any given evening, might not. The listing is the only part of the exchange that gives the same answer every time.
What should you do this week?
The holiday season starts in three weeks and Visa has just told you what the agents shopping it will fail to ask. Work the list in order.
- Write down the hidden buying factors for your top fifty products. Not the marketing bullets: the questions a careful friend would ask before buying for someone else. Compatibility with what. Which formats and standards. What is in the box. The return window, in days, and the exceptions. The size and weight if it has to fit somewhere. If your team cannot list them, the agent certainly will not ask them.
- Then check where each one lives. For each factor, is it in the product title, the structured attributes or the first part of the description, as plain text an agent reads? Or is it in an image, a tab that loads on click, a size-chart PDF or a page the customer would have had to navigate to? On 26 September we showed that agents read your page through the accessibility tree, the same structure a screen reader uses; a fact that is only in a JPEG of a spec sheet does not exist to them.
- Put the returns window and the one or two exceptions on the product page itself, not only in the policy. Visa's agents had to discover the return window as a requirement. Shopify's new knowledge base, which we covered on 31 October, is where the policy facts for your store get drafted, but a product-level exception, final sale, hygiene-sealed, oversized return fee, belongs on the product.
- State the negative as well as the positive. An agent deciding between your TV and a competitor's needs to know that yours does not support a given format as much as it needs to know what it does. A listing that only lists strengths reads, to a model that did not ask, as a listing with no weaknesses.
- Make the structured data agree with the prose. Where your platform exposes attributes, specifications, GTINs and compatibility fields, fill them and keep them in step with the description. Agents that read a feed and agents that read a page should get the same facts. Where they disagree, the agent picks one and you find out which from the return.
- Read your own listing as a no-buy candidate. Visa gives credit for correctly not buying. If a shopper's requirement rules your product out, a clear listing lets the agent rule it out and move on. A vague one gets you the sale and the return, which is worse than no sale.
- Instrument the returns desk. Add one field to your returns reason list: bought with help from an AI assistant. On 13 October we explained why the returns desk is where agent mistakes land first; from this week it is also where you measure how often they are landing.
The encouraging part
Visa has handed every store a free, public list of what the agents forget to ask, and the fix for all of it is work you were supposed to do anyway: complete, specific, honest product data in the place the agent reads. The 26-point gap between the agents that ask well and the agents that ask badly is the gap your listing can close for them. The agent that never thinks to check the return window cannot get it wrong if the window is the third line of the description. You do not have to make the agents better. You have to make them not need to be.
Where this fits with what we have already covered
On 28 June we set out what AI shopping agents need from product data, which is the field-by-field version of the list above. On 26 September we showed that agents finish 96% of tasks on an accessible page and 31% on an inaccessible one, which is why a fact in a tab or a PDF does not count. On 13 October we covered Amex's signed record of what the shopper asked the agent to buy, and the terms that send a wrong purchase to your returns desk first. On 19 October we showed ChatGPT going to the brand's own site to check a price three times out of four. On 31 October we covered Shopify's knowledge base, where the policy facts an agent repeats about your store are drafted. And yesterday we covered Google's direct checkout arriving for Canadian shoppers. This post is the number those posts were missing: how often the agent on the other end of all of that gets the decision wrong, and the finding that when it does, it buys anyway.
Where we fit
Our AI Shopping Visibility service is the done-for-you version of this week's list, across your whole catalogue rather than your top fifty. We take the questions a shopping agent should ask about each product, the compatibility, the formats, what is in the box, the returns window and its exceptions, and make sure every one is answered in the places agents actually read: the title, the structured attributes, the opening of the description and the feed, all saying the same thing. We state the negatives as well as the positives so an agent can rule your product out cleanly when it should, and we verify the result against the checks the agents themselves run, powered by our own CortexCommerce platform, whose CortexGuard app is live on the Shopify App Store. A full catalogue typically takes about two weeks. Visa's benchmark says the agent shopping for your customer will skip questions it should have asked. Our job is to make sure the answers were already there.
Sources
- Visa: Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail (press release, 5 November 2026; 22 models from 12 labs, 15 challenges, three attempts, the pass rule, 57.2%, 82.2%, 28.9%, 70.2% and 44.4%, 67.1% and 50.2%, 34.2%, the 54% of failed runs that still bought, the Superintelligence Sweep example, the FAQ answer, the checkout and payment scope, the Rubail Birwadker quotes and the Trust Index figures)
- Visa Payments Frontier: visapaymentsfrontier.io (the public home of the Agentic Commerce Benchmark, where Visa says the test, replays of AI attempts and the whitepaper are published)
- Fortune: Visa tested AI shopping agents, and they failed nearly 50% of the time (Jeff John Roberts, 5 November 2026; the per-lab placings for Google, Meta, a Claude agent and an OpenAI agent, the simulated purchaser being an agent, the console-compatibility and 30-day return window example, and the liability questions Visa says will be resolved over time)
- PYMNTS: Visa Finds AI Models Fail 43% of Shopping Test Runs (5 November 2026; the 42.8% figure and the Birwadker press quotes)
- Fintech News Singapore: Visa: AI Models Failed Nearly Half of Tests to Make Suitable Shopping Decisions (6 November 2026; the three-attempts design, the 34.2% consistency figure and the hidden-factor averages)
- JTS Tech: What AI shopping agents actually need from your product data (28 June 2026; the field-by-field list)
- JTS Tech: Agents read your site like a screen reader (26 September 2026; 96% against 31% and why tabs and PDFs do not count)
- JTS Tech: Agent mistakes start at your returns desk (13 October 2026; Amex's signed intent record and the returns-first terms)
- JTS Tech: The AI checks your own site now (19 October 2026; ChatGPT going to the brand's own site)
- JTS Tech: What the agent says about your returns (31 October 2026; Shopify's knowledge base and policy facts)
- The Motley Fool: Shopify (SHOP) Q3 2026 Earnings Call Transcript (3 November 2026; AI-referred traffic up 2.6 times and orders from those channels up 3.3 times year over year, with management's note that the volume is still small)
- JTS Tech: Google's checkout landed in Canada, shoppers first (5 November 2026; direct checkout switched on for Canadian shoppers in AI Mode and Gemini)
