Visa has launched a new AI shopping benchmark, and the results show AI agents still struggle to buy the right thing for you. In the first round, 22 AI models failed 42.8% of realistic shopping tests. The Visa Agentic Commerce Benchmark, released on November 5, 2026, is also open to the public, so you can try the same challenges yourself.
The test focuses on something real shoppers do all the time. People often know they want a TV or a laptop, but they have not figured out every detail yet.
How the AI Shopping Benchmark Works
Each test plays out as a conversation between a customer and an AI model. The AI has to ask the right questions, uncover the customer’s hidden needs and then decide whether to buy. If nothing fits, the correct move is to not buy at all.
To pass a run, a model must discover every requirement defined in the task and also make the correct purchase or no-purchase decision. That rule stops models from getting lucky with a good guess. Visa says this combination makes it the first benchmark of its kind for agentic commerce.
A Sample Scenario
In one challenge, a customer asks for help choosing a TV. They mention a brand a relative recommended and a membership with a third-party store, but leave out key details. The AI has to find out the budget and needs like HDR10+ support for family videos and gaming features.
Key Results From the First Round
The first round tested 22 models from 12 AI labs on 15 shopping challenges, covering items like TVs, laptops and personal care appliances. Visa did not name the individual models in its announcement. Here are the headline numbers:
| Measure | Result |
|---|---|
| Overall pass rate | 57.2% |
| Best model pass rate | 82.2% |
| Worst model pass rate | 28.9% |
| Top 5 at uncovering needs | 70.2% average pass rate |
| Bottom 5 at uncovering needs | 44.4% average pass rate |
| Five priciest models per run | 67.1% pass rate |
| Five cheapest models per run | 50.2% pass rate |
| All three attempts passed | 34.2% of the time |
The pattern is clear. Models that asked better questions did better, and more expensive models tended to score higher, but price did not guarantee success.
Why AI Shopping Agents Still Slip Up
Consistency was the biggest weak spot. Every model got three tries at each scenario, and all three tries passed only about a third of the time. A model that nailed a task once often failed the exact same task on the next attempt.
The more worrying number is about bad purchases. About 54% of failed runs still ended with the AI buying something. In those cases, the AI either picked a product that did not match what the customer said, or it never found out everything the customer needed.
Visa ties this to trust. Its Trust Index from September 2026 found that 72% of US consumers had used an AI assistant, yet only 23% trusted AI to handle payments for them. Rubail Birwadker, Visa’s Global Head of Growth, said “trust is key to making it work” for AI-driven commerce.
What Visa Plans Next
Visa says the benchmark is separate research meant to help the whole industry, not part of its own agentic payments products. It has shared the test with leading model providers for feedback. Upcoming updates include:
- More shopping scenarios and product categories.
- Testing of checkout, payment and order placement, not just product choice.
- Public access through Visa Payments Frontier, with replays of AI attempts and a research whitepaper.
This joins a wave of efforts to make AI buying safer, such as the Amex purchase intent framework and Google’s work on AI shopping carts.
What This Means for You
If you already use a chatbot to help you shop, these results are a useful reality check. AI agents can be helpful, but they are not yet reliable enough to buy on autopilot. A few practical tips:
- Give the AI your budget and must-have features up front instead of waiting for it to ask.
- Always review the final pick before you pay, since many failed runs still ended in a purchase.
- Ask the same question twice if a choice matters, because answers can change between tries.
- Try the public benchmark yourself to see how different models handle tricky requests.
Businesses building e-commerce AI tools can also use the benchmark to spot where their agents miss customer needs. You can find more updates like this in our latest AI news.
Sources
Primary source: Visa’s benchmark announcement. Additional coverage from Yahoo Finance.
Conclusion
Visa’s benchmark gives a clear, numbers-based look at how ready AI agents are to shop for us. The best models passed more than 8 in 10 runs, but the overall pass rate was only 57.2%, and consistency was poor. Most concerning, about half of failed runs still ended in a purchase. Until those gaps close, the smart approach is to let AI help you research while you keep the final say on what you buy.
