Last Updated on November 12, 2026 by Fintech On
Global payments giant Visa has launched an industry benchmark for AI shopping that simulates shopping experiences and requires models to work through related challenges. According to an official Visa news release, nearly all test runs failed.
The first round of tests involved 22 AI models from 12 labs. These models faced 15 different challenges, including situations like purchasing a laptop or a TV. Visa noted in the news release that the overall benchmark pass rate was about 57.2%. It also specified that while the best performers passed 82.2% of tests, the worst performers passed only 28.9%.
Results of Visa’s First Tests
Visa explains that shopping is fairly difficult for AI models because buyers are sometimes unsure about what they want. Even when the desired item is clear, consumers may be uncertain about exact specifications. Models must establish these specifics through conversation and decide not to buy if the person’s requirements are unmet. The AI model must extract as much information as possible to make this decision.
AI models that extracted more information about the customer’s requirements were more likely to pass the benchmark tests. The five models that uncovered the most details averaged a 70.2% pass rate, while the five that extracted the least information passed 44.4% of the time. However, the same models usually failed a repeat scenario, showing significant difficulty with providing consistency in the shopping experience. While every model attempted three scenarios, the pass rate across all models and scenarios was only 34.2%.
In general, the most expensive models are more likely to pass. However, the higher price point does not guarantee success. The five least expensive models averaged a 50.2% pass rate, while the five most expensive averaged 67.1%.
Visa Hopes to Improve AI Utilisation in Commerce
Visa says it’s working to improve the use of AI in commerce. According to Global Head of Growth, Rubail Birwadker:
“AI agents have the potential to power the next era of commerce, but trust is key to making it work. Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
Even among failed runs, AI models made a purchase. In about 54% of these failures, the AI failed to extract enough information about the customer’s requirements, or it identified enough requirements but recommended a purchase that failed to meet them.
Visa recently concluded a study showing that customers are willing to use AI for different purposes to make shopping easier. However, trust is low with payments, as only 23% of consumers in the US trust generative AI to handle payment transactions autonomously.
