AI agents can now spend hours, even days, working through a coding project or a research task on their own. But a new benchmark suggests their AI agent decisions along the way are far shakier than their final scores make them look. Even the best model picked the better path only about 60 percent of the time.
The benchmark is called Taste-Bench, and it comes from researchers at Microsoft and City University of Hong Kong. Their paper, first posted in September, argues that good agents need something closer to judgment, which the team calls “taste.”
Why AI Agent Decisions Are Hard to Grade
Most agent tests only ask one question: did the task get finished? That misses the dozens of small choices an agent makes in between, like which idea to try first or which version of the code to build on. A bad call often looks perfectly fine at the moment and only shows its cost much later, after time and compute are already spent.
Grading those choices by hand would be slow and expensive. You would need experts in each field to review every step, and that does not scale.
Using hindsight instead of human graders
The researchers found a clever shortcut. Agent systems often try the same task several times, and those attempts eventually split in different directions. Since the final result shows which branch worked better, each split becomes a ready-made test question.
The team calls these splits “decision forks.” To test a model, they freeze the run at the fork, hide everything that came after, and ask the model to pick the better of two paths.
How the New AI Agent Benchmark Works
Taste-Bench holds 502 questions built from thousands of scored agent runs on coding and AI research work. The raw material includes SWE-bench Pro tasks and a public set of agent transcripts released by the evaluation group METR. Human reviewers checked the automatically generated labels and agreed with almost all of them.
There are two kinds of forks in the set:
- Parallel forks, where two separate attempts at the same task go different ways.
- Detour forks, where one agent heads down a wrong path, then corrects itself later in the same run.
Detour forks turned out to be harder for every model. They ask whether a model can spot a mistake earlier than the agent that actually made it.
How the Top Models Scored
The team tested 14 current models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families. Each question was asked twice with the answer order flipped, and a model only got credit if it was right both times. That means pure guessing would land at about 25 percent.
| Result | Score |
|---|---|
| Best model (GPT-5.6 Sol) | 59.7% |
| Runner-up (GPT-5.5) | 59.5% |
| Average when the key clue appears early | 62.3% |
| Average when the key clue appears late | 21% |
| Random guessing baseline | About 25% |
Every other model scored well below the two GPT models. None came close to acing a test where each question is just a choice between two options.
Thinking longer did not help
The most telling result is how fast accuracy falls when the deciding evidence only shows up much later in a run. In those cases, models scored about as well as random guessing. Turning up the reasoning budget barely changed the scores, and models actually spent the most thinking on the questions they missed most often.
The researchers read this as a sign that the needed clue simply is not visible yet at those forks. Getting them right would mean predicting future work, not reasoning harder about the past.
Good News: Taste Can Be Trained
The paper does not stop at bad news. The team trained a smaller student model, built on Qwen3.6-27B, to copy the judgment of a teacher model that had seen how each fork turned out. On tasks it had never seen, the student’s accuracy rose from 30 percent to 47.9 percent.
That gain carried over to real work. When the student’s advice was fed to a separate coding agent at each fork, its success rate on 41 held-out SWE-bench Pro tasks jumped from 14.6 percent to 33.7 percent. Perfect advice would have reached 39 percent, so the student captured most of the possible gain.
Our takeaway: finishing a task and making smart choices along the way are two different skills, and today’s agents are much better at the first one.
What This Means for You
If you hand long jobs to tools like Claude AI models or coding agents, this research is a useful reality check. Passing tests at the end does not prove the agent took a sensible route. The same lesson showed up recently when agents chasing refunds made poor calls on the way to a goal.
- Check in at key turning points instead of only reviewing the final output.
- Ask the agent to explain why it picked one approach over another before it commits.
- For code, review structure and maintainability, not just whether tests pass. Our guide on explaining a codebase can help you dig in.
- Do not assume more reasoning time fixes weak planning, since this study found it barely helped.
You can follow more research like this in our daily AI news section and our coverage of AI agent tools. The benchmark and its code are publicly available for developers who want to test their own setups.
Sources
This article is based on the research paper on arXiv and a detailed report from The Decoder.
Conclusion
Taste-Bench puts a number on something many agent users already sense: AI agents are better at finishing work than at choosing wisely along the way. The best models picked the right path only about six times out of ten, and they fell apart when the evidence came late. Extra thinking time did not close the gap, but training on hindsight did help a lot. For now, the smartest move is to keep a human eye on the big decisions, even when the final result looks fine.
