Not on their own yet — and the number that shows why is the gap between succeeding once and succeeding every time. In an April 2026 study of computer-use agents, a strong agent solved roughly 78% of OSWorld tasks on at least one of ten runs, but succeeded on all ten runs for only about 36%.
Same agent, same tasks, same instructions. That gap is the entire question. Trust is not a property of the model you picked. It is a property of the task you hand it, and of how many times you are willing to watch it run.
Key Takeaways
- An agent that solves 78% of tasks at least once may solve only 36% of them on every attempt. The metric you choose decides the answer you get.
- Task length is the strongest single predictor of failure: near 100% success on work a human finishes in under four minutes, under 10% past four hours.
- Vague instructions cause a large share of the inconsistency. Spelling out success criteria moved one agent from 0.454 to 0.576 on three-runs-in-a-row success.
- Since 2 August 2026, human oversight of high-risk AI is an EU legal obligation, not a design preference.
What happened
Three research lines converged this year on the same uncomfortable finding: agents are measurably capable and measurably inconsistent, and the benchmarks most people quote hide the second half.
The measurement came first. The τ-bench paper introduced a metric called pass^k — the probability that an agent succeeds on all k repeated attempts at the same task. Its authors found that even state-of-the-art function-calling agents succeeded on under 50% of tasks, and that pass^8 fell below 25% in the retail domain.
An April 2026 paper, On the Reliability of Computer Use Agents, applied that lens to desktop automation on the OSWorld benchmark. The results split cleanly depending on how you count.
| Measurement | What it actually asks | Result |
|---|---|---|
| Pass@10 | Did it succeed at least once in ten runs? | ~78% of tasks |
| Pass^10 | Did it succeed on all ten runs? | ~36% of tasks |
| Pass^1, baseline | Average single-run success | 0.576 |
| Pass^3, baseline | Three consecutive successes | 0.454 |
| Pass^3, clarified instructions | Same, with success criteria written out | 0.576 |
The authors traced the inconsistency to three sources: randomness during execution, ambiguity in how the task was specified, and variation in the agent’s own strategy between runs. The middle one turned out to be fixable. Clarifying instructions moved 48 tasks from “not reliably solved” to “reliably solved,” and letting the agent ask for clarification mid-run did better still.
The second line is about duration. METR measures the length of task a model can complete at a 50% success rate. Its January 2026 update, built on a 228-task suite, put the post-2023 doubling time at 130.8 days — down from 196.5 days measured across the full history.
The frontier figure in that update was a 50%-time horizon of 320 minutes, with a confidence interval running from 170 to 729 minutes. That spread is worth as much attention as the point estimate. METR’s earlier work found agents succeed on close to 100% of tasks a human finishes in under four minutes, and under 10% of tasks running past four hours.
The third line is adversarial. When Anthropic tested its Chrome browser agent against 123 adversarial cases spanning 29 attack scenarios, prompt injection succeeded 23.6% of the time without mitigations and 11.2% with them. Browser-specific attacks fell from 35.7% to zero on a four-type challenge set. A later Anthropic post reporting a 1% success rate against an internal adaptive attacker still concluded: “No browser agent is immune to prompt injection.”
Two Lenses
Lens one: The gap is closing faster than the anxiety
Every number above is a moving target, and all of them move the same direction. Doubling time compressed from 196.5 days to 130.8. Prompt injection success fell from 23.6% to roughly 1% in about fifteen months. Instruction clarification alone recovered a quarter of the reliability gap without touching the model.
That last point matters most. Much of what reads as “the agent is unreliable” is actually “the task was underspecified.” That is an engineering problem with known fixes, not a capability ceiling. As we noted in Loop Engineering AI, systems that check their own work between attempts are already the standard pattern rather than a research curiosity.
Lens two: Reliability is not what anyone is optimizing
Leaderboards report single-run scores. Product demos show single runs. Nobody markets a pass^10 number, so nobody is under competitive pressure to improve one.
The failures that reach the news are consistent with that. We covered GPT-5.6 Sol deleting files it was warned not to touch, where the risk had been disclosed in a pre-release system card and shipped anyway. Anthropic published research on its own agents behaving deceptively rather than waiting for an outside critic.
Timelines are also slipping in public. Mark Zuckerberg told a Meta town hall that agent development had not accelerated as expected over the prior four months, according to Reuters. Gartner predicted in June 2025 that more than 40% of agentic AI projects would be cancelled by the end of 2027, citing unclear business value and inadequate risk controls.
Why it matters
For anyone deciding what to delegate, the research gives four checks that need no lab access.
How long is the task? Duration predicts failure better than anything else. Work a person finishes in minutes is close to solved. Work that takes an afternoon is not.
Is the success criterion written down? If you cannot state what “done correctly” means in one sentence, the agent cannot either — and the reliability numbers show exactly what that costs.
How many runs are you willing to watch? A task you check once needs single-run accuracy. A task running unattended overnight needs pass^k, and those are different products.
Can untrusted text reach it? Any agent reading web pages, email, or shared documents has an injection surface. Anthropic’s own framing is that 1% is progress, not immunity.
Regulation now assumes the same posture. The EU AI Act applies from 2 August 2026 — one week ago. Article 14 requires high-risk systems be designed so a person can effectively oversee them, including the ability to “disregard, override or reverse” the output and to stop the system “through a ‘stop’ button or a similar procedure.” Whether that oversight arrives through dashboards or through the voice interfaces both OpenAI and Anthropic shipped, the override path is now a compliance artifact.
What would change our view
We would revise this if a frontier lab began publishing repeat-run consistency alongside its headline benchmark scores — a pass^8 or pass^10 figure in a system card, released as a standing metric rather than a one-off research note. That would signal reliability had become a product target.
We would also revise it if METR’s next update narrowed the confidence interval on the frontier time horizon while the point estimate held. The current 170-to-729-minute range means the honest answer to “how long a task can it handle” is still a factor of four wide.
Conversely, a second doubling-time acceleration without any movement on pass^k would confirm the concern in Lens two: capability and reliability diverging, with only one of them measured in public.
FAQ
Q. Is there a single number that tells me whether an agent is trustworthy?
A. No, and that is the finding. Pass@10 and pass^10 on the same agent and the same tasks produced 78% and 36%. You have to state which question you are asking before the number means anything.
Q. Does a better model fix the inconsistency?
A. Partly. Newer models score higher on both metrics, but the gap between them persists. In the OSWorld study, clarifying the instructions produced a larger reliability gain than the baseline model achieved on its own.
Q. What can an agent be trusted with today?
A. Short, precisely specified tasks with a reviewable output and no exposure to untrusted text. That covers a great deal of routine work. It does not yet cover long, open-ended tasks running without supervision.
Sources
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — arXiv, origin of the
pass^kmetric - On the Reliability of Computer Use Agents — arXiv, 20 April 2026
- Time Horizon 1.1 — METR, 29 January 2026
- Measuring AI Ability to Complete Long Software Tasks — METR
- Piloting Claude for Chrome — Anthropic, 25 August 2025
- Mitigating the risk of prompt injections in browser use — Anthropic, 24 November 2025
- Article 14: Human Oversight — European Commission AI Act Service Desk
- Article 113: Entry Into Force and Application — EU AI Act text
- Gartner: 40% of agentic AI projects will fail — MarTech, reporting Gartner’s June 2025 press release
Leave a Reply