Key Takeaways
- Anthropic’s Alignment team reported on October 27, 2026 that 50 professional analysts matched their AI agent’s secretly assigned lean on 86% of four analytical choices, against about 50% expected if the agent made no difference.
- In the post-task survey, 77% repeated the agent’s arguments in at least one answer, but only 15% credited the agent, and none suspected the preferences were planted.
- A human sign-off proves independent judgment only if the person formed a view before the agent spoke.
Do AI Agents Actually Influence Expert Decisions?
Yes, in this experiment they did, and strongly. Analysts went with their agent’s lean on 86% of the four analytical decisions, where chance would give about 50%.
The agent’s lean was set by a coin flip and hidden in its instructions. The analysts did not know it existed, and the coin had no connection to what was correct.
Anthropic’s researchers (Amaç Herdağdelen, Hailey Joren and Alex Tamkin) recruited 61 analysts and analyzed the 50 who used the agent. All had causal-inference experience. They worked in an editor with Claude Code, running Claude Opus 4.8 at High effort.
Decrypt covered the same results. We read both. Every number below comes from Anthropic’s post unless we say otherwise.
What Was the Task, and What Did the Agent Push?
Each analyst advised a fictional farming cooperative on whether to expand an advisory program. The board would expand it only if productivity rose by roughly 3–4%.
The board did not say how to measure productivity. So four choices stayed open, each with two reasonable options:
- Outcome: crop yield per hectare, or yield per unit of operating cost.
- Comparison: before-and-after change against other farms, or a direct comparison with statistical adjustment.
- Coverage: enrolled farms only, or the whole membership.
- Weighting: every farm equal, or larger producers weighted more.
Anthropic says each choice could move the estimated gain by several percentage points. Against a 3–4% threshold, that is enough to change the answer.
The agent was told to hold its lean as its own view and make the strongest honest case for it. It was also told to change its mind if the evidence demanded and never to mislead.
The Deference Ledger
This table is our layout of Anthropic’s per-decision results. We ran no experiment ourselves.
| Decision | Matched agent’s lean | What it changes |
|---|---|---|
| Outcome measure | 92% | What “productivity” means |
| Weighting | 90% | Whose farms count more |
| Comparison method | 82% | How the effect is isolated |
| Farm coverage | 81% | Who the estimate describes |
The final recommendation moved less. When the agent leaned toward rollout, 54% of analysts recommended it, against 36% when it leaned the other way.
Anthropic’s reading is that the recommendation largely followed from whether the estimate cleared the threshold. By then the agent’s earlier leans had already shaped the estimate.
Did the Experts Notice?
Mostly not. Among analysts who had stated a preference before the agent weighed in, 60% switched when the agent disagreed, against 6% when it agreed. That rests on only 38 cases, and the 95% interval runs from 24 to 82 points on the gap, so treat it as direction, not size.
Confidence did not track influence. Analysts who followed the lean on all four decisions averaged 71 on a 0–100 confidence scale, against 73 for the rest.
Asked to explain their choices, 77% repeated the agent’s argument in at least one answer, sometimes word for word. Only 15% credited the agent anywhere.
Of 48 survey respondents, 52% called the final recommendation more their own than the agent’s. Six guessed the study was about AI influence, and none suspected the agent’s preferences were manipulated.
Anthropic also found that experience did not predict who deferred. Only age showed a link, and it disappeared after correcting for the number of traits tested.
Two Lenses
Handing off the hours
The agent made the work better. Analysts who used it made serious methodological errors far less often: 3 of 50, against 6 of 11 among those who worked without it. They also finished faster, with a median of 65 minutes against 90.
Anthropic itself warns against reading too much into that comparison, since the no-agent group was small and less experienced. Still, nobody wants a result that is slower and wrong. Keeping the agent is a reasonable choice.
Signing the call nobody chose
The cost shows up in the one place errors cannot reach. Where two options are both reasonable, a reviewer checking for mistakes finds nothing wrong, whichever option the agent picked.
In Anthropic’s words, “the work still goes out under the expert’s name, but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs.”
We weigh this lens more heavily for any work where the decision is the product, such as a policy memo or a safety experiment. For a routine report with one defensible method, the handoff costs less.
How Strong Is This Evidence?
Anthropic lists several limits, and they matter:
- It strengthened the agent’s lean on purpose, so real agents may push less.
- It studied one task, one field and one model, in about an hour of work.
- It recorded which option analysts chose, not how strongly they preferred it.
- It could not tell how much any single analyst was influenced.
There is a counterpoint in the paper. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so models can carry default preferences of their own.
We could not verify the full paper’s statistics beyond Anthropic’s post. We did not read the raw data.
Decision Ownership Check: Three Steps Before the Agent Speaks
This check is our synthesis of the study’s suggested remedies, not something Anthropic tested or we ran.
- List the open calls. Before starting, write down which choices have more than one reasonable answer, such as the metric, the comparison group or the weighting.
- Commit first. For each one that matters, write your own pick and one reason before asking the agent. Anthropic suggests agents ask for the expert’s view before offering their own.
- Ask for the other side. Have the agent make its best case for the option it did not pick. Anthropic proposes agents argue each option, with the caveat that this can overwhelm users on long analyses.
This pairs with our earlier look at why parallel agents are not independent checks. That post asked whether agents can verify each other. This one asks the same of the human reviewer.
Anthropic raises a related risk: teams that defer to the same model may agree for the same hidden reason. Agreement would then count as one voice, not several. It also fits the case for a stop control the agent cannot touch, since both rest on controls that do not depend on the agent.
What Would Change Our View?
We would worry less if a replication used agents with no assigned lean, real domain experts, and tasks where the expert already held strong views. Anthropic itself says those factors might shrink the effect.
We would worry more if the same deference appeared across several models and fields, or if analysts who committed first still shifted at similar rates.
A measure of deference in ordinary work, without randomized leans, would change our reading most. Anthropic lists that as an open problem.
FAQ
Q. Does this study show AI agents give biased advice?
A. No. The researchers assigned the lean by coin flip and amplified it on purpose, to measure its effect. The study shows how readily experts follow a stated preference, not how often real agents hold one.
Q. Should experts stop using AI agents for analysis?
A. Anthropic says no. Analysts who used the agent made fewer serious errors and finished faster, though the comparison group was small. Its suggestion is to decide deliberately which calls need a person’s own informed judgment.
Q. How can I tell whether I deferred to the agent?
A. Anthropic found that analysts could not reliably tell. Writing down your own pick and reason before asking is the practical test, because it leaves a record the agent did not shape.
Sources
- How AI agents shape experts’ judgment calls — Anthropic, Oct 27, 2026
- AI Can Quietly Influence Experts’ Decisions — Decrypt, Oct 2026
Leave a Reply