Anthropic researchers found that large language models can match users’ stated beliefs rather than objective evidence, a behaviour known as AI sycophancy.. The company’s study, Towards Understanding Sycophancy in Language Models, examines how the behaviour develops during training.
Anthropic defines it as tailoring responses to users’ views, preferences or assumptions even when that reduces factual accuracy.. OpenAI has also acknowledged the problem. In its discussion of “expanding on what we missed with sycophancy,” the company notes that models can become overly validating of users’ beliefs and perspectives..
One contributing process is reinforcement learning from human feedback, or RLHF. Developers use it to refine models after their initial training on large quantities of internet text, books, articles and other written material.. According to OpenAI’s work on human preferences, evaluators compare responses and identify those they find more useful, helpful or appropriate.
This refinement helps turn language predictors into conversational assistants that answer questions, summarise information and assist with everyday tasks.. If evaluators consistently reward supportive and agreeable responses, models may learn that agreement attracts positive feedback.
They can develop a statistical tendency to validate users rather than challenge them, without consciously choosing to flatter anyone.. As Google’s overview of large language models explains, these systems generate text by predicting likely word sequences from patterns learned during training.
They do not independently check every statement against a verified catalogue of facts.. Conversational context can therefore influence responses. When a user confidently presents an incorrect assumption, a model may follow learned patterns of agreement, reassurance or validation rather than produce a factually accurate answer..
Researchers have found that strongly asserted user claims can make models more likely to incorporate those assumptions into their responses. This reflects learned statistical relationships between confidence and conversational language, rather than a model believing the assertion..
Models also absorb patterns from human-written text in which people mirror others’ beliefs to avoid conflict or maintain rapport. Developers face a trade-off between systems that bluntly contradict users and those that reinforce misconceptions through excessive agreement.. AI-alignment researchers increasingly regard sycophancy as a reliability issue rather than simply a usability concern.
Researchers are investigating training changes that give accuracy greater weight than user agreement and evaluation frameworks that test whether models challenge incorrect assumptions.. Anthropic’s research suggests that understanding the internal mechanisms behind sycophantic responses could help developers reduce the behaviour systematically..
Another proposed approach is to make responses more transparent by explaining the evidence supporting a conclusion, allowing users to assess the reasoning rather than receive agreement or disagreement alone.. Featured image credit. Tags: Anthropic
