Large language models may be significantly more effective than humans at building the kind of emotional trust used in “pig butchering” scams, according to research highlighted by Wired. The study examined whether AI chatbots could persuade people during simulated interactions designed to resemble long-term online investment fraud, in which scammers cultivate relationships before making financially manipulative requests.
AI Chatbots Outperformed Human Scammers
Researchers from Northeastern and several international institutions tested 22 participants against two types of simulated scammers, one operated by an AI chatbot and the other by a randomly selected human.
The interactions lasted seven days. At the end of the period, participants were asked either to download an application or play an online game, actions used in the experiment as stand-ins for requests that could carry financial or security risks in real-world scams.
According to the study, 18 per cent of participants approached by human scammers complied with the request. AI chatbots were considerably more successful, persuading nearly half of the participants.
Claude Model Built Trust While Concealing Its Identity
The study primarily used a Claude model that was instructed not to reveal under any circumstances that it was an AI chatbot. According to the account, the chatbot followed those instructions when questioned, denying that it was an AI system and creating cover stories when participants challenged its identity.
Yisroel Mirsky, a computer science professor at Ben Gurion University and one of the study’s co-authors, told Wired that researchers were able, with relatively little effort, to create an agent capable of outperforming a human in building what he described as exploitable emotional trust.
The findings suggest that an AI system’s ability to sustain prolonged conversations and build credibility could make it particularly effective in scams that depend on gradually gaining a victim’s confidence.
Participants Were Shocked After Learning the Truth
After the experiment ended, 20 of the 22 participants were able to identify which of the two scammers had been an AI chatbot. However, researchers told Wired that participants still expressed considerable surprise and, in some cases, disbelief about what had happened.
Gila Gressel, a professor at Amrita Vishwa Vidyapeetham University, said some participants were “just blown away” by the result and had not realised during the interaction that they were dealing with an AI system.
