Cybercriminals are using artificial intelligence to create fake obscene photographs and videos of individuals and threatening to circulate them on social media unless money is paid. Cybersecurity experts have warned that such manipulated content can appear genuine, making it a growing concern for online users.
A recent case in Mangaluru, Karnataka, involved a 19-year-old man whose photograph was allegedly manipulated using AI to create an obscene image. Fraudsters reportedly threatened to circulate the image on social media and cheated him of ₹5.24 lakh.
The incident has highlighted the risks of AI-generated content being used for blackmail, harassment and financial fraud.
What Are Morphed Photos and Videos?
Morphed photographs and videos are digitally manipulated images or recordings that falsely portray a person in situations that never occurred.
Such content may involve altering someone’s face, changing their appearance, adding objectionable material or creating entirely fabricated scenes.
Advances in artificial intelligence have made these manipulations more convincing. A photograph available on social media can potentially be used to produce fake images or videos that appear authentic.
Cybercriminals exploit this technology to threaten victims with public embarrassment, damage to their reputation or financial consequences.
In many cases, the victim may never have created or shared the content being used against them.
How Are Criminals Creating Fake Obscene Content?
Cybercriminals use several digital techniques to manipulate photographs and videos.
Traditional photo-editing software allows changes to a person’s face, body, background or other visual elements. Face-swapping technology can place someone’s face onto another person’s photograph or video.
Deepfake technology goes further by using artificial intelligence to manipulate facial expressions, lip movements and voices.
AI image and video generation tools can also create entirely new content using an existing photograph or a text description.
Similarly, video manipulation techniques can alter original recordings to make it appear that a person participated in an activity that never occurred.
These methods can be used to create misleading content without the knowledge or consent of the person depicted.
FCRF Launches CP-FRM to Build India’s Next Generation of Fraud Risk Professionals
How Are Victims Being Blackmailed?
Fraudsters may obtain photographs from publicly accessible social media profiles or other online platforms before manipulating them into obscene content.
They then contact the person depicted and threaten to circulate the material among friends, relatives or social media users.
The threat of embarrassment or reputational damage is used to pressure victims into making payments.
The Mangaluru case illustrates the financial consequences of such threats. A 19-year-old man was allegedly cheated of ₹5.24 lakh after scammers created an obscene image using his photograph and threatened to publish it.
The risks are not limited to financial fraud. Morphed photographs and videos can also be used for online harassment, impersonation and attempts to damage someone’s reputation.
Cybersecurity experts have warned that the apparent authenticity of AI-generated material can make these threats particularly distressing.
What Should Victims Do When Threatened?
People receiving threats involving morphed photographs or videos should preserve the available evidence and report the incident.
Screenshots of the manipulated content, threatening messages and conversations can help establish how the harassment occurred.
Victims should also retain the accused person’s mobile number, social media username, profile link and any other available contact details.
If the content has already been published, the link to the post or profile should be preserved.
Where money has been transferred, victims should retain transaction details, including UPI or bank payment records.
The offending post, account or profile should be reported directly to the relevant social media platform.
A complaint can also be filed through the National Cyber Crime Reporting Portal. Victims may additionally approach the nearest cyber police station.
Experts advise against forwarding the manipulated content or continuing arguments with the person making the threats.
Keeping complaint acknowledgements and reporting reference numbers can also help victims track subsequent action.
How Can Fake Photos and Videos Be Removed?
Victims whose manipulated intimate photographs or videos have been circulated online can seek removal through the relevant social media platforms.
The first step is to preserve screenshots and links before reporting the objectionable material.
The complaint should identify the content, the account responsible for posting it and the nature of the violation.
Adults aged 18 and above can also seek assistance through StopNCII, a free online tool designed to help prevent the circulation of non-consensual intimate images.
The system creates a digital fingerprint, known as a hash, of the relevant photograph or video.
Participating platforms can use these fingerprints to identify matching content and take action against its circulation.
The process is intended to help prevent the repeated sharing of intimate material without consent.
However, removing one post does not necessarily prevent an offender from creating another account and uploading the material again.
If the content reappears, victims should report each new post or account separately and preserve the associated screenshots, links and messages.
They should also update their cybercrime complaint with the new information.
Where StopNCII is being used, the digital fingerprint of additional content can be submitted when necessary.
What Legal Action Can Be Taken?
Creating and circulating obscene morphed photographs or videos can attract criminal action under the Information Technology Act and other applicable laws.
Section 67 of the IT Act provides punishment for publishing or transmitting obscene material in electronic form.
For a first conviction, the provision allows imprisonment of up to three years and a fine of up to ₹5 lakh.
Section 67A deals with electronic material containing sexually explicit acts or conduct. A first conviction can attract imprisonment of up to five years and a fine of up to ₹10 lakh.
The penalties may increase for subsequent convictions.
The legal provisions applicable to a particular case depend on the nature of the content, the actions of the accused and the circumstances of the offence.
The use of manipulated images to threaten someone or demand money may also require investigation into the accompanying harassment or financial fraud.
How Can Social Media Users Reduce the Risk?
Cybersecurity experts have identified several online practices that can increase the possibility of photographs and videos being misused.
Keeping social media profiles publicly accessible allows strangers to view and potentially obtain personal photographs.
Accepting friend requests from unknown individuals, sharing large numbers of personal images and uploading photographs to unfamiliar websites can also increase exposure.
Users are advised to avoid suspicious AI photo-editing applications and exercise caution before granting apps access to their phone galleries.
Private photographs and videos should not be shared through insecure platforms or with unfamiliar individuals.
Strong passwords, two-factor authentication and updated mobile applications can provide additional protection against unauthorised access.
Users should regularly review their social media privacy settings and limit unnecessary permissions granted to applications.
They should also avoid downloading unknown links, files or applications and remain cautious when communicating with strangers online.
These precautions can reduce opportunities for misuse, although they cannot eliminate the risk of publicly available photographs being manipulated.
AI-generated obscene content is creating opportunities for criminals to turn ordinary photographs into tools of blackmail. Victims should preserve evidence, report threats promptly and seek removal of manipulated material through social media platforms and available complaint mechanisms. The Mangaluru case demonstrates that even a fabricated image can lead to substantial financial losses when criminals exploit fear and embarrassment.
Follow for daily updates on cybercrime, corporate fraud, DFIR, hacking, investigations, and digital forensics
