In the rapidly evolving world of cybersecurity, experts are sounding the alarm over the next frontier of cyber threats—AI-powered hacking. With the rise of generative AI and large language models (LLMs), the barrier to entry for launching cyberattacks has dropped dramatically. Now, even individuals with little to no coding expertise—dubbed “vibe hackers”—can command AI tools to generate malicious code, often by simply describing what they want to achieve.
“Vibe hacking” represents a new trend where users can prompt AI to solve complex cybersecurity problems—or create them. This shift, mirrors earlier evolutions in the hacking ecosystem, where once-manual attacks became increasingly automated and scalable.
A key case study in this AI revolution is XBOW, a purpose-built system that sits atop several leaderboards on HackerOne, a bug bounty platform. Built for white-hat penetration testers, XBOW reportedly autonomously discovers and exploits vulnerabilities in 75% of benchmarked web systems. It is, in essence, a prototype of the future: an AI that can learn, adapt, and execute high-level attacks with minimal human input.
WormGPT to FraudGPT: The Rise and Fall (and Rise) of Blackhat LLMs
AI models capable of producing malicious code have already appeared in the wild. In 2023, a controversial tool named WormGPT surfaced on Discord servers, Telegram groups, and dark web forums. Designed as a blackhat LLM, WormGPT allowed users to generate phishing scripts, malware payloads, and more. Once exposed, it vanished—but was quickly replaced by clones like FraudGPT.
However, most of these were likely just jailbroken versions of ChatGPT or Claude, with slight modifications. OpenAI, Anthropic, and other AI developers have repeatedly emphasized their efforts to implement guardrails, but these have proven vulnerable to so-called jailbreak prompts—tricks to bypass content restrictions.
Some companies, like Anthropic, now offer bug bounties to users who discover new jailbreak exploits, underlining just how seriously the AI industry views these threats. Meanwhile, mainstream LLMs like ChatGPT, Gemini, and Claude continue to be tested and, at times, exploited for generating harmful scripts.
Security researchers at Trend Micro have demonstrated how AI can be manipulated into generating malware by posing as a security researcher competing in a capture-the-flag exercise. This subtle manipulation is often enough to trick the AI into producing sophisticated scripts.
Algoritha: The Most Trusted Name in BFSI Investigations and DFIR Services
The Real Threat: Scaling Skilled Hackers, Not Replacing Them
While the narrative of AI enabling anyone to become a hacker is compelling, cybersecurity professionals argue that the greatest danger lies not in the amateurs, but in the skilled professionals using AI to scale their operations.
These professionals can use LLMs to automate reconnaissance, craft custom malware, and even develop polymorphic code that rewrites itself as it learns in real-time. Smith warns of a future where a single hacker could unleash 20 simultaneous zero-day exploits across global systems, overwhelming cybersecurity teams and tools alike.
Moussouris adds that while AI-generated tools won’t replace elite hackers, they will accelerate the pace of cybercrime, making attacks harder to detect, triage, and respond to.
Security analysts note that while “script kiddies” using AI will pose some risks, the more serious threats will come from nation-state actors, APTs (advanced persistent threats), and organized cybercriminal groups integrating AI into their attack arsenal.