Introduction to AI in Cybersecurity
To better understand the current state of artificial intelligence (AI) in cybersecurity, dozens of security practitioners, researchers, vendors, analysts, and AI experts were consulted. The result is a comprehensive snapshot of how AI is being used across the security landscape today.
Key Topic Areas
The report examines the role of AI through multiple lenses: whether it can be trusted, how organizations are using it, how it can be misused by legitimate insiders, how it is being exploited by cyber adversaries, and where the technology is likely headed next. The five key topic areas are: Generative AI (gen-AI), Agentic AI, Shadow AI, Machine learning (ML), and Artificial general intelligence (AGI).
Generative AI
Generative AI (gen-AI) is the bedrock of contemporary AI, and it generates new content from an AI model. Chatbots are the users’ interface to the LLM, enabling questions to be applied and responses received in natural language.
“Gen-AI trains on massive data sets, learns statistical and relationship patterns, and then uses those patterns to synthesize original output from a prompt,” explains Ahmad Shadid, co-founder and CEO at ORGN.com. This is important, as it does not create factually correct answers to prompts, but rather predicts probable answers based on the relationship patterns it has learned.
Trust in gen-AI
The biggest question in the use of AI is whether you can trust an output that is based on probability rather than grounded in known truth. The answer is not a simple yes or no, but rather 56 shades of ‘No’. “It can be considered both trustworthy and not trustworthy, depending on the intent, the models used and the overall data flow involved,” comments Melissa Ruzzi, senior director of AI at AppOmni.
Gen-AI is not inherently trustworthy, says Yichuan Zhang, CEO and co-founder of Boltzbit. “It is prone to hallucinations (confident but false statements) and data leakage (reproducing the training content or the context content exactly).”
Gen-AI Use
Gen-AI use offers benefit in three areas: SOC productivity, secure coding, and vibe coding. “Many enterprises use these models to generate documents, write articles, generate software, or replicate the messages a human would send when orchestrating a larger workflow,” says Sant-Miller.
Gen-AI Misuse
The misuse of gen-AI within enterprises is usually unintentional, and it emanates from a failure of governance around the technology. Ungoverned use of gen-AI is always a misuse of AI. Individuals begin to rely on AI to provide quick (but not necessarily accurate) answers to questions or problems.
Gen-AI Abuse
Gen-AI abuse refers to the use of AI by bad actors. The primary reason is the power and complexity of AI. When an enterprise develops an internal AI application, it must be certain to get it right or face a possible self-inflicted catastrophe. This takes time, and criminals don’t have this concern.
Zhang highlights three primary examples of gen-AI abuse: hyper-realistic phishing, polymorphic malware, and vibe coded phishing websites and/or aggressive attacking software. Gino Sciretta, CEO at BranditScan, warns, “Generating a convincing fake identity now takes seconds. Detecting one reliably still requires specialized tools and trained analysts.”
Conclusion
Artificial intelligence is a powerful tool that can be used for both good and bad. Its use in cybersecurity is a growing concern, and its misuse by insiders and exploitation by cyber adversaries is a threat that must be taken seriously. Governance is the key to preventing the misuse of gen-AI, and trustworthiness is a complicated question that depends on the intent, models used, and overall data flow involved.
Source: SecurityWeek