In a recent talk, I explored how Generative AI can aid us when working with IT security, going into the use-cases, some of the benefits and also something that you need to be cautious about . Here, I’ll summarize the key points and insights from the presentation.
Generative AI originates from large pre-trained models designed to generate new content based on patterns in their datasets and user prompts. When ChatGPT was launched in November 2022, it was primarily focused on text generation. Fast forward nearly two years, and Generative AI now extends to multiple media types, including music, images, videos, and even application code, thanks to services like GitHub Copilot.
The Capabilities of Generative AI
It’s important to note that Generative AI models don’t rely on facts. Instead, they predict the most likely next word or letter in a given context. For example, typing “Why is the sky…” would probably result in “blue,” or “Why did the chicken cross…” would likely be followed by “the road.” These models behave based on their training datasets and their specific use case, whether for chat interactions or automation tasks.
The Evolution of Generative AI Models
The ecosystem of Generative AI models has expanded dramatically. Some are cloud-based, while others can run locally. Modern models can handle various media types, such as images and audio, using the same neural network. The latest version of GPT, for instance, can process images, audio, and text simultaneously.
Initially, models like ChatGPT could handle up to 2,500 words. Today, advanced models like Google Gemini can manage up to 1.5 million words in a single context, demonstrating exponential growth in their information processing capabilities.
Integrating Functions and APIs
Generative AI now includes functionalities known as “functions,” which allow models to generate specific outputs that can trigger scripts or API calls. For example, using the Google Search API requires valid JSON code, which these functions can generate.
Several open-source frameworks build on function calling, such as Hacker GPT, which includes predefined integrations for tasks like port scanning or domain scanning. Major providers like Microsoft Copilot for Security, Palo Alto Copilot, and Google AI Workspace use these functionalities to enable API calls across various data sources and trigger actions within their ecosystems.
Cloud Providers and security services.
Many of the security service listed above use fine-tuned language models, enhancing their efficiency for specific security contexts and provide better results in regards to creating search queries within their own systems. Such as Microsoft Copilot for Security, uses a fine-tuned model which is much better equipped at understanding and generating Kusto queries and understanding security context.
So what about the use-cases?
Security Copilots
Generative AI can be leveraged in IT security through pre-created services like Copilot for Security from Microsoft and Palo Alto. These tools help security analysts find necessary information using natural language queries. This approach eliminates the need to know specific queries, information, and tools, simplifying the process of understanding incidents. However if you know what you are looking for and know your way around the query engine, well then you will not save any time doing this. In fact it will also be a lot cheaper to do it manually.
Example here with Crowdstrike and Charlotte AI
The idea behind this is that you can use native language to interpret the data and context and use functions/integrations to lookup data or enrich the data and use a language model to help you understand the context.
Another use-case is of course using the LLM to enrich the context before an analyst has had time to look at it, you might save some time but you better be certain that the context is correct. Since hallucination is still a big issue, even if there are mechanisms and improvements constantly happening within the ecosystem of GenAI, it is a complex issue and will take some time to solve it.
Code Analysis
Generative AI is also valuable for code analysis. Instead of setting up a sandbox virtual machine to analyze obscure PowerShell or batch scripts, security professionals can use Generative AI to understand the code’s functionality quickly. That is of course dependent on that the code or script is in a language that the LLM understand or is trained on. There are also other LLMs that are trained on ONLY application code such as StarCoder that can be a better model to this type of analysis. It is also easy to create custom applications with predefined system prompts and instructions on how it should analyze and reply to the user. This is an example using GPT-4o with Langchain and Streamlit. Which is essentially just a code analysis tool.
Information Collection
When a new vulnerability becomes public, developers and security teams often scramble to gather relevant information to assess the risk and determine the appropriate mitigation strategies. While there is usually a wealth of data provided by vendors, security bulletins, and threat intelligence platforms, these sources might not always provide enough context or details specific to your environment. This is where generative AI can significantly streamline the process by acting as a virtual assistant to support independent searches and deliver actionable insights.
Application development
For application developers, there are many exciting features coming with GitHub Copilot that aim to enhance security. In particular, these features focus on automating code reviews and security scans, making it easier for developers to identify vulnerabilities and secure their applications.
One of the most notable features (currently in preview) is automated pull requests. This feature leverages AI to scan the codebase and generate pull requests that enhance the security of the code. These AI-generated pull requests can identify common security issues, such as outdated dependencies with known vulnerabilities, or propose optimizations in the code to avoid common pitfalls like buffer overflows, SQL injection, or other common vulnerabilities. By automating this process, developers can integrate security improvements without the need for manual audits, reducing the chance of human error while maintaining an efficient development cycle.
Another powerful feature is the use of Large Language Models (LLMs), like GitHub Copilot, to scan the source code and search for security issues that traditional tools might miss. For example, conventional regex-based scanning tools are effective at identifying secrets such as hardcoded API keys, passwords, or other sensitive information by matching specific patterns. However, these tools can sometimes miss secrets that are not easily identifiable by regular expressions. With LLMs, GitHub Copilot can interpret code more intelligently, understanding the context in which variables are used, spotting patterns of misuse, and detecting embedded secrets in complex formats or unconventional locations. This greatly enhances the scope of code scanning, providing developers with more comprehensive security feedback.
While generative AI can introduce many new capabilities that enhance both security analysts’ and developers’ ability to defend against threats, it also, unfortunately, provides powerful tools for malicious actors. Just as AI assists defenders in securing their systems, it can also be exploited by hackers to create more sophisticated attacks. The dual-use nature of AI presents a new set of challenges for cybersecurity, as it can fuel various malicious activities with unprecedented speed and precision.
One of the most concerning aspects is the ability of generative AI to help create malicious code more efficiently. Hackers, even those with minimal coding skills, can leverage AI to generate malware, ransomware, or exploits that target known vulnerabilities in software systems. With AI’s ability to rapidly scan large codebases and find weak points, malicious actors can automate the process of vulnerability discovery, enabling them to create exploits faster than ever before. AI can also help attackers obfuscate their code, making it harder for traditional detection tools to identify malicious software or reverse-engineer it for analysis.
Another area where AI can assist attackers is in creating more convincing phishing attacks. Traditionally, phishing emails are often poorly written or contain telltale signs of fraud that alert users to the potential danger. However, with the help of AI, attackers can now generate highly personalized and grammatically flawless emails that mimic the writing style of trusted contacts or organizations. These AI-generated phishing emails can be tailored to specific individuals or groups by analyzing publicly available information on social media, websites, and other digital footprints. This makes the attacks more believable and increases the likelihood of success.
In addition to phishing, AI-powered social engineering can take more advanced forms. Hackers can use AI to analyze voice or video data to create realistic impersonations of trusted figures, such as company executives or family members, in what are called vishing (voice phishing) or deepfake attacks. Deepfakes, which leverage AI to generate highly realistic but entirely fabricated audio or video content, are becoming a growing concern. In a deepfake attack, an AI can generate a fake video of a CEO instructing an employee to transfer funds to a fraudulent account, or impersonate a trusted figure in a video call to gain access to sensitive information.
These AI-driven deepfakes are particularly dangerous in corporate environments where video conferencing is common, and employees might not think twice about following directives from someone who appears to be their superior. As AI technology advances, these deepfakes are becoming more difficult to detect, increasing the risk of such attacks being successful.
But, fortunately, most AI companies which are creating LLMs or GenAI capabilities are adding a lot of security filters or mechanisms in their models to prevent it to be used for malicious purposes (but not everyone)…
We are still in the early phases of GenAI so it will be interesting to see how it evolves the next years.