AI security concerns for CISOs
Already nearly 3 out of 4 companies have adopted Artificial Intelligence (AI) globally in at least some area of their business, with nearly 2 out of 3 having hopped on the Generative AI (GenAI) bandwagon.
Adoption is happening so quickly, and the tools are becoming so accessible, scalable, and commonplace that it’s astounding to think that ChatGPT is barely just two years old. At this rate, GenAI seems to have skated right over the trough of disillusionment and reached peak productivity.
But in the rush to adopt AI and Large Language Models (LLM) across various business functions, security analysis shows that many organizations have gotten ahead of themselves. While many are aware of the potential threats and vulnerabilities inherent in GenAI, they’re unaware of or unsure how to address them.
It’s become a major blind spot for CISOs who are nonetheless on the hook for protecting their organizations.
The problem arises from two angles:
- Misguided assumptions: AI and LLM models produce a lot of data, but not all of it is trustworthy. Developers architect systems based on the assumption that the model they built will behave exactly the way they intend or expect. They don’t anticipate that it can be manipulated by opportunistic bad actors. And it’s not just the model itself at risk—manipulating even a small piece of data the model looks at can create vulnerabilities. Worse yet, the model doesn’t know it’s been manipulated, so it easily becomes an agent of the attacker.
- Unrealistic expectations: Because of their incredible potential, too often teams believe AI and LLMs are capable more than they are. But at their core, LLMs are nothing more than language pattern completion engines—they leap to the next logical word based on the patterns they’ve previously seen. They lack contextual awareness and can easily “lose the plot” without users or the models themselves realizing it—a risk developers can’t control.
These are significant risks for businesses, especially considering that survey shows the top use case for GenAI is developing business strategy, analysis, and planning. Entire roadmaps could potentially be built on questionable AI output. So, unless organizations find a way to mitigate vulnerabilities, their entire future and very survival could be at stake.
That puts CISOs under immense pressure to identify gaps, implement processes, and install guardrails for integrating AI into secure environments, helping steer developers towards architecting solutions that are resistant by default. That requires good governance, which includes keeping humans in the loop to guide AI implementation with an eye for compliance and risk mitigation.
How can CISOs overcome these issues, build security into AI applications by design, and mitigate risk—all while accelerating the adoption of LLMs for competitive advantage?
How business leaders can build more trustworthy AI systems
-
Recognize that pollution moves downstream.
LLMs access documents, prompts, and images from other models or untrusted sources, which means they can be exposed to data beyond what the developers specified. If an LLM accesses a resource that contains malicious information, the model can be influenced to move that data downstream, where it poisons the output.
When that happens, users and organizations can no longer trust the output of the model. It’s difficult to sufficiently sanitize that data and understanding and building security protocols based on this fundamental feature are essential.
-
Be aware of MATAs: Models as Threat Actors.
When an attacker can manipulate the model, they’re in control. It’s the equivalent of being positioned within the architecture. The model itself becomes a threat actor because it’s trained to perform the prescribed actions with ineffective sensibility for whether they’re right or wrong. Depending on the functionality exposed to the model, it can execute commands and/or extract all the resources it’s trained on (which could be your proprietary data).
If a threat actor writes malicious data into documents the model reads, and a user asks for a summary of those documents, the model can act on that embedded instruction, like executable code.
-
Implement data-code separation.
The most prudent approach to model security is one you’re probably already familiar with: create a gatekeeper code-separation that functions similar to a firewall so that trusted and untrusted models and data never touch or interact directly. The gatekeeper operates between two LLMs: the code-facing trusted model and the data-facing untrusted model. It manages the interaction between the two so that untrusted data can’t command any code functions.
When queries are made, the gatekeeper ties the two together and returns the filtered, curated output. This way, the code-facing model is never exposed the to contents of poisoned data, which mitigates the output of the attack chain.
-
Utilize good data management practices.
Even quarantined data can’t be easily cleaned—once it’s compromised, it’s nearly impossible to extract errant or malicious input. Instead, it’s best to make sure all content is tagged and masked so quarantined data is always obscured from the LLM, preventing it from executing any instructions or jailbreaks it may contain.
If you have to pass data between them, convert it to a strict and limited data type—using numbers instead of English text, for example—to avoid interpretation of the text as an instruction.
As we’ve covered before, while some artificial intelligence technologies and associated vulnerabilities are new, the same security fundamentals CISOs have applied over the last 30 years to traditional software apply equally well to AI and LLMs. The problem is that most CISOs just don’t have the tooling or expertise to apply those principles to this new paradigm.
Not to mention, it’s all happening way too fast. Developers are spinning up LLMs and AI applications, eager to experiment, turn out new capabilities and fully leverage the potential. Unfortunately, many are getting ahead of themselves when it comes to mitigating risk. CISOs need to pump the brakes to provide guidance, a measured approach, and proven strategies.
However, that's a lot to take on when the scope and pressure to scale are growing fast. Having an experienced partner is key to helping organizations avoid wading into dark waters.
Technical Director, NCC Group NA
David Brauchler III is an NCC Group Technical Director in Dallas, Texas. He is an adjunct professor for the Cyber Security graduate program at Southern Methodist University with a master's degree in Security Engineering and the Offensive Security Certified Professional (OSCP) certification
David Brauchler published Analyzing AI Application Threat Models on NCC Group's research blog, introducing new Models-As-Threat-Actors (MATA) methodology to the AI security industry, which provided a new trust flow centric approach to evaluating risk in AI/ML-integrated environments. David also released several new threat vector categories, AI/ML security controls, and recommendations to maximize the effectiveness of AI penetration tests.
AI security by design with NCC Group
NCC Group’s comprehensive AI security practice is designed to guide organizations in getting ahead of and solving their AI vulnerabilities. We go beyond prompt injection, diving deep into how pollution moves downstream, and how untrusted interaction with trusted models can compromise AI capabilities, output, and the organization itself.
Beyond pen testing and threat modelling, our experts can perform a full architecture review, validate your protocols, and provide best practices based on our breadth of experience across a wide variety of industries, use cases, and architectures. This helps you to identify previously unrecognized vulnerabilities and build processes and controls to rein in future risk.
Because securing AI is a multi-faceted concern, we can help bring CISOs and dev teams to the table to collaborate and achieve shared goals. This ensures everyone’s working from the same playbook to secure what you have and move forward with confidence, building trust by design into AI applications.
Learn more about NCC Group's AI security solutions
Our research-driven experts are ready to help with even your most complex challenges.