Download our 2024 Annual Research Report
Non-Deterministic Nature of Prompt Injection
As we explained in a previous blogpost, exploiting a prompt injection attack is conceptually easy to understand: There are previous instructions in the prompt, and we include additional instructions within the user input, which is merged together with the legitimate instructions in a way that the underlying model cannot distinguish between them. Just like what […]
Analyzing AI Application Threat Models
Abstract The following analysis explores the paradigm and security implications of machine learning integration into application architectures, with emphasis on Large Language Models (LLMs). Machine learning models occupy the positions of assets, controls, and threat actors within the threat model of these platforms, and this paper aims to analyze new threat vectors introduced by this […]
Machine Learning 104: Breaking AES With Power Side-Channels
This executable blog post is the fourth in a series related to machine learning and is a fascinating trifecta involving hardened cryptography software, embedded IoT-type hardware, and deep machine learning techniques. While the AES algorithm is designed such that a brute-force secret key guessing attack would likely finish ‘sometime near eternity’, the power side-channel attack […]
Exploring Overfitting Risks in Large Language Models
In the following blog post, we explore how overfitting can affect Large Language Models (LLMs) in particular, since this technology is used in the most promising AI technologies we see today (chatGPT, LLaMa, Bard, etc). Furthermore, by exploring the likelihood of inferring data from the dataset, we will determine how much we can trust these […]
Security Code Review With ChatGPT
TL;DR: Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does), and there are some other major problems that make it impractical. Also, both the CEO of OpenAI and ChatGPT itself say that you shouldn’t. Large Language […]
Machine Learning 102: Attacking Facial Authentication with Poisoned Data
This blog post is the second in a series related to machine learning, and demonstrates exactly how a data poisoning attack might work to insert a backdoor into a facial authentication system. The simplified system has similarities to that which the TSA is running a proof of concept trial at the Detroit and Atlanta airports. As background, […]