PostHole
Compose
Login
SE Radio 692: Sourabh Satish on Prompt Injection
Sourabh Satish, CTO and co-founder of Pangea, speaks with SE Radio's Brijesh Ammanath about prompt injection. Sourabh begins with the basic concepts underlying prompt injection and the key risks it introduces. From there, they take a deep dive into the OWASP Top 10 security concerns for LLMs, and Sourabh explains why prompt injection is the top risk in this list. He describes the $10K Prompt Injection challenge that Pangea ran, and explains the key learnings from the challenge. The episode finishes with discussion of specific prompt-injection techniques and the security guardrails used to counter the risk.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Sourabh Satish, CTO and co-founder of Pangea, speaks with SE Radio’s Brijesh Ammanath about prompt injection. Sourabh begins with the basic concepts underlying prompt injection and the key risks it introduces. From there, they take a deep dive into the OWASP Top 10 security concerns for LLMs, and Sourabh explains why prompt injection is the top risk in this list. He describes the $10K Prompt Injection challenge that Pangea ran, and explains the key learnings from the challenge. The episode finishes with discussion of specific prompt-injection techniques and the security guardrails used to counter the risk.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 673: Abhinav Kimothi on Retrieval-Augmented Generation
- SE Radio 661: Sunil Mallya on Small Language Models
Other References
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Brijesh Ammanath 00:00:18 Welcome to Software Engineering Radio. I’m your host, Brijesh Ammanath, and today my guest is Sourabh Satish. Sourabh is CTO and co-founder of Pangea and a serial entrepreneur with 25 plus year track record of designing and building security products and technologies. Sourabh has more than 250 issued patents, Sourabh most recently founded and served as CTO of Phantom Cyber, which was acquired by Splunk in 2018 and he previously served as a distinguished engineer at Symantec. Sourabh, welcome on the show.
Sourabh Satish 00:00:47 Thank you Brijesh. It’s a pleasure to be on your show
Brijesh Ammanath 00:00:51 Though we have not covered specifically on prompt injection in previous episodes of Software Engineering Radio. There are a few episodes which I’ve worth listening to get broader context. These are Episode 673, 661 and 582. In this session today, we will focus on prompt injection, but before we get into the details of prompt injection risk, I wanted to take a step back and clarify the context of the risk. For a lay person the use of LLM is usually asking ChatGPT or Gemini some question asking it to analyze some data or asking it to create an image for you. Since this is interfacing directly with the LLM, am I right in assuming there is no security risk here and the focus is rather on organizations that have built applications on top of a large language model or a small language model?
Sourabh Satish 00:01:38 Yeah, I mean it’s a great question. Let me try to give a little broader context and answer the question. LLMs are basically models which are trained on data up to a certain amount of time. So they typically cannot answer questions on current events like stock price or news events and so on so forth. And in case of a consumer application, it is usually about asking LLMs about some information and it is about things which are baked into a foundation model. And when we talk about foundation models, these are models which are trained on internet scale data on all kinds of data and information. Consumer use cases predominantly about augmenting these LLMs which have been trained up to a certain amount of time with current information because they are, as I mentioned, are not aware of current information. Whereas in case of enterprises the use cases mostly about augmenting these LLMs with input from enterprise specific data that is sitting in enterprise data lakes document stores, enterprise applications, which are usually restricted by access control measures and so on so forth.
Sourabh Satish 00:02:45 With regard to consumers, the extra data that is being augmented is still mostly public data or user’s personal data, but in case of enterprise data, the risks of the information that is being sent to the LLM has different implications. It could be data from role or group of internal users, it could be sensitive customer data, company proprietary information, IP and so on and so forth. And hence the risk level of interfacing with LLMs in case of consumer applications and enterprise application really is all about what kind of data is being exposed to the LLM and what kind of data is being leveraged by the LLMs to answer the question. Hope that makes sense.
Brijesh Ammanath 00:03:27 It does. So what you’re saying is that that is risk, but the level of risk is different based on higher on the enterprise end and a bit lower on the consumer facing generic LLMs.
Sourabh Satish 00:03:38 Absolutely. I mean the risk still lies. I mean users are still at risk of exposing their own personal information to the applications of the likes ChatGPT, I mean hopefully nobody’s asking what their credit score is by providing a social security number to ChatGPT. So there is risk, but the risk is really about users’ own personal information that they’re accidentally disclosing to the generative AI applications. Whereas in case of enterprise application, the risk is magnified because it’s not just about user’s personal information, but it is also about other users’ information, aka customer information or proprietary information about financials of the company or sensitive intellectual property information, code, secrets and tokens, et cetera, which has, as I mentioned, really different lens to the risk and magnitude to the risk.
Brijesh Ammanath 00:04:28 Thanks or that explains it. Going on the same theme, what is it about LLMs that make them so powerful and also risky compared to traditional software components?
Sourabh Satish 00:04:40 LLMs are traditionally generative AI models which have an awesome ability to interpret unstructured text and ability to predict next tokens based on the history of tokens it has seen and the ability to produce content which looks and mimics human text is really what makes LLMs really, really compelling for consumers. So it emulates a conversational experience for users because users can continue to interact and ask questions and on the basis of the history that it’s able to analyze, it’s able to carry on a conversation because it can answer the second question on the basis of continuity of the information that it was collecting based on previous questions and answers that were given in a conversational style interaction with the LLMs. So the whole conversational experience that is now possible by LLMs with huge memory and context windows really makes LLM very unique and powerful and just very easy to use by any and all kinds of users.
Sourabh Satish 00:05:45 It does not require technical expertise; it does not require programming experience and so on so forth. It serves the needs of all technical versus non-technical audience in a very easy to use fashion. That property of the
[...]