SE Radio 603: Rishi Singh on Using GenAI for Test Code Generation

Rishi Singh, founder and CEO at Sapient.ai, speaks with SE radio's Kanchan Shringi about using generative AI to help developers automate test code generation. They start by identifying key problems that developers are looking for in an automated test-generation solution. The discussion explores the capabilities and limitations of today's large language models in achieving that goal, and then delves into how Sapient.ai has built wrappers around LLMs in an effort to improve the quality of the generated tests. Rishi also suggests how to validate the generated tests and outlines his vision of the future for this rapidly evolving area. Brought to you by IEEE Computer Society and IEEE Software magazine. This episode is sponsored by WorkOS.

Rishi Singh, founder and CEO at Sapient.ai, speaks with SE radio’s Kanchan Shringi about using generative AI to help developers automate test code generation. They start by identifying key problems that developers are looking for in an automated test-generation solution. The discussion explores the capabilities and limitations of today’s large language models in achieving that goal, and then delves into how Sapient.ai has built wrappers around LLMs in an effort to improve the quality of the generated tests. Rishi also suggests how to validate the generated tests and outlines his vision of the future for this rapidly evolving area.

This episode is sponsored by WorkOS.

Show Notes

• Sapient.ai

• SE Radio 474 – Paul Butcher on Fuzz Testing

• SE Radio 167 – The History of JUnit and the Future of Testing with Kent Beck

Transcript

Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.

Kanchan Shringi 00:01:01 Hi all. Welcome to this episode of Software Engineering Radio. This is your host Kanchan Shringi and today we welcome Rishi Singh. Rishi has been a platform architect at Apple, a co-founder and CTO of @Harness.io, which is a CICD platform and he is now founder and CEO at Sapientt.ai. Today we will explore the technology and methodology behind how Sapientt.ai leverages GenAI to help developers automate test code generation. Rishi, is there anything else you’d like to add to your bio before we get started?

Rishi Singh 00:01:39 Hey Kanchan, thank you so much for inviting me. Great to be here. I think you really covered it well other than CPTI, @Harness.io and my stint at Apple, one thing that I can add is, I’m really, really passionate about the developer’s tooling. That anything that leads to help a developer become more productive. So before founding the CPTI, was more into the software delivery space with the CPTI, this is a bit in the upstream environment especially for the developers so that they don’t get stuck in the testing process. So yeah, I’d love to discuss more.

Kanchan Shringi 00:02:13 Before we get into the first set of questions. I’d like to point our listeners to episode 167, which is the history of J-unit and the future of testing with Ken Beck, which I think sets a good stage or how we are going to change some of those methodologies. Well while the methodology changes, the problems that need to be solved are probably the same starting with identifying what to test and input. Can you comment on that and maybe help us understand all the things a tester needs to solve or the developer needs to solve?

Rishi Singh 00:02:54 Yeah, that’s a great question. The software testing has been as old as the software itself, right? So ever since, started building the product you need something to test it. Even with the testing, has it itself evolved with the software development. Now if you recall back in the days we used to have the waterfall, software testing used to be a very significant stage in this entire software development lifecycle. And as you referenced about the Ken Beck episode, I had a chance to listen to that episode before this regarding. So, if you look back in the early 2010 to where we have come today, the problem statement remains the same. Because we want to assess the quality of our product, we want to make sure the product that we deliver is meeting the requirement and it’s helping the customers or their users to have a good experience.

Rishi Singh 00:03:48 But the software development landscape itself has changed. I think the way we are building the software, the way we are delivering the software, that itself has changed. And if you dig deeper underlying the requirement is the same thing. Broadly speaking, almost every product will have some functional testing requirement. We’ll have some kind of non-functional testing requirement and then you can just break it down into this multiple areas and you start tackling each one of them in a respective way and you solve it. So it’s really a right observation that the requirement itself is same, but it’s just the way we tackle has changed.

Kanchan Shringi 00:04:23 In addition to identifying the functional requirements and hands what needs to be tested to meet them, things keep changing all the time. So identifying what has changed and how to test the delta is another problem that one probably has to focus on.

Rishi Singh 00:04:41 Yes, yes. So back in the days we used to have a testing done by Acuity. Not everything was manual and that evolved as sort of test automation. So you’re not only just doing the testing one time, but you’re writing a program to simulate the entire process, set the steps so that you can test as many times. And now the test automation itself is getting replaced by some kind of automated process. So the test automation is generated, right? So the underlying philosophy that most of this QA engineer used to follow is what we call a test pyramid, right? So, what about the testing requirement that you had? You just break it down, otherwise it becomes quite overwhelming. So I’ll give you an example. Let’s say we have a web application, there is some sort of authentication layer in the front and then you have the actual web application doing it.

Rishi Singh 00:05:36 Imagine that it’s a brokerage application. It might have a, let’s say thousands of different use cases or thousands of different test cases that are emerging out of it. But then you have some kind of authentication that can vary. You might have the Google based authentication, you might do Okta based authentication, or it may be traditional, the user item, the password kind of authentication. But the way a senior QA engineer will do it is that they will take these two things in two layers of the application and test it independently. And so you have a three mode of authentication, but then you don’t do three times hundreds of these test cases. That becomes 300. Instead you do three testing of this authentication separately and you do these a hundred test cases separately.

Rishi Singh 00:06:20 And so it’s just total hundred three. So that is one example, right? It is the same philosophy that most of the QA engineer will follow. They will break down the overall testing requirement in the form of what we call it, unit testing, where you just go and focus on this individual, the test classes, individual methods, let’s make sure each and every methods are behaving the way it’s supposed to behave. They will do a sort of integration testing. Just try to identify those different logical layers within the code or your application and making sure these are all coming together. It’s all coming along. And then finally those end-to-end testing, some of the flows that you want to make sure as a user, when they are using the application all of them are coming together in result performing.

[...]

Original source