PostHole
Compose
Login
SE Radio 693: Mark Williamson on AI-Assisted Debugging
Mark Williamson, CTO of Undo, joins host Priyanka Raghavan to discuss AI-assisted debugging. The conversation is structured around three main objectives:
understanding how AI can serve as a debugging assistant; examining AI-powered debugging tools; exploring whether AI debuggers can independently find and fix bugs.
Mark highlights how AI can support debugging with its ability to analyze vast amounts of data, narrow down issues, and even generate tests. From there, the discussion turns to AI debugging tools, with a particular look at ChatDBG's strengths and limitations, with a peek at time travel debugging. In the final segment, they consider several real-world scenarios and evaluate the feasibility and practicality of AI acting autonomously in debugging.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Mark Williamson, CTO of Undo, joins host Priyanka Raghavan to discuss AI-assisted debugging. The conversation is structured around three main objectives:
- understanding how AI can serve as a debugging assistant;
- examining AI-powered debugging tools;
- exploring whether AI debuggers can independently find and fix bugs.
Mark highlights how AI can support debugging with its ability to analyze vast amounts of data, narrow down issues, and even generate tests. From there, the discussion turns to AI debugging tools, with a particular look at ChatDBG’s strengths and limitations, with a peek at time travel debugging. In the final segment, they consider several real-world scenarios and evaluate the feasibility and practicality of AI acting autonomously in debugging.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episode
Related Resources
- LinkedIn: @mark-williamson-undoio
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Priyanka Raghavan 00:00:19 Hi everyone, this is Priyanka Raghavan for Software Engineering Radio and today we’ll discuss the use of AI for debugging. And we look at three aspects in this show. One is going to be about using AI as an assistant to debug. Two, AI debugging tools. And three, is it possible that an AI, if given a bug, can help fix and do this autonomously? For this we have Mark Williamson as a guest, and Mark is the CTO for Undo. He’s also a specialist in kernel-level, low-level Linux embedded development with a wide experience in cross-disciplinary engineering. He programs a lot in C and C++ and one of his proudest achievements from the Undo website is his quest towards an all green test suite. So Mark, welcome to the show. Is there anything else in your bio that you would like to add apart from what I’ve just introduced you as?
Mark Williamson 00:01:18 I think that’s a pretty good summary. I guess in my time at Undo, most of my last 11 years has been a quest to get people to appreciate debuggers more, and I’m glad to be here talking about them. They’re one of my favorite subjects.
Priyanka Raghavan 00:01:30 Great. So we’ll kick off the show by asking you to define debugging in a professional software engineering context and how does it differ from simply fixing bugs?
Mark Williamson 00:01:42 Thanks. I really like this question because I think it’s often misunderstood. Developers spend most of their time at the computer debugging. It’s easy to have a view that bug reports come in from the field from customers. They go into a GitHub issue tracker or something like that. They get taken out and a developer fixes the bug. But I would say debugging is the quest to understand what your program is doing and why it’s not what you expected and that starts the instant you’ve typed in your first code. So I would say that most of development is self-debugging. I’ve seen a lot of stats recently that only about 30% of developer work is programming and therefore coding agents aren’t solving the whole end-to-end problem. But I would say that probably 80% of that 30% is debugging, not typing in the code. Code generation is a very small part of what developers do, and a lot of the technical work is this debugging process of answering questions and gaining understanding.
Priyanka Raghavan 00:02:55 One of the things I wanted to ask you is what do you use debugging for? So if a program does not function as the way it is supposed to, then that’s called a runtime issue, then that would be something that you would debug. But how about a case when it’s not performing very well? Is that also a case you would use debugging for?
Mark Williamson 00:03:18 I would say yes. I think different developers might call this a different process, but I would say debugging is any time you are trying to answer what happened or why did that happen, and that includes performance issues, but you have to then broaden your understanding of what a debugging tool is. So I would say there’s lots of tools you can use. You can add printfs into your code. They’re often logging frameworks. There’s also system level utilities like Strace, GDB, Valgrind, and perf and even also version control. In order to go back and figure out when a regression came in, I would say performance is in that continuum. So you might use a performance profiler, but actually why did the control flow bring you to the hot path? Well that’s maybe logging or a debugger, and it’s also a question of using each tool’s output to figure out what the next tool you apply should be.
Priyanka Raghavan 00:04:16 That’s interesting. This brings me back to one of the episodes we did in SE radio, which is 367 which was on debugging and the host there had questioned the guest how debugging differs based on the languages paradigms, whether you are debugging a monolith versus a microservice or just the way how you use the tools like as you said. So in your experience, maybe could you talk a little bit about that as well?
Mark Williamson 00:04:45 Yeah, there is a lot of variation I’d say. So what I’ve noticed in the field in my experience is there is a common denominator in everybody’s debugging experience and that is putting more print statements in. It’s probably the first thing you do when you are learning to program, and it carries on. And then there’s the grownup version of putting more print statements in, which is structured logging and open telemetry and things like that. I’d say that is common to all languages and all paradigms of programming. When you get into different more advanced tooling, I think there’s often analogs but it’s different. So most languages have a decent debugger and the tool we call a debugger is only one aspect of debugging, but it typically has some core operations. It lets you step through code; it lets you print variables. The way that works is very different depending on your language and your runtime.
Mark Williamson 00:05:42 So interpreted languages tend to need to tackle these problems very differently to compiled languages. And different languages have different takes on these as well. Same goes I think for any kind of mechanical tracing, any performance monitoring. Most of them also have tools that let you time travel debug these days. But again, the exact implementation and the technique can vary depending on what language you’re talking about. One last point is the distributed case where you’ve got multiple processes. I would say that is just hard. One thing about a monolith, it’s a lot of code to understand, but at least it’s all in one place. Once you start having multiple interacting systems, that is another level of complexity to kind of wrangle and manage even though the individual pieces might be simpler.
Priyanka Raghavan 00:06:30 So I think now let’s move on to the part about using AI for debugging. And it’s been now about two years where we’ve been losing a lot of LLMs to generate code and what do you do once the code is produced you need to debug. So in your opinion, how can AI be used to debug? Is it promising? What’s the, I think message from the field and the work that you’ve been doing?
[...]