S1xE3: Teaching Students to Debug with Amy Ko
In this episode, we talk with Amy Ko, an Associate Professor at the University of Washington Information School. She directs the Code & Cognition Lab and studies human aspects of programming.
Our conversation focused on how to teach students to debug, a skill many of us undoubtedly struggle to get our students to do effectively. Amy suggests: step 1 is to have students articulate what is happening versus what should happen (current output versus correct output). Step 2 is brainstorm different ways (hypotheses) that might be causing the discrepancy and exploring each idea to see if it is the cause. If a student runs out of ideas before they find the bug, go back to step 1 and confirm they understand what should and should not be happening.
When asked to share something awesome in computer science, Amy talked about her interest in computer science history and Donald Knuth. Knuth is one of the originators of many core algorithms in computer science. He also spent 10 years cataloging every mistake he made while working on the typesetting programming language LaTeX. So his interests were broad and he also wrote bugs!
In Amy’s Too Long; Didn’t Listen (TL;DL) she emphasized that debugging is a primary skill and is something we should teach. And we are starting to find ways to teach this skill.
You can also download this episode directly.
Transcript
Kristin: Hello, and welcome to the CS-Ed podcast, a podcast where we talk about teaching computer science, with computer science educators, to learn how they teach and manage their classrooms. I am your host, Kristin Stephens-Martinez, an Assistant Professor of the Practice at Duke University; and joining me today is Amy Ko, an Associate Professor at the University of Washington. Amy, how are you today?
Amy: I’m doing great.
Kristin: How about you tell us about yourself? What do you teach, how many students do you have?
Amy: Yeah, sure. I’ve been a professor at the University of Washington at the information school, so not in a computer science department, for about 11 years now and I teach a lot of different things. I’m kind of mirroring some of my different expertise in research and teaching. So, I’ve taught about foundations of information to freshmen and sophomores. I teach a lot of HCI classes to our undergraduates and also to Master’s students. I teach software engineering courses. These are primarily electives to juniors and seniors who are on their way to jobs. And then lately over the past three or four years I’ve been teaching high school students computer science in the summertime in our Upward Bound program. So, first generation college students–
Kristin: Cool! And how big are those classes?
Amy: They vary quite a bit. That freshman seminar course can be 200 students at a time. So a big team of 10 TA’s, and a lead PhD student TA, and me, all the way down to the summer high school courses, which are often 15 students on campus. Where–they’re coming from their various high schools in South Seattle and visiting campus. And then all of the ones in between are usually about 30 to 40 students usually undergraduates, and master’s courses are about that size too.
Kristin: Alright, so, I wanted to talk to you today about debugging–because you have done a lot of work in debugging; and I’m really curious about how to help my students with debugging more. Because I feel like on the forum posts, or in my office hours, I often get this thing where the student comes to me and goes, “Something’s wrong with my code, fix it for me.” And I’m like, “What’s wrong? What have you tried?” And the student’s like, “I don’t know, fix it for me.” I’m like, there must–there needs to be more productive discussion here, and it’s not quite clear to me how to go about doing that. So what are your debugging steps that you’ve been researching and have your students do?
Amy: Yeah. I’ll talk about the one that we taught to the high school students that we’ve been teaching in the summertime. This one was intended to really target students that had very little programming experience, no debugging skills, and it was also a heavily scaffolded strategy. So it starts off by basically saying: first, write down what you think the problem is. And the reason for this is because often times learners are looking at some program output, and they think it’s wrong; but they actually don’t quite understand what the program output’s behavior is, or what what the problem is. If you can’t write down precisely what they want to be different about the output, they can’t really debug it. So trying to be precise about this didn’t–shouldn’t print, right? This number should never appear, or this number is 6 and it should’ve been 5. Like getting that really precise description of the problem down first is key.
Kristin: Oh, okay. I think Iike, I do that naturally, but that–that never occurred to me to make students articulate that.
Amy: Yeah. And when they don’t, what we found is that they often have really vague conceptions of what the–the faulty behavior is. They might say, “It’s not right. Something’s broken.” Right. But it turns out that behind that ambiguity and those vague statements, is that they don’t even understand what the program was supposed to be doing. They don’t understand the prompt that an assignment gave them; or if they were the ones envisioning the behavior of the program and trying to design their own behavior, they may not even have a really good understanding in their head of what behavior they want. And so the wrongness of some program output isn’t always that well specified. And if you don’t have that you can’t really debug something, right. It’s–you have to know exactly what you’re trying to fix. So that’s the first step. So, the second step, once somebody has figured that out, is really trying to enumerate possible explanations for what might be going wrong. And this is just the observation that one approach to debugging is to do it scientifically, and generate some hypotheses and test them.
And by doing multiple hypotheses, it really helps learners brainstorm possible divergent reasons why the program might’ve gone wrong. Maybe it was that line of code you just edited, maybe it was something that you wrote a while ago, maybe it was an interaction between two things. We found that that student had a lot of time generating these possible hypotheses, but it was a great way to bring a social element into that active learning. So, they could pair up with somebody that they were sitting next to and say, “Can you help me think about reasons this might be happening?” Or, go to a teacher or a teaching assistant and brainstorm together collaboratively. Just like we’ve seen experts do in practice. And then, once you have a list of those possible explanations, going through that list iteratively and exploring each of them independently. And that that exploring of those hypotheses involves going and trying to precisely understand how the program was executing, and whether or not that part of the program you thought might be wrong was involved in it doing something incorrect, or undesirable. And this is the hard program comprehension that I–that I mentioned before. This is the part that is time consuming, and requires some detailed careful precise thought–that not all students want to have to do.
And so that process requires, sometimes, some support of, you know, they’re in the middle of trying to understand what they’re, you know, what some function call is doing. And then they’ll say, “Well, I don’t think I understand what this is.” Right? And then, there’s a little bit of teaching that might happen that a TA might help with. And once they understand it, maybe that was the defect, maybe it wasn’t, but they’re learning, then, about the behavior of the program. Sometimes that leads to new hypotheses. “Oh! Well if that’s doing that, then maybe this is happening.” Right? And then they might explore those.
Kristin: Yeah. So–so, is that like a breadth for search, or a depth for a search for the students, as they’re trying to debug these various ideas and what’s wrong?
Amy: Yeah. This is–this is very breadth wise, with respect to hypotheses, but then depth wise for the program behavior. Right? If you–if you think that a particular thing is causing the problem, fully explore that until you’re confident that either it is the problem; or it’s not and then move on to the next one. And this is very different from, let’s say, a strategy that, let’s say, follows program execution forward from execution. Right? You could have a strategy that says, step through the program using your debugging tool, line by line, and understand the behavior of everything that’s happening, until–until you notice the defect. And that is a pretty reliable way of doing it. It just might take a really long time, depending on the complexity of the program. So, it’s in some ways more of an optimization for that. Right? It will always work to watch every single step of the program execute until you encounter the defect, but it might take hours.
Kristin: You might take a very long–well, if we’re talking about, say, CS1 students, that probably won’t be as bad–
Amy: Right.
Kristin: As–as others, though, I definitely, I’m thinking of some particular bugs I’ve seen students go through like every single semester, so I know it’s going to happen. And some of these bugs are the–it manifests in one file, but it actually appeared and had like–the fix is in a completely different file! And the students are like flabbergasted, like, “Where is this bug coming from? How is this possible?” And I’m like, I know exactly what’s going on. But now I have to figure out how to guide you through the process… to figure out on your own what is wrong.
Amy: And that really surfaces these really interesting questions about: what is the learning objective of a programming assignment, right? Are you trying to help them understand the algorithm that they were studying? Do you want them to engage in that learning about programming–I mean about debugging strategies, right? Do you want them to learn how to be resourceful to get help, to support all of these different problems they encounter? Some programming assignments have the goal of teaching all of those things. And–and that makes them really hard to teach. If debugging is not a huge priority for an assignment though, maybe you just give them a list of all of the possible defects they might encounter along the way, right?
Kristin: Yeah.
Amy: And they’ve removed that from the problem.
Kristin: I’ve seen some people approach–try and help students with debugging in that way; where if the assignment’s goal is not really about debugging, they give them a list of possible ways that they could screw up and like where all the bugs are. But that–that results in a laundry list of an FAQ that no student reads. And I–it’s, it’s not–and I don’t know how much I would. I guess part of me is thinking, how, part of an assignment’s goal is–I guess you could say is– the apprentice model, where you’re going through this process to learn the whole process of creating a thing. And–but that does sound like a very large learning goal, that is not clear, is maybe not the best learning goal for the circumstance, now that I’m thinking about it, and talking about it out loud.
Amy: That is the big challenge, I think, of any teaching around programming and computer science is, there’s a lot to learn, right? And in any good teaching, it has to really, in some ways, elegantly deconstruct all of that and serialize it in some way, so that it’s possible to learn. Because if you throw all of it at somebody, all at once, they will probably fail. It’s too much.
Kristin: Yeah.
Amy: And I think we see that happen a lot, even in CS1 classes, where–we assume a lot of prior knowledge, but they–a lot of students actually don’t have it, and even that first week is an overwhelming amount of information to have to process and internalize.
Kristin: Yeah. I wonder if we’ll–I would hope that at some point, we will come up with an actual sequence for CS1, whether or not–when we do, will probably take time. I am currently working my way through your–the computer science education journal article, that Ben wrote–
Amy: Oh, yeah, Benji’s paper.
Kristin: And into the four pieces of–was it… tracing, syntax, writing? I’m missing one. Templates.
Amy: Yes.
Kristin: And, it’s like a combination of: I love this. This is great. But, at the same time it seems like a lot. To break it down that much and, it probably is partially just because of my own biases; because I was lucky enough to have computer science in high school, where I programmed in high school. I programmed in QBasic, so that dates me. But, part of me–I know is– I’m working through my biases, of like, it makes sense for students to go through this process. But I–I learned it through–so much through–osmosis in high school, that I’m like, “Of course it’s normal!” And I’m like, “No, I am not normal.” I should not think of myself as normal in this context!
Amy: And that notion of osmosis, I think, what osmosis usually means, is that all of that learning happens somewhere, sometimes, supported by something, we just don’t often remember what it is.
So like me, learning to program on my T-I basic graphing calculator, right. Where did I learn the syntax of that basic language, for that programming language? And I realized, after trying to go back to some of those memories, there was this instruction manual for the graphing calculator, and there were about 80 pages on the programming language. And it was basically a little textbook, explaining to me the semantics of every little construct, every little control flow idea, right? Somebody sat down and wrote a textbook to teach it to me, and I read it! Over, and over, and over again, until I understood what the program was doing. And that was kind of separate from the debugging that I was doing. And, you know, in some ways, I wasn’t even writing anything. I was taking some Tetris game, that somebody had already authored; very–very carefully and methodically trying to understand the language, and then understand that program, and then modify one small part of it to make it be–more performance, so that it was faster, right? And that was months, right? Months of–
No teacher, I probably would have–have been faster, if I had had a teacher. You know. But I think that my memory of all of that, before I’d really piece through what had happened, probably would have described as osmosis. You know, I was just looking at a lot of code and I figured it out.
Kristin: Yeah. What’s horrible about a statement like that, if you think about it, is students say those kinds of statements all the time and they make those who have–who are such novices–feel so insecure, and so like, “Aww, everyone else has got it except me, I must be stupid.”
Amy: Yeah.
Kristin: And my heart goes out to those students so much.
Amy: I think it also interacts a lot with–going back to the culture of computing–computer science, especially in higher education, is a place where people like to signal their intelligence, right? It’s not a place where we say, “You know, I had a hard time with this. I worked really hard to get to this level of understanding.” It’s a place where people say, “I figured it out because I’m smart and I’m clever. Look at this clever thing I did.” Right?
Kristin: Yeah. I try so hard to normalize, like, struggling and taking forever on something in my classroom. I think I’m partially succeeding. I don’t know how well I’m succeeding, but from some of the anecdotes. and some of the thank you notes I get I feel like, I’m succeeding! I succeeded with this student. I show them that like–
Amy: I always struggle with that as a teacher as well, because how do you balance projecting your authority, so that you have that resource to–to teach? But at the same time, sort of model that you’re not perfect, and that you have to work through hard problems, too. And–
Kristin: Yeah, I think I–for me, I draw on stories from when I was younger, to make it seem like the Old Me, when I was your age, was like totally crazy and did not know what they were doing. The new–the Current Me knows totally what I’m doing. Most of the time. But yeah. It’s one of those funny things, where like, you don’t want to lose–I guess, maybe a better way of putting it is, you don’t want to lose the respect from the students, so that the student no longer believes that you can teach them what they need to learn.
Amy: Right. They still have to believe in your–your knowledge. They still have to believe in your abilities as a teacher.
But they also have to see that your abilities came from somewhere.
Kristin: So, to get back on track, considering the time. You–for your debugging steps, are there just two steps? Or are there more steps than that.
Amy: Ah, so. So, in that loop of exploring those different hypotheses –
Amy: And trying to localize, you know, was this particular guess that you made a good explanation of what happened.
Amy: You know, if you find the defect, then inside that loop it’s about trying to identify what you can change about the program to prevent that behavior from happening, while still preserving the behavior of everything else. And that’s a hard task as well. Right? That becomes a collaborative thing as well. Can you talk to classmates, to a teacher about identifying what that patches.
Amy: And then we have this really fun thing, outside that hypothesis testing loop at the end, which is, what happens if you didn’t find it? Right. What happens if all of your hypotheses were wrong? What do you–what do you do next?
Amy: And in some sense there’s a big outer loop, which is, we’ll go back to the beginning, and try–try again. Because, there’s probably an explanation out there, and maybe it has to do with you not quite understanding what the–what the goal of the program is in the first place.
Kristin: Okay. So, in some ways yours is not–I don’t know if I’d really call that a three step process, it’s more kind of step one, step two, repeat step two until you run out of ideas, step three is go back to step one.
Amy: Yeah, there’s absolutely control flow in these strategies. There are loops, there are lists of things you need to remember to go through. Right? And this is why they–they require in that same way lots of self-regulation. You have to be, you know, writing these things down somewhere.
We’d actually worked on a tool to support that process, right? So, the strategy is written down, there’s a scratch pad to write down all of the data you need to keep track of while you’re going through your process. And then there is a tracker, of where you are in that strategy, so you don’t lose track of where you are.
Kristin: So, as we wrap up, the next thing that we want to talk about is something awesome in computer science. Do you have anything in particular that you’re–you’d like to talk about in computer science? That you think are as not as well known, as maybe some people should know about it, but you think is really interesting.
Amy: Yeah, there’s–I mean–there’s so many things to choose from. I get really interested in history around different things, and how the field has evolved over time. And, especially since it’s such a new field, right? We didn’t really know the origins of how people think about this in interesting ways. So, one particular piece of history that I’m really fascinated by is–there is a computer scientist, Donald Knuth, who many people know as the originator of lots of core algorithms and computer science. He wrote lots of very large tomes, you know, really describing the performance qualities and–of many popular algorithms that are sort of ubiquitous today. So, hugely impactful researcher. But he wrote this paper, in the middle of all of that work, while he was working on this typesetting programming language, called LaTeX–
That a lot of people still use to write research papers, both in computer science and mathematics. So, for ten years he was working on this typesetting program, while he was doing all of these algorithms things, and he just spent every moment of those ten years working on that project cataloging every single mistake he made. Right? So some people think of him as this sort of god of computer science, amazing algorithms and inventions, really shaping the field. And here he is in the background, just writing down every single little, tiny mistake he made, and how it just kind of caused all kinds of problems in his programming.
So, it’s just this wonderful example of, you know, that that really sort of hidden, secret thing about computer science culture, where, yeah, we’re signaling all of these really amazing things that we’re doing; but we’re often not talking about the mistakes we make, and all of the challenges that we face in trying to arrive at some of those amazing things. So what Knuth did that was amazing, was he did tell people about all of those mistakes. So after he’d catalogued them for 10 years, he wrote a journal paper writing about all of those mistakes that he made, classifying them into different categories. We know now that many of them make no sense, and have, like, better science on it now. But, he was really one of the first people to say, “Well there are these different kinds of errors that people make, and we should understand those kinds of errors.”
This was also at a time when computer science was really trying to formalize what rigorous research looked like. And it was very much heading in the direction of mathematical formal proofs about program correctness, and at–at most maybe some quantitative empirical data to talk about performance and other types of things.
But his journal paper was a purely qualitative effort. He was describing journal entries that he’d written for a decade. Right? He was analyzing qualitatively what he wrote in those journal entries. He was coming up with categories about his behavior and his thinking, right? And this is a person who is doing mostly formal stuff–
Kristin: Yeah.
Amy: And mathematical proofs of complexity for algorithms. So I find it just to be the most fascinating example of just how nuanced and–and sort of pluralistic computer science is at its heart. It’s really both mathematical, and social, and cognitive. And one of the founders of the fields embodied all of those things in that work. I think we should always remember those–those details and go back in that history to remember where we came from.
Kristin: Yeah I–I love the idea of trying really hard not to appear perfect to people, because I feel like we do that a little too much in our society right now, appearing perfect. And that puts pressure on everyone else to also feel–be perfect, and we’ve gotten to the point where if you’re not perfect, some people think that something’s wrong with them. And that–that is–that is a tragedy that I don’t think is–is healthy on any level. And, I feel like this is–that’s one part out of our society that I wish would change.
Amy: Yeah, and in our own little corner of the world in computer science, when we’re teaching it, we have a huge opportunity to do that, right? I think it’s particularly challenging, because programs and computers expect us to be perfect.
Kristin: Yeah!
Amy: We sort of–relentlessly unforgiving of our mistakes.
Kristin: Alright, so let’s close out with our last segment, TLDL, Too Long, Didn’t Listen. So, given that everyone is busy, what would you say is the most important thing that you want our listeners to get out of our conversation?
Amy: I think the really central idea is that debugging is a really primary skill, that is necessary from the first line of code that somebody writes, all the way through them finishing a program that they think is correct. It’s just embedded and interleaved in every single thing that we’re doing when we–when we write programs, when we write software. And therefore, we should probably teach it. When we don’t teach it, students really struggle with it. And I think we’re starting to discover some ways of teaching it more effectively. Some of those ways involve having step-by-step strategies that learners can follow, and then helping learners practice those strategies over time. And I’m hopeful that with some further research, we’ll get a really strong sense of exactly how to do that, how to support that, for students with diverse prior knowledge, but also students who are neurodiverse in lots of ways.
Amy: So, I think there’s a lot of interesting challenges that teachers have to face now in classes.
Amy: And that, through some research, we’ll be able to really support them in the next few years with further discovery.
Kristin: Thank you so much for joining us, Amy. And this was the CS-Ed Podcast hosted by me, Kristin Stephens-Martinez at Duke University, edited by Susannah Roberson, and funded by a SIGCSE special project grant. And remember, teaching computer science is more than just knowing computer science, and I hope you found something useful for your teaching today.