S4xE5: Parson's Problems
Episode Summary
In this episode, we have Dr. Barbara Ericson, assistant professor from the School of Information at the University of Michigan. Our topic is Parson’s Problems, which are like mixed-up code chunks that students need to put in the correct places. We discuss the research behind them, how she uses them in her class, and her current work investigating how to use Parson’s Problems to improve student learning.
You can also download this episode directly.
Episode Notes
Barbara J. Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S. Miller, Briana B. Morrison, Janice L. Pearce, and Susan H. Rodger. 2022. Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. In Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR ‘22). Association for Computing Machinery, New York, NY, USA, 191–234.
Barbara J. Ericson, Janice L. Pearce, Susan H. Rodger, Andrew Csizmadia, Rita Garcia, Francisco J. Gutierrez, Konstantinos Liaskos, Aadarsh Padiyath, Michael James Scott, David H. Smith, Jayakrishnan M. Warriem, and Angela Zavaleta Bernuy. 2023. Multi-Institutional Multi-National Studies of Parsons Problems. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR ‘23). Association for Computing Machinery, New York, NY, USA, 57–107.
Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 124–130.
Transcript
[00:04] Kristin: Hello and welcome to the CS-Ed podcast, a podcast where we talk about teaching computer science with computer science educators. I am your host, Kristin Stephens-Martinez, an Assistant Professor of the Practice at Duke University. Joining me today is Assistant Professor Dr. Barbara Ericson from the School of Information at the University of Michigan. Barb. Thank you so much for coming on the podcast.
[00:25] Barb: Thanks for having me, Kristin.
[00:27] Kristin: So, Barb, first, I’d like to ask, how did you get to where you are today? I am sure it is a wonderfully fascinating story.
[00:33] Barb: Well, I had got a bachelor’s in computer science and started working in research labs, and I’ve mostly worked in research labs in my career, and I did a variety of things. I did 3D graphics, user interface design, and case-based reasoning. And then, I was working as the Director of Outreach for the College of Computing at Georgia Tech. And I started doing research on what I was doing, but I didn’t really know what the research was. And I, I thought, oh, gee, I, I would ask my husband, Dr. Mark Guzdial, hey, what do we know about this? And, and he would tell me, but I didn’t really understand him. Like, what’s LPP? You know, what is Legitimate Peripheral Participation? What do you even mean?
[01:11] Barb: So, at one point, I was a research scientist and I was up for a promotion, and another research scientist was helping me in my promotion case. And I was kind of complaining, saying, “Boy, they’re making me basically do a tenure case, and I’m on soft money. I could be gone tomorrow.” And she said, “Oh, you’re right, you should leave.” And I said, “Oh, well, not so easy to leave because I don’t have a Ph.D.” And she said, “Well, if you’re going to stay, you should get the Ph.D.” And I thought, “Oh, no, I can’t imagine going back to school at my age or taking the GRE. Are you crazy?” And she said, “Well, sit down and see if they’d waive that.” That never would have occurred to me. And I, so I did, I sat down and I said, “Would you waive this?” And he said, “Sure.” I said, “Oh, ok.” And I could do it for free because I was already working there. So, I was like, ok, I’ll do my Ph.D. for free while I’m working full time, with three kids. That sounds fun.
[02:02] Kristin: I, I only have two kids and I did most of my grad school without them. I can’t imagine.
[02:09] Barb: They were older was one of the reasons I could do it. I did ask my son before I started because he was, he was an undergrad at Georgia Tech. And I said, do you mind if mom goes to school where you, you’re going to school? He said, oh no, that’s fine. You’ll be doing a Ph.D., and I’m an undergrad, except then he did go for his Ph.D. So he was right after me also, doing his Ph.D.
[02:29] Kristin: Oh That’s hilarious. Did do, was there any comparison between the professors of like Barb was better at this than…
[02:36] Barb: It’s very funny. We both took AI. He is now a computer science professor in, doing generative AI sort of stuff with creativity. And we both took this AI class, but it wasn’t generative AI class. It was, you know, old symbolics AI stuff, which I thought it was fun. And we were doing Raven’s intelligence tests, which are visual, this is like this as what is to like what? And we wrote programs to solve them. And I thought this was so cool and so much fun. So, I spent a lot of time doing it, and then I took it, and then the next semester, my son took it, and then the professor said to my husband, what’s with your family? Because we both spent a lot of time on them and did a lot.
[03:17] Kristin: Oh, it’s, it’s so much like birds with a feather flock together for you all. I love it. All right. Well, I love hearing how people got to where they are because it’s always a more windy path than you think it is. And it’s important, I think to, to remind people of that because the CV doesn’t tell it all.
[03:35] Barb: Totally true. I never would have expected this is where I would end up.
[03:38] Kristin: All right. So let’s segue to our primary topic, which is Parson’s Problems, which I know you’ve worked on a lot. And at first, you suggested it would be a teaching practice byte, but I felt like it wasn’t quite, it was too much for a practice byte. So I decided, like, let’s just have a full interview and see how we go, and I’m sure we can riff for a while and enjoy a conversation. So let’s start with what are Parson’s Problems?
[04:03] Barb: Well, Parson’s Problems are a kind of completion problem or partially complete problem where learners put mixed up code in order, in order to solve a problem. There are a lot of varieties of Parson’s Problems. But you can think of those like refrigerator magnets, right? Where you can create a sentence with blocks, and you can put the block, the blocks are words, and you can put them in order to write a sentence. So some Parson’s Problems, the original kind, were one dimensional where you had blocks that had one or more lines of code in them, and you put them in order vertically. And then there became two dimensional Parson’s Problems where you had to put them in order vertically but also indent them correctly horizontally like you have to in Python.
[04:41] Kristin: Okay, Is there any, is there any Parson’s Problems that the three dimensional sounds wrong? But I’m almost kind of like, do you give the students only pieces of the line, and they have to put the line together first? Is there anything like that?
[04:52] Barb: This is a new version one of my Ph.D. students has been working on Zihan Wu has been working on micro Parsons is what we’re calling them where you put pieces of a statement or like regex. So, the little characters that make up a regex. You put those together to make a statement.
[05:10] Kristin: OK. So, but I’m assuming that that hasn’t had nearly as much research then on it compared to Parson’s Problems. So where do Parson’s Problems like fall in the ecosystem of how to help someone learn to code?
[05:22] Barb: Well, typically, many computer science courses or intro programming courses require students to write code from scratch. And that’s what’s called a complete, a full problem or a whole complete problem. And that’s why we call this a completion problem. This is different, right? This is everything from scratch. This is saying, hey, you have all the correct code there. It’s just mixed up, right? So you don’t have to type it all. It’s partially complete, which reduces the the search space. How do I even solve this? Right? We know that a lot of novice programmers find that extremely intimidating to start from nothing. Now, a lot of people will do worked examples, post-practice approach, but that’s still very difficult for students to go from. OK. Here’s an example problem. OK? Now you go write one. So we’re trying to bridge that gap between, oh my gosh, you know, we gave you some example code, and you’re just learning how to write code and OK, how can we assist you get to a correct solution?
[06:18] Kristin: So have you done much work, or is there much work on looking at, like, by giving students Parson’s Problems, we can help them get to, I don’t know if it’s faster at writing their own code? Like what is the performance goal in the end, then? Because, like, by the end of CS1, your goal is for them to write code on their own.
[06:38] Barb: Yes. So, we do want them to write code on their own, but we want to scaffold that process. So we don’t overwhelm them in the meantime. So a lot of our work is based on cognitive load theory that says, you know, that you can only hold so many things in working memory. And if you overload working memory, you can’t learn. You impede learning. So you have to manage the difficulty of a problem as you’re learning it. And so we’re trying to help students chunk things even literally into blocks, right? These are blocks that go together to get them to chunk about those ideas and to start to recognize common errors using Parson’s Problems. So, you can also have distractors or blocks that you don’t use in a correct solution. But that can help you point out common syntax or semantic errors students make.
[07:23] Barb: So we see them as a nice scaffolding from OK, well, you, you saw some example problems great and to get you to attune, to pay attention to those example problems, we give you a Parson’s Problem to use. OK. Well, here’s some example code that we explained to you. But now, OK, try to solve the Parson’s Problem, and there’s distractors that have common errors to try to get you to pay attention to the kinds of things people get wrong. And then there are also some other types that are trying to fade. So, scaffolding should fade as you develop expertise. In fact, one of the things we found is that even though most students, and like 75% and up, say that they find Parson’s Problems useful for learning to code. There are people that have more experience often who have a very strong negative reaction to them, saying, oh no, I’d rather write the code myself and solve this problem.
[08:12] Kristin: So you say students find it useful to help them learn to code. Does Parson’s Problems actually make them more efficient at learning how to code? Like how does that fit into? Because, like, when I teach CS1, it’s a lot of, like, here’s a bunch of problems to solve and here’s a bunch of unit tests. So, you know, when you’re done and that kind of thing, and there’s no, I don’t use a bridge, but would a bridge like be worth removing some of the problems that they have to do so that they, and then instead solve those problems with Parson’s Problems. Like how would I fit this into my curriculum to try and make the class better for the students?
[08:52] Barb: Yeah, we know that giving people lots of practice. That’s part of why we use worked examples plus practice. So, just practice is less efficient learning than worked examples plus practice, where you give them worked-out examples and then give them practice and interleaved, worked examples plus practice. But in the practice, you know, how do you get them to pay attention to the worked examples? So we see Parson’s Problems as one way to use work examples plus practice approach. And yes, the research, one of the studies that I ran, was a randomized controlled study of people in different conditions. Some people solve Parson’s Problems. Some people wrote the equivalent code, some people fixed code with errors, and we found the people in the Parsons condition were significantly faster than either other condition. The worst was actually fixing code, novice students are the worst at fixing code than anything else.
[09:41] Kristin: It kind of makes sense, though. It makes me worried because of, I don’t know if we want to talk about large language models, but it does make me worried if that’s if that’s true.
[09:50] Barb: Yeah, well. But they were so they were significantly faster at solving the practice problems with no difference, no significant difference on the post-test. So they didn’t learn the same amount in less time. And we know from lots of other research that people find writing code from scratch very frustrating. It takes an unexpectedly long amount of time because they get stuck and they can’t get help. So, we’re trying to increase learning efficiency. Can you learn faster, better if we don’t make you be stuck forever?
[10:21] Kristin: Yeah. So, if that’s true, then does that mean we should like throw out all the practice problems and replace them all with Parson’s Problems? Are you more saying like let’s do a 50/50 split?
[10:33] Barb: Well, what I typically do in my class is when I introduce a new concept in lecture and I often will do pre-readings and the readings, the interactive ebooks that I’ve been creating have Parson’s Problem in them as well. So, in the pre-readings, they’ll see some Parson’s Problems to try to figure out how they’re doing things. And then as I’ll do a lecture, I’ll have a lecture. I have an hour and 20 minutes, which is a long time. So I don’t try to do a lot of lecture. I try to do a lot of active learning. I’ll do POGOL exercises, which are process-oriented, guided inquiry learning, which is a huge mouthful, which basically means group work where you’re giving them things that you’ve carefully designed so that they learn the things you want them to know, but you don’t just tell them them. So we do POGOL and then I’ll have them practice with the things, the code that I’ve been talking about, the code that they’ve been reading about. They’ll do Parson’s Problems.
[11:22] Barb: OK. So now, as a transition from seeing me talk about them, you may be having to do some Parson’s Problems as part of your reading now and, and I give them choices. So, since we learned that some people would rather write the code and scaffolding should fade as you develop expertise, I’ll usually give them a Parson’s Problem first, where they can switch to the write code if they would rather, and we grade when they do. And then secondly, I’ll give them a write code problem where they can pop up a Parson’s Problem to help them. And then I’ll give them a set of four where I’ll do Parsons, but I can switch to write code, write code, but I can switch to Parsons. I can pop it up and then the same thing again. Parsons and I can switch to write code or write code and I can use the Parson’s Problems. So I’ll do about four practice problems and have them put questions on a Padlet so they can post questions anonymously. And so that’s a typical thing I’ll do in lecture as a way of practice.
[12:15] Barb: And then in discussion, which is when we’re in small groups because I have 200 people in a lecture. It’s a lot easier to help people that are stuck in a smaller group. So, in discussion, they will have them write code, practice writing code. But in working in groups and working on functions that are similar to what we’re asking them to do in the homework. So, they get this scaffolded approach.
[12:38] Kristin: OK. So, it’s the Parson’s Problems are most present in class where you can handle it at scale because there’s so many of them, and it’s slowly fading. And then does that mean that their homework is actually not Parson’s Problems? Their homework is actually practicing and writing code.
[12:54] Barb: Homework is writing code.
[12:55] Kristin: Got it. OK. So it’s Parson’s Problems. Like I think in my head. At first, I was imagining you were assigning Parson’s Problems with homework, but really, it’s more, it’s higher up the chain when they experience the concepts for the first time, as well as in lecture when you’re there to kind of get a sense of what’s going on because you very carefully designed a thing for them to practice on. So, could you go into detail on that one real quick? Like the POGOL? Like how do you carefully design a thing for them to practice on in class?
[13:27] Barb: So, I’ll do something like I turned most of my lectures into POGOL exercises. So, I look at my lectures. OK, what is it? I’m trying to get them to learn. And so for example, one of the things my students, I teach a second required Python programming course for School of Information majors. So coming in, they’re supposed to already know what a variable string, conditional, loop, function are not necessarily in Python. I don’t restrict it to only people who have taken the first course, but that’s what they’re supposed to know coming in. So that’s where I start day one lecture. OK. Here’s some code. So often, POGOL is around a model that they look at, and then they answer questions about. So the model is some code I want them to look at and it’ll be a function and then another function that calls that first function and then maybe a main that’s calling that other function. And I’ll ask them to predict, for example, what’s the first thing this code is going to print? And sometimes, they have the mental model that the code is going to run in the order I see it. Rather than OK. Well, the main is called first, and then it is calling this function, which is calling this function. So, I asked them to predict code. What’s code gonna do? I asked them to put you there might be Parson’s Problem as part of the POGOL exercise, but it’s often around a model. Here’s a model of code. You can run the code, you can see what happens. But I’m asking you to predict or explain things like in Python, you can have default values in functions. And so I will say, well, what’s the value of this variable that didn’t get specified when it called this function? And, and then points out, oh, Python has this ability to have default values.
[15:05] Kristin: OK. So you’re doing a mix of targeting a particular learning objective as well as thinking about common ways students misunderstand that concept and making sure that they kind of, if they have that understanding, they’d like pass right through and be fine. But if they have one of the misconceptions, they also then would hit a particular roadblock, and it would show you that they have that misunderstanding.
[15:27] Barb: Right. So I’ll ask them to predict what’s code gonna do below where I don’t call the function, I define it, and I don’t call it because that’s a common problem is they’ll, they’ll define a function, and they’ll never call it. And the answer is nothing’s gonna happen, you know? But I’ll ask them to predict that. Right. What do you, what do you have? I can also though see that they can’t just run the code and see what happens, and then they could answer the question, but we can see that in the log, you know, did they do it the way we asked? Did they predict or not?
[15:57] Kristin: Do you actually audit and check?
[15:59] Barb: I haven’t, as we’re running the course, but it is a fun thing to go do in the log files afterward to go see what actually is happening.
[16:07] Kristin: And this is probably all in Runestone because you’re using a lot of Runestone stuff.
[16:09] Barb: Yes
[16:10] Kristin: Ok. Yeah, there’s definitely moments where I’m like, I could audit that to find out how much students are actually doing what I’m telling them to do, but I don’t have that kind of time right now.
[16:20] Barb: Well, so that’s how I do the pre-readings, you know, Runestone, we do autograde those, and it is based on them doing it before lecture, which they’re supposed to do. They’re supposed to do pre-readings before lecture, and we just autograde it. And so we’ll know, well, ok, this many people got points, so this many people didn’t. So it’s a quick way to find out. Are people doing it?
[16:39] Kristin: Yeah, that part, like that part. I do, too. Like I have, I have classes flip, so it’s like they watch videos, and then they watch reading quizzes. I call them Prepare quizzes, and then I actually look at the results. And I tell them, like, the reason this is due 24 hours before class, like two days before class, is so that I can look at it the day before and figure out what to do in class. And so that’s, that’s like straightforward to, to like, you know, check the statistics. I was more wondering about, like, during class with this, when you ask the students to predict the code, but they have the code in front of them, like do they answer the question or do they actually predict it, or do they just run it? And then they answer, that would be funny to find out.
[17:15] Barb: We did look at this for some other things that we were doing, you know, do people predict code before they run it? And we found that it varied based on how hard the problem was. So way more people cheated by running the code first before they predicted if it was a harder question.
[17:31] Kristin: Okay, I wonder, I wonder what that means. Like, is it, is it bad that they did it? Is it OK? Because they can at least like metacognitively assess that they’re like, I’m not sure what this is, so I probably should just run it rather than guess.
[17:49] Barb: Well. Right. And what I really care about is, did they try to think about it first? Right. You know, and partially in POGOL, I have them work in pairs at least, in these large lecture halls, whatever. So that, hopefully, they discussed it and maybe they’re not 100% sure of their answer. So they go ahead and run it to be sure before they answer. I don’t know why because they can answer as many times as they want. Doesn’t matter.
[18:08] Kristin: Students hate being wrong with a firey passion.
[18:10] Barb: They hate being wrong. So, but at least if they then discuss it right, because the research shows that if you don’t make a prediction, you’re going to remember it wrong. So that’s why we’re trying to get them to make a prediction and at least at least verbally, if not in the book.
[18:32] Barb: I also use peer instruction. We’ve been adding support for peer instruction to Rpunestone so they can vote right in the ebook. And so we can, we show them hard multiple-choice questions. They vote individually in the book, and then they can either discuss with people around them or we also added a text chat. So we could, we could maximize the number of groups that had people with different answers. And we’ve actually found that that gives us better learning gains, right? Because they tend to sit with people that are like them and they don’t discuss it very much. Whereas if we matched them up with people who have different answers, well, now they have to discuss it.
[18:59] Kristin: Any other experiments with Parson’s Problems?
[19:01] Barb: Oh, we’ve done lots. But so have other people. One of the nice things, too, is the study that I did to show that they were, people are usually significantly faster than writing the equivalent code, but with no significant difference in learning gains. There was another study in Snap. So, using a block-based language even that found similar things. So, that was nice. There’s also evidence that they can be used to help people learn common algorithms, and also when people use them on summative assessments, they correlate with scores on other write-code problems, and they’re more sensitive to learning. People can do better on those than they can in write-code problems. You know, in general, I find a lot in my exams that students, if students are really lost, they’ll still mess up Parson’s Problems. But they’ll do, they’ll get more points on Parson’s Problems than they will on write-code. Because if I do auto unit testing and they just, you know, get zero points. That’s not, that’s not great. Right.
[19:56] Kristin: I, if I recall correctly this, this is reminding me vaguely of, I didn’t do Parson’s Problems when I was a grad TA, but the professor that I worked for, he did skeleton code, like, fill in the blank. Right. And some of those problems were really hard in my opinion because it wasn’t how I would write it.
[20:18] Barb: Yes, and this is one of the things we found, right, that students really don’t like Parson’s Problems if it doesn’t use the solution that they have in mind, which is why we’ve started using generative AI or large language models to generate personalized Parson’s Problems. So if we’ve asked you to write code and you’re struggling, instead of just popping up a canned Parson’s Problem that we created that may or may not match your approach. We are now using generative A I to create a personalized Parson’s Problem. What’s the closest correct solution to your incorrect solution and serve you that.
[20:47] Kristin: So, how, how good is it?
[20:50] Barb: Pretty good so far?
[20:52] Kristin: OK. Or like, has it ever led a student to like super astray or anything?
[20:57] Barb: No, so what we are careful to do because we have an answer, right? We know what the answer is, and we have unit tests, is we test the code that it generates, and if it doesn’t pass the unit test, we don’t serve that to a student. We always as a fall back, we can always go back to our one solution that’s based on the most common student solution. So we can always fall back to that. So you’re not going to get a wrong answer from what we’ve done. So we’re sort of guard railing.
[21:21] Kristin: Which, which makes sense like CS1, especially like my understanding of how large language model works is pretty good at solving all of those CS1 level problems. And, and, and since I’m mainly teaching data science at this point like the code for the data science is easy. It’s, it’s like, do you understand what the code did? That’s really hard. And ChatGPT sometimes gives very bad answers to that part. But yeah. So, do you want to say anything else about large language models, or do we want to, like, put that aside now and pretend it doesn’t exist?
[21:55] Barb: We could, well, we’re still playing with those, a couple of other interesting things we found with Parson’s Problems is, is one right? If, if your solution is different, then then it also increases cognitive load because that’s not how I want to solve it. It’s one of the reasons people want to be able to solve it themselves rather than solve a Parson’s Problem. But the other problem is people, one of the things I did is make my Parson’s Problems adaptive because some people struggle even with a Parson’s Problem to put it in the right order. And so if you are struggling while you’re solving a Parson’s Problem, I have a little help button that after you make at least three incorrect solution checks to see, OK, is this correct? No, it’s wrong. It’s wrong. It’s wrong. It says, hey, the help is available. If you click on the help button and you used any distractor blocks or incorrect blocks, those animate going back to the source. So usually, there’s a source on the left with all the mixed-up blocks, and you’re constructing a solution on the right. And so those animate going back, and then they gray out. And if they were paired originally with the correct code, they repair with the correct code, which is a big hint. Hey, don’t use that. Use that.
[23:00] Barb: And the other thing we do is if you don’t have any distractors and you ask for help, it starts combining two blocks into one, starting with blocks that are the furthest apart. And we keep combining blocks if you keep asking for help until we get down to three blocks. So we have found that that helps people get to a correct solution. However, there are times when people get to a correct solution but still don’t understand the answer. They don’t understand why the distractor was wrong. Why is this answer? Right. So another thing we’re doing with large language models is trying to generate explanations of the code so that people can better understand even if they got to a correct answer with the Parson’s Problem, but they don’t understand it that they can learn from the explanation, but we don’t have to generate all ourselves.
[23:43] Kristin: So, have you done the explanations in the wild yet with the students?
[23:46] Barb: No, we have, this is just, we’re starting this week. I, actually, years ago as part of the free and interactive ebooks, we made an audio tour of code, and I recorded audios so it would play and you could have different kinds of tours. There could be a line-by-line tour or major structures tour and it would highlight lines of code as it talks about them. And I recorded all these, we put them in an ebook and we found almost nobody used them. So, I went, well, that was a lot of work and not great and it was a lot of work for me. So, I didn’t do that. Though, we did get feedback because we were creating books for teachers to try to get teachers to teach CS principals. And the teachers told us they found them valuable as modeling how to talk about code. Like how do I read aloud to my students? And it was like, great, but not many people listen to them. So that’s a lot of effort for a few people who want to listen to them.
[24:41] Barb: And we were trying to use audio because of the multiple modality effect in cognitive flow that, you know, you can listen to things and hear things at the same time, which is what we’re doing now. But you know, but unfortunately, a lot of people don’t want to listen to audio while they’re doing it. And there has been work, other people have done work with Parson’s Problems where they have explanations that are in text, but that can be, you know, text code plus text explanations can be a little bit of a split attention effect. I’ve got to read this and read this and.
[25:12] Kristin: Yeah, you’re going back and forth trying to remember what you read to go back to the code and remember what that is.
[25:16] Barb: Right. But we are now interested in using LLMs to generate explanations and see how well at, at different levels. Do a summary explanation, a high-level explanation, do a little bit more detailed explanation, explain in you know, a micro-level explanation. What is each thing doing in a line or a statement. So that is something one of my Ph.D. students Xinying wants to do
[25:40] Kristin: I, I don’t, I don’t know how well LLMs will be able to explain code in plain English. Like.
[25:49] Barb: Oh no, they’re pretty good at it. There’s been some papers on that already, especially the kind of things we do in CS1, you know, small things
[25:56] Kristin: Because part of me was wondering, is it, is it like the novice CS1 student who’s like line one does this, line two does this?
[26:04] Barb: They can do the line-by-line very well, right? So the line by line works pretty well, but you can ask for different levels of explanation. And there’s, there’s already been some papers, Paul Denny and crew have been publishing a bunch of papers on LLMs and they, they have one about explanations, and they compared expert generator human-generated to LLM and most people preferred the LLM.
[26:24] Kristin: Oh, interesting. I wonder why did they think it was more clear or…
[26:29] Barb: I don’t know, I’d have to go back and look.
[26:32] Kristin: So we, we’ve talked a lot about Parson’s Problems. We know that it’s in Runestone, you use it in Runestone, and everything. But is there a very easy way to start using Parson’s Problems in your classroom to see if they work for your particular either style of teaching or your cohort of students? Like what’s a low-barrier way to start using them?
[26:52] Barb: Right. So, the easiest thing is to cut up paper, make strips of blocks, and have people put those in order, right? Ask, ask people to do that, that’s a very simple way to get started. I people when people did exams, use them on exams, they often did this, they would, they would give numbered blocks basically on the page and then tell people just to write the block numbers. But that’s too much cognitive load for students. I always say if you’re going to do that, give them room to actually write all the code down, not just the block number, or I tell them to write as much as they have to in order to know what line they’re talking about because it’s hard to keep it all in your head if you don’t see it. So, you can do paper-based, Parson’s Problems as formative or summative assessments, that’s an easy way to get started, and easier for students to answer questions on summative assessments or exams, easier to grade too, more consistent grading and easier to grade.
[26:52] Kristin: Though part of me wonders about that one because, like, it’s basically ordering blocks, right? So, what does it mean if a student swapped two blocks? Like it probably par partially depends on context where like some swapping of blocks doesn’t matter, but some swapping of blocks is vital, but it’s a spectrum. And then, like, how do you grade that?
[28:09] Barb: Well, early on I used to worry about, ok, when I’m auto grading Parson’s Problems for exams, how do I auto grade them? And I started with, usually, people mess up in the middle, you know, they get the top right. Maybe they get the bottom right. But they mess up in the middle. So I started doing longest common subs string, you know, between this and the correct, you know, and all this other stuff. And eventually, I just said, you know, what really just given the points for things in the right place with the right indentation and, you know, they don’t usually mess up so badly that, that’s gonna hurt them.
[28:40] Kristin: So, it’s, it’s kind of like, did you check all the right boxes in essence?
[28:43] Barb: Yeah. So, just a very simple grading works. But early on, I did more. Ok, start at the top. How many things did they get right here? Start at the bottom, go up, you know, and then do sort of like, ok in the middle, and if they used a distractor, well, that’s only half a point off because they use the distractor rather than the correct. So I had a much more complex grading scheme, but I found it really doesn’t impact a whole lot.
[29:04] Kristin: Okay, even if they’re like, what if they’re off by one? Like, for some reason they put one of them in the wrong spot or never used it?
[29:12] Barb: Yeah. No, I, I just had that on an exam and a student said to me, well, what they did is they put something above the loop that should have been in the loop.
[29:19] Kristin: Ok. That’s kind of important, though.
[29:21] Barb: It’s kind of important. And that’s kind of what I said ok, if you’d looked at your order, you would have said, wait, I isn’t defined, how can I use I, you know, so that doesn’t make any sense.
[29:33] Kristin: I was more thinking of, like, what if a student, like, just forgets to initialize a variable and so everything is correct, but off by one because everything is shifted up by one.
[29:43] Barb: Well, that’s originally why I tested top and bottom because usually they mess up in the middle, they don’t mess up at the top or the bottom and, you know, I took the max of either, you know, top until I found the wrong thing or bottom up until I found the wrong thing.
[29:57] Kristin: Yeah. My, my, my, my immediate thought after I said it though, was that a student would realize that they have an empty line. Like a student would notice, like, oh, I don’t, I, I’m supposed to use ten lines, and I’ve written in nine, I must be missing something, and maybe that’s when they would course correct or something.
[30:11] Barb: Sure. Now, I, we use them on paper originally when I was doing them and, yeah, they, they were easy to grade and people didn’t really complain about the grades.
[30:22] Kristin: Yeah. Well, well, maybe, maybe there’ll be a resurgence of Parson’s Problems on paper exams because that’s where we might be heading. Given LLMs.
[30:30] Barb: Right, and there has been research that Parsons don’t, and that LLMs don’t solve Parson’s Problem very well yet. So.
[30:36] Kristin: Oh, interesting.
30:38] Barb: So, if you give them Parson’s Problems, you may have better luck on exams with them actually trying to figure things out.
[30:44] Kristin: Interesting that not, I wonder why is it just like it can’t handle.
[30:50] Barb: Well, I mean, for one thing, they’re blocks. Right? So, they’re visual, and a lot of don’t handle. But you, you could, you could give them a mix, you know, they could retype it. It would just take a lot of time. Right. They could give them the prompt and see if, you know, and then, but then it might generate different code and then they have to map to the Parson’s. So, I think it’s, it’s one way to sort of LLM proof your or, or at least make it harder to use LLMs on your exams.
[31:12] Kristin: Yeah, it’s LLMs make, make everything more difficult. Like, the security model has changed in essence and you have to now think through, like, do I care if the students are learning in LLM? Am I going to basically force them to own that responsibility of whether or not they use it? And like, yeah, it kind of depends, like if it’s on a formative assessment, I’m like, well, it’s kind of on them. If it’s on a summative assessment, then I’m like, ok, now I have to think about the security model here because I want to make sure the summative assessments especially actually measure what I think they’re measuring because like, the formative of assessment, like that’s on them to a certain extent, to my opinion because I’m like, this was, this formative assessment was for your benefit. Not really mine. It was for yours. So if you want to fool yourself into thinking you know, what, know it rather than don’t know it. Then I’m like, ok, you’ll find out later.
[32:03] Barb: This is what I tell my students because I’ve always had this policy that they could work together as long as they both. I said together doesn’t mean, hey, you finished your work, and you just send it to me, right. Together means sit down together. Let’s figure it out because they were doing it anyway. They would, they would do that in front of me. And I’m like, I’m not going to call you out for doing that all the time anyway. Like you’re gonna do it. So let’s just do it. Let’s just make that the policy: you can work together on anything other than exams as long as you’re working together, not just taking somebody’s work. But with LLMs, I’m like, you know, it’s the same sort of thing. You can work with an LLM. Just say what you did with it and how you used it and it’s still on you, you own, you, you got to turn things in, you’re getting graded at not the LLM. So you gotta check that. It’s correct and, and use it. I say use it wisely to help you learn. So if you’re stuck, you can’t go to office hours, you know, use it to tell you what, how do I fix this to get unstuck? Don’t have, it just don’t just take a prompt and generate the LLM and dump it in and never look at it. Right. That’s your mind is never engaged. You’ve learned nothing.
[33:02] Kristin: Yeah, I think the thing that, that LLMs make me maybe most worried about is that students, novices do not know when, what they don’t know because by definition, they’re a novice and therefore they can’t, don’t even have the skill set to recognize what they’re missing. And LLMs short circuit that even further because it’s, it’s the like I can read it and understand it. Therefore, I must actually understand it and it’s, it’s never that case.
[33:34] Barb: Yeah, and I find my students still struggle with writing code, right? And they can solve Parson’s Problems, especially with the adaptive to help them get to a correct solution. But then, when they go to write code, they don’t know what to do. So we’re actually exploring, can we can we do can we break a problem into subgoals? A lot of them have trouble translating from English to code. How do I get from English to code? And, of course, we’ve given them work examples that do that, but still getting them developing that skill is complicated. So one of my Ph.D. students, Zihan is interested in, can we, can we scaffold the whole thing with more adaptable parsons so that OK, given this, this description, do you want to just write the code yourself? Great. Go do it. Do you want help with understanding how to go for the English description to subgoals? What are the goals or subgoals that you need, and then serve that as a Parson’s Problem? And then, OK, now do you know how to take this one subgoal and make code out of that if not? Oh, do you want a Parson’s Problem for that?
[34:35] Barb: You know, so giving them a lot more very individualized scaffolding and but, but putting them in control, but you know, one of the things we want to check is, you know, I told you, we made these toggle problems because sometimes, some students told me, oh, I don’t want to solve the mix of code problem. I want to write, write the code. So I gave them that option. And about a third of students when I, when I surveyed them, you know, end of course, student survey said that they wanted that ability to switch and write the code themselves. One of the things we want to do is go back and look at the logs. Ok. Did you do that? I’ve now given you lots of, lots of that ability. Did you do it at all? I’m, I’m betting no that they’re, they’re like, oh no, the Parsons are easier and quicker. I’ll just solve that.
[35:14] Kristin: Yeah, I, I would love to know that one, and I bet you it’s highly correlated with prior coding experience.
[35:22] Barb: Yes. That’s typically what we find.
[35:41] Kristin: Yeah. Rolling back a little bit to what you said. So you said, does the student want help on subgoal, figuring out the subgoals? And then you said so there are Parson’s Problems that are subgoals, is that what you said?
[35:45] Barb: We give them a Parson’s Problem that is subgoal labels that they have to put the subgoals in order. How what are the subgoals to put this to solve this problem?
[35:48] Kristin: And so the subgoal is basically like a word explanation.
[35:52] Brab: Yeah.
[35:53] Kristin: And do you do anything with helping students label chunks of code from Parson’s Problems as a subgoal, too?
[36:01] Barb: I haven’t done a lot of that but you know, the work with Briana Morrison and Lauren Margulieux, I, I would like to do that. I, I started doing a little bit of it. We did, we have an ITiCSE working group on Parson’s Problems, and we did a 2022 working group. And we piloted some studies, and there was a group in Finland that wanted a 20-minute study, a Parson’s Problems when you’re just learning, starting Python. I was like, what the heck am I supposed to do in 20 minutes with when you’re just learning Python? And I thought swap, you know, we know that people have trouble with swap. I’ve put Parson’s Problems with swap in books. People struggle to get that right. It’s very simple code, right? Just temp equals X, X equals Y, Y equals temp. But they all wanted to use X equals Y, Y equals X. And so we did this in this working group, the 2023 working group. We had some people try this and fascinating, because we thought, let’s see if the Parson’s Problems would help people learn common algorithms. And we started with subgoal labels by themselves, just put that together as a Parson’s Problem, then subgoal labels with code, and then just code. And then, we asked them to write the code. And there were students who got really frustrated with Parson’s Problem because they kept trying to do just X equals Y, Y equals X. And it told them it was wrong, and they’re like, wait, but why is that right? You know, so they, they solved all the Parson’s Problem, but they were really annoyed, and then they got to the right code, and they did X equals Y, Y equals X, and it was wrong, and they were like, oh, shoot it is wrong! but they didn’t believe the Parson’s telling them it was wrong. So we’ve revised the, the study now to ask them to predict what that code’s gonna do. What do you think this is gonna print? Then show them it doesn’t work, then do the Parson’s so that they have a chance to, to address their misconception and realize, oh, I thought that would work, and it doesn’t. Now, I’m going to pay attention to maybe what the Parson’s is trying to teach me. They were all able to write the code, but they didn’t always remember the algorithm we gave them, like they might use two temps, they’d save both X and Y in a temp, which works.
[38:07] Kristin: So, I want to be careful of time. So let’s segue into too long. Didn’t listen or TL; DL, what would you say is the most important thing you would want our listeners to get out of our conversation?
[38:14] Barb: That it’s easy to get started with Parson’s Problems. They might want to try them. There’s a lot of evidence that shows that they can be helpful for people learning to code and just stay tuned because we have some exciting new things coming up.
[38:25] Kristin: Sounds exciting. All right. Thank you so much for joining us, Barb.
[38:29] Barb: All right. Thanks for having me.
[38:31] Kristin: And thanks to everyone for listening. We’d also like to thank our patrons on Patreon! They help us keep this podcast ad, commercial, and sponsor-free. And a special heart-felt thank you to Vanessa Coote, Michael Shindler, and Owen Astrachan, three of our patrons that we chose randomly to shout out today! If you are currently not a patron but like what you hear, please consider supporting us on Patreon link on our website. And if not, at least share the podcast with your colleagues and give us a five-star rating in your app of choice so the algorithms know to recommend us to more people. And remember, teaching computer science is more than just knowing computer science. And I hope you found something useful for your teaching today.