S4xE8: Multi-Part Question and Answer (Teaching Practice Byte)
Episode Summary
Dr. Luther Tychonievich from the University of Illinois Urbana-Champaign shares with us his multi-step Q&A process where he solicits questions from his students to get more diverse questions and strongly signals to them that he wants questions. Dr. Tychonievich goes into detail about how to shorten the exercise if you have less time, as well as considerations and ways to respond to the questions when an answer is not necessarily appropriate.
You can also download this episode directly.
Transcript
[00:00] Kristin: Hello and welcome to the CS ed podcast, a podcast where we talk about teaching computer science with computer science educators. I am your host, Kristin Stephens-Martinez, an Associate Professor of the Practice at Duke University. Joining me today is Dr. Luther Tychonievich, Teaching Associate Professor at the University of Illinois Urbana-Champaign. Luther, thank you so much for coming on the podcast. How about you give a brief introduction of yourself.
[00:26] Luther: Oh, yeah. Thank you for having me. I as you mentioned, I’m a teaching associate professor at Urbana-Champaign. Prior to that, I spent nine years at University of Virginia as a teaching professor, which is where I developed some of the practices we’ll be talking about today, but I greatly enjoy teaching and lots of aspects of it, classroom management and so forth, which is what we’ll talk about today.
[00:46] Kristin: Awesome. So today’s episode, as you hinted, is a TPB, teaching practice byte, in a single sentence. What would you say is your practice?
[00:55] Luther: Five-step Q and A, a way of getting more variety of questions from a larger percentage of students.
[01:01] Kristin: Yes. So when you told me about this, I loved this idea because I was like, I struggle a little bit getting my students to give me questions. Like usually, it’s a blank stare. I pause and go like thoughts, questions, concerns, and then I like, try and remind myself to count like five breaths, and still nothing. But first, let’s talk about context because I think it’s important to always talk about like, where do you most often use this technique? Because obviously, context always matters. So what is the class? How many students? Where do you use this?
[01:37] Luther: Yeah. So, I use this in many different classes. But my, I’ve used it in synchronous classes only, both in person and online, with success, the the other thing I found is you need to have enough students that some level of student anonymity might exist. So when I’ve tried this in classes of 10 to 15, 10 to 30 rather, it hasn’t worked so well. When I’ve tried it in classes of 50 plus, it’s worked quite well.
[01:59] Kristin: OK. So when you say both in-person and online is there a hybrid piece where you’re like, doing both modalities at the same time?
[02:10] Luther: Sorry. I mean, I have used it on, in, on, in-person classes and with online classes, I have not used it in a hybrid modality class, although it would probably work with some adjustments, but I haven’t tried it yet. Ok.
[02:22] Kristin: I, I am one of the people that goes extra and I do teach hybrid. So I’m always curious to know if it works in that context or well as well or if you tried it, but maybe I’ll try it this coming semester, and I can report back at some point.
[02:36] Luther: My gut is when I’ve done hybrid Q and A without this technique for a much smaller class, I had a person in person whose job is to read the chat and verbalize the questions, and I suspect that model would still work, but I haven’t tried it.
[02:49] Kristin: Yeah, I would think so too. I also usually have a TA that is the like Zoom TA and I tell the students online like this TA is your advocate in the room. Like if you need someone to raise their hand because I’m not paying attention enough like they are the ones that will raise their hand and go, “Hey, Professor KSM, there’s a question on the chat.” And I’m like, oh, yes, thank you. All right. So what is the tip itself now that we know the context
[03:16] Luther: Yeah. So, the goal is to get a lot, get questions without students self-filtering what they ask or holding back because my reply to the first question changes how they think about that question or about the questioner or about future questions. And so, I have a five-step process. So, the first step is to ask for questions on a topic and tell students to pair up and brainstorm a question, either one that you have or when you think someone else in the room might have. And then I randomly call on someone. I tell them in advance, I’m going to randomly call on some of you to share your questions. So, come up with a question to share, even if you don’t have any questions yourself.
[3:54] Kristin: Yes
[3:55] Luther: So, that’s the first step is to prep them to think they need to share a question. I’m going to call on them at random, and then I give them a minute sometimes 2. And at the end of that, I pull up a random seat generator on the screen and make it pick a random seat and ask the person in that seat for a question and write it down on something that’s projected on the screen, and then I pick another one and do it again.
[04:17] Kristin: So when you say random seat generator, do you mean like seat 520 like not five, like seat G23?
[04:25] Luther: it depends on the room. So, I whip up one of these for each class I teach. But if there are labels on the chairs, great. If there aren’t labels on the chairs, it will be some number of rows back and some number of seats in from some, from some aisle.
[04:38] Kristin: So what if no one’s sitting in that seat?
[04:41] Luther: Then I’ll just, I rarely have that in the classes I’ve done this with. But, if I do have that I’ll either just slip to the next seat over or just do it again.
[04:54] Kristin: Ok. Oh, I like, I like how you manage not having numbered seats by literally just going like row 6, 5 seats over from the left.
[05:01] Luther: Yeah. And that one is also useful if you do have empty seats because you can just skip the empties when you’re counting over. They only need to think it’s random. They don’t have to think it’s fair in any particular way.
[05:09] Kristin: True. So, what if no, like, do you ever randomly pick row one? Because usually students hate sitting in that row.
[05:18] Luther: I have not run into that case, but again, like, if, if it doesn’t work, you just do it again.
[05:23] Kristin: Yes, that’s true.
[05:25] Luther: And the other is, you don’t have to have your random seat generator generate all seats. They just have to believe it’s random. Right. The, the goal isn’t to get every student. The goal is to communicate to them that I want a question from every student.
[05:36] Kristin: Ok. All right. So you’ve picked random seats, you’ve solicited questions from the students from in those seats. Now, what happens?
[05:43] Luther: And when they give them to me, I write them down on a blank text file. I have projected on the screen. I don’t make any comment about the question at all. I just write it down, and after I have two or three picked randomly, I just say, does anyone else have more questions? And I just keep writing down questions in that same document again without any comment, except perhaps I didn’t quite hear you. Can you repeat? And I keep going until uncomfortably long silence follows the last question. I might occasionally, in some of the earlier silences say, “any more questions?” To prompt them that that’s what the silence is for.
[06:16] Kristin: So how many questions do you often end up with then?
[06:24] Luther: 20 is not uncommon.
[06:27] Kristin: Really? So I guess like my, my, my other thought then is how, is it a very broad topic that they’re an they’re writing questions for them? Because like, I can’t imagine coming up with, having 20 questions come up when we’re talking about like, how does the neuron in a neural network work?
[06:45] Luther: Yes. That’s a good point. So I do this the first day of class where I’ll do a brief summary of the class and the syllabus and me, and then I’ll say any questions about the class syllabus, me, or anything else. And we’ll do it a big one. Then they will often take up half of that first class or more.
[6:56] Kristin: Ah Ok.
[6:57] Luther: And that sort of gets them in the, in the mood and then at sort of the end of topics or transitions, I will do it again. And that establishes this pattern. And then when I have a smaller thing, like how does a neuron in a neural network work? I don’t need to go through the whole thing. I will still say any questions about neurons and then write them down the text file and they already know the structure, and I might get three or five questions there. But I’ll still have them in the same process.
[07:30] Kristin: Awesome. OK. And this is typically how many students again? Like it’s, you say this works over 50 usually?
[07:36] Luther: The smaller class I’ve had to work effectively was, was around 55, 60.
[07:42] Kristin: And so the 55 students are still coming up with 20 questions?
[07:45] Luther: Yeah. I usually use it in classes of 200 to 300, but I get about the same number of questions.
[07:53] Kristin: Interesting so like that, it’s not a 1 to 1 linear scale. It’s kind of the same. Which is, that’s nice to know. I think, I would hope that that’s how that would happen.
[08:02] Luther: I hope the questions are the variable thing, not the students.
[08:05] Kristin: Yes. All right. So you have all the questions in the doc. Now, what happens?
[08:10] Luther: Right. So once I have them on the doc, I then reorder the questions with the class watching. I just move the lines of the file. Grouping by topic and sorting in whatever order makes sense to me.
[08:20] Kristin: OK. And then what happens after that?
[08:22] Luther: And then I work through that reordered list, answering every single question. Just I work from the top to the bottom, answering every question.
[08:30] Kristin: OK. Do you write notes under like the answer under each question or anything like that?
[08:34] Luther: Sometimes, I’m not consistent on that. If I feel like I think it’s something they want to refer to later, I will. But usually, if I feel like I need to write an answer, I also feel like it’s something I should have added to some other component of course material.
[08:53] Kristin: Yeah. And, and also I would imagine that sometimes the answer is quite long and typing it while you’re saying it does not necessarily make sense. Do you put those questions anywhere for future reference, for students or yourself?
[09:04] Luther: They become part of the, the material from the day like like slides, like code we create, like the recording of class, they just are available.
[09:16] Kristin: OK. So like I, I guess for me, I was imagining that one way I would do this, I, I have a class forum post per module, and like we are, we have one module per week. So I like, I could imagine taking that long thing and sticking it as a comment on that post so the students could easily find it in reference. Is that do you think a reasonable thing to do or do you think it makes more sense to put it in like the box folder where I put the PowerPoint slides?
[09:44] Luther: I treat it more like the second. I don’t see this necessarily as a reference. I see this as a component of eliciting conversation and discussion. And so, in that context, it feels more like a class activity to me than like a reference material, a summary of the topic.
[10:04] Kristin: OK. So, I think in my context, I would be tempted actually to put it as a comment in the class forum post for that week or for that module. What are your thoughts? What’s your gut reaction to that?
[10:16] Luther: I suppose that could work. I have not focused on asynchronous forums as a medium for communication and clarification. So, I couldn’t really comment on how well it would work.
[10:27] Kristin: Yeah. I think you’ve hinted a little bit at, like, you’ve put a lot of thought into this five-step process.
[10:32] Luther: Oh indeed.
[10:33] Kristin: Could you give us a bit more like method behind the madness, for lack of a better way of saying it?
[10:39] Luther: Yeah. Absolutely. Right. So, what do I want? I want to communicate to students that I actually want questions and that part of their learning should be figuring out what questions they have. This is a goal I have. And so, to start off by this sort of forced random calling, you will give me a question, but it doesn’t have to be your question as a way of forcibly communicating that priority.
[11:04] Kristin: Yes. Okay.
[11:05] Luther: But then I also want to communicate that the person isn’t the interest who asked the question isn’t the interesting thing about the question? And so I have this process of creating separation both in time and in order between the question asker and the question answer. And I want to communicate that I genuinely want to answer every question they have. So, I mentioned I answer every question, and this may not seem odd until you try doing it. There are odd questions you will receive. And because of the way things come, sometimes you get several very similar questions back to back, and I answer all of them. Right? If, if they ask if one of the questions was, what’s your favorite kind of cake? I will pause and answer, what is my favorite kind of cake between answering two questions about whatever the technical topic was.
[11:58] Kristin: I’m wondering now the, what rules, for lack of a better way of putting it? Do you do with the reordering of the questions then? Because like if I had three questions that are super similar, even if they’re like sparsely all over the place in the doc, I would rearrange it so that they’re right next to each other, then kind of answer all three at once.
[12:15] Luther: That is how I do it. So, I group similar questions. If they’re so similar I can say these three are the same, I will do that. But often, they’re worded slightly differently. And so I’ll answer the first one, and then I’ll read the second one and say, I think I’ve answered this in the previous question. But I’ll still, I won’t skip anything.
[12:30] Kristin: Yes.
[12:31] Luther: And then the ones that I don’t group by topic are the ones that are completely off-topic. I will intersperse them because having a chunk at the end where I answer five questions about my personal life doesn’t lead to an interesting Q and A.
[12:43] Kristin: I was going to wonder about that? Like, did you put all the random ones at the end? But I think you’re right. I think interspersing might be better just because the grouping of, like on-topic, off-topic for a second is almost like a border between the question groups.
[12:57] Luther: And the number of off-topic is something I’ve noticed varies dramatically from semester to semester, class to class. So sometimes you have a class where we get lots of those every time some of the class will never ask a single off-topic question the entire semester that I, I can’t predict that.
[13:11] Kristin: Have you ever been asked a question that you kind of don’t want to answer or you want to refuse to answer?
[13:17] Luther: Yes, frequently. There’s broadly speaking, two kinds of these. One is questions that I have asked them as an assignment, asked back to me. And my answer to that is, I think this is one of the quiz questions. Maybe we can revisit this after the quiz is due. And, and, but if it’s inappropriate in some fashion, I have no problem meta-commenting on the question and explaining why I feel it’s inappropriate. That doesn’t happen very often. But if it does, I feel it’s worth having that full conversation because I don’t want to just act like I didn’t hear it. I don’t want to blame the person right in the moment, but I also don’t want to, and I, so we need to have a conversation about why this is not an appropriate kind of question to be asking.
[14:01] Kristin: Can you give an example?
[14:03] Luther: I once had a question about my personal personal sexual sexual life.
[14:07] Kristin: OK. That I feel like that. That was a bold move by that student.
[14:14] Luther: Yes, I’ve only had that something as bold as that once. I there are very few, but it does happen, right? When you open it up to anything, when you establish a precedent to answer anything, you might get asked anything.
[14:25] Kristin: I see. Ok, like my, my brain went to like if someone asked me like, what are your kids names? I’ll be like, no, I’m not going to tell you that.
[14:32] Luther: Yeah, personal information. I had somebody once asked me my home address, and I just said I live near enough to campus to walk and that was my entire answer. And I think this brings up an answer to a question doesn’t have to be what the student was hoping you would say. It just needs to engage in the question. Right. And so it can be, that’s a great question. Here’s a resource. Why didn’t you study it? And maybe we’ll revisit this. It can be, that’s on a quiz. It can be. Here’s why this is not the kind of question you should be asking, right? All of those are responding to the question and all communicate to the students that they can ask things and that they don’t have to filter. I will do the filtering. It’s OK, they can ask five versions of the same question. They can ask things they’re not sure are OK to ask. I’ll handle them all without blaming anybody and, and it’s a safe space to ask more questions.
[15:19] Kristin: So it sounds like step five is not actually answer every question. It is respond to every question.
[15:26] Luther: That’s fair.
[15:27] Kristin: So what happens if you run out of time? Like, you didn’t actually expect it to take that long?
[15:32] Luther: I think I’ve only had that happen once, and when we, that happened, I just said we’ll resume this next class.
[15:38] Kristin: Ok. So, it’s not that big of a deal going, like, oh, we’ll pick this up next class.
[15:43] Luther: Yeah, I, I also once had a, a single question that I tried answering a couple of times, and they didn’t understand and, and so I said, I’ll try to come up with a better way of explaining this, and I’ll post something because it was a sort of a pre-exam review and I didn’t want to say after the exam.
[15:57] Kristin: So how do you gauge understanding of, like obviously the, the more content-focused questions rather than the personal ones? Because you don’t want to single out the asker, right?
[16:06] Luther: Right. Right. So, I generally don’t gauge correct understanding. I just gauge self-perception of understanding, which can usually be gained from facial expressions, nodding, and so forth and sometimes from, did that answer your question? I find that a very useful response. If I’m not sure my answer worked, I’ll ask it. I say, I’m not sure who answered this. But did that answer the question? And if there’s a lot of blank stares or uncomfortable faces, I’ll say, ok, maybe it didn’t, and I’ll try again.
[16:34] Kristin: Ok, all right. So let’s go to the next section of this TPB, which is, where else could it work and where else would it not work? Because I think it’s important to talk about both parts of that.
[16:49] Luther: Yah. So when I have less time, if I want to do a brief thing on a single topic, I focus primarily just on the last three steps. The, make a list of questions, reorder it, and respond to them.
[16:59] Kristin: Ok.
[17:00] Luther: The first two, the pair up and think about a question and cold call some people to start the list adds a significant time. And I primarily only use that when I want to communicate the desire for questions when I have a large amount of time.
[17:12] Kristin: Got it. So the first day of class makes sense because that’s, that’s syllabus day anyway. So, might as well.
[17:18] Luther: And then sort of when we’re moving from one major topic to another is another place I’ll typically do that. But then other times I’ll just, I’ll just, if I think I’m going to get more than one question, I’ll just start writing them down.
[17:28] Kristin: Got it. What classes do you normally teach that you’re using this in? I just realized, I’m not sure. I know. I know the answer to that question.
[17:35] Luther: I’ve taught a wide variety of classes. I have used this in introductory CS1 class. I’ve used it in a software development methods. I’ve used it in computer architecture, computer systems, computer graphics. I tried using it in our TA training course, but we didn’t have enough people for it to work out well. What else have I used it? I used it in discrete mathematics. I use it everywhere I teach.
[18:00] Kristin: OK. So it’s definitely more the size of the class that is a, “this might not work here” than the content of the class.
[18:07] Luther: Yes. That has been my experience.
[18:08] Kristin: Awesome. Any other thoughts of where it would work or where it would not work?
[18:12] Luther: So, I just mentioned when I tried it in my TA training course where we had under 30 people, it just didn’t work. There were, there were two few questions to make the list feel like a list to make it worth reordering. The sense of anonymity never arose. It just didn’t translate well in that smaller class setting.
[18:27] Kristin: Though. I think in a TA training class, the, the self-selection bias is strong where the students probably are more willing to ask questions in a class like that.
[18:38] Luther: That’s fair. And, and I’ve occasionally had other small sections, and I have not had trouble getting less formal questions when I have a fewer number of students. I’m not saying that will always be the case. I have a few enough experiences with small classes. I’m not sure I am trusted as a source on what you should do in them.
[18:54] Kristin: Do you have any future plans for this practice at all? It seems pretty like you’ve refined it already, but any other thoughts on it?
19:00] Luther: Yeah, I mean, I’ve used it basically without changes for the past nine years. I use it in every synchronous class I teach. I don’t have immediate plans to change it, but I plan to continue using it.
[19:12] Kristin: Do you think you’ll ever do some like evaluation or research or an experience report on it?
[19:17] Luther: I could, it’s not immediately clear to me how I, what I would compare it to how I would measure it. Mhm. And since I’m primarily focus, I’m a teaching professor, I’m primarily focused on teaching. I’m not as focused on measuring and, and publishing. So, yeah, I’m, I’m happy to have this opportunity to share it with a broader community and would be happy to work with others if they have ideas about how we would measure it and turn it into research papers.
[19:39] Kristin: I think that this, in some ways, sounds like a nice experience report where there’s no data in it. It’s just like this is what I’ve done, and the the data might be things like this is how long it takes me. And here is a like-
[19:55] Luther: If you were to do this, what you should expect.
[19:57] Kristin: Yeah, like if you’re going to do this, this is what you expect. This is how many questions are typically happening. This is the size of the class of the students. And like as you said already, if it’s less than 30 it doesn’t really, really work. And if it’s over 50 you can be reasonably confident it’ll work. And like those technical details, you can totally fill a six-page paper on just that.
[20:17] Luther: That’s fair. Maybe I’ll consider that for the future.
[20:19] Kristin: Yeah, maybe this podcast episode will inspire you. If enough people are like, I liked your episode. Can you please tell me more, write it down so I can read it instead of read the transcript? So let’s go and move on to the very end, which is Too Long, Didn’t Listen. TL; DL. Can you give me a rundown of your practice in three minutes or less?
[20:40] Luther: Yes. Number one, ask students for questions and communicate a real desire and confidence in their ability to pose questions by first giving them time to brainstorm their questions in pairs and second, randomly calling on a few people to share questions, whether it’s one they have or when they think others might have. Also, create a separation between the question and the questioner and prevent earlier questions from getting more response time than later questions by step three of this five-step process would be to list all questions without comment on a visible doc, then reorder those questions, both clustering and priority, sorting them and answer each one in the new order. Or respond to each one. I suppose it doesn’t need to be an answer. I use this on the first day of class to help set the mood for how I want questions, that I want questions, that I want questions regardless of whether their question or somebody else’s. And I use it at the end of each major module or topic as well as any time, I feel like I’ve lost my students, but I’m not sure why. And then the the pair brainstorming and the random cold calls, those first two steps have value, but they can be skipped leaving just the listing of the questions, the reordering, and the answering if time is pressing.
[21:49] Kristin: Awesome. And if you’re getting random questions, you use them as borders between question groups. I like that a lot. So I, I’ll throw that in as the end of the TLDL. All right. Well, thank you so much for joining us Luther.
[22:01] Luther: Thank you so much for having me.
[22:03] Kristin: And thank you for listening. Do me a favor. If you like what you hear, share this episode with someone you know. It’s one of the best ways to help this podcast grow. Another way to support us is on Patreon. Our patrons help keep this podcast ad-free and pay for production. So, I’d like to give a special thank you to Michael Shindler and Yesenia Velasco, two of our long-time supporters. And with that ask and thank you done. This was the CS-Ed Podcast hosted by me, Kristin Stephens-Martinez, and produced by Chris Martinez. And remember teaching computer science is more than just knowing computer science. And I hope you found something useful for your teaching today.