S4xE15: Literature Mapping with Undergraduates (Teaching Practice Byte)
Episode Summary
In this teaching practice byte (TPB), Dr. Brian Harrington discusses his SIGCSE Technical Symposium 2025 paper on Literature Mapping, a scaffolded, scalable, low-overhead way to introduce undergraduate students to research and bootstrap a student research group. We discuss how literature mapping helps students practice reading many papers in progressively more depth. His process assigns each paper to two different students, builds in flexibility for students who leave partway through, and culminates in a publishable artifact that students can be proud of. Moreover, he has found this helps him build a community that goes beyond the students graduating. Dr. Harrington has packaged up his process into a GitHub Repository and would love for anyone to adapt what he does to their own school.
You can also download this episode directly.
Episode Notes
Brian Harrington, Aditya Kulkarni, Rohita Nalluri, Anagha Vadarevu, and Angela Zavaleta Bernuy. 2025. Literature Mapping: A Scaffolded, Scalable, Low-Overhead Undergraduate Research Experience. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSETS 2025). Association for Computing Machinery, New York, NY, USA, 464–470. https://doi.org/10.1145/3641554.3701938
Brian Harrington’s GitHub Repository
David M. Torres-Mendoza, Saba Kheirinejad, Mustafa Ajmal, Ashwin Chembu, Dustin Palea, Jim Whitehead, and David T. Lee. 2024. Evaluating Exploratory Reading Groups for Supporting Undergraduate Research Pipelines in Computing. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (ICER ‘24), Vol. 1. Association for Computing Machinery, New York, NY, USA, 389–405. https://doi.org/10.1145/3632620.3671104
Transcript
[00:01] Kristin: Hello and welcome to the CS-Ed podcast, a podcast where we talk about teaching computer science with computer science educators. I am your host, Kristin Stephens-Martinez, an Associate Professor of the Practice at Duke University. Joining me today is Dr. Brian Harrington, a teaching stream professor at the University of Toronto Scarborough. Thank you so much for coming on the podcast.
[00:23] Brian: Well, thank you, Kristin. It’s great to be here.
[00:24] Kristin: So today’s episode is a TPB, teaching practice byte. In a single sentence, what would you say is your practice?
[00:32] Brian: So I’d say it’s using literature mapping as an exercise to bootstrap an undergraduate research program.
[00:39] Kristin: Awesome. So when I read this research paper at SIGCSE Technical Symposium, I was like, oh my goodness, I like this. It would be cool to have Brian on the podcast because this feels like something that would be good and a lot, and reasonably approachable. So, we’ll start with the context, and where have you currently done this with your students?
[00:59] Brian: So actually, interestingly, this is not part of a class. I purposely run this at the moment as just a totally voluntary situation. So the idea is this is really geared towards students who want to get a taste of research or want to sort of dip their toes in research in a very light way. So what I do is I run weekly reading groups where we read a paper and discuss a paper, and so students sort of earn a spot in this program just by showing up. I say, if you show up and you show me that you’re interested, at the end of the semester or the beginning of the next semester, I say, hey, do you want to be part of a literature mapping project?
[01:37] Kristin: So this is like a 15-week reading group, and then as it closes out, you kind of look to see who has been consistently coming and invite them to do the, a literature mapping at the following semester?
[01:49] Brian: Yes, so I mean, we don’t get the full, you know, however many weeks in, sometimes depending on how things go. Usually, I would say 8 to 10 weeks is sort of the norm for the reading group, cause the first couple of weeks you’re not doing stuff, usually around midterms we might skip a week. And it’s, it’s very ad hoc. It’s very much whoever’s in the room, you show up and say, if you’re there, great. If you can’t make a week, you don’t have to report not being there. But at the end of the semester, I say, OK, next semester we’re gonna do a literature mapping project, who’s interested, and I’m gonna select based on the people who I’ve seen you’ve been showing up and showing me you’re interested.
[02:31] Kristin: OK. And who picks the papers for the reading group?
[02:36] Brian: So usually we do this as a group, but honestly, we often do it a bit of a cheat. What we do is we say, I have some other project going on, and I kind of need to do this literature review. Well, if I’m gonna have to do this anyway, I might as well use this to kind of get a head start on that.
[02:58] Kristin: So, like, I’m suddenly intrigued by this because since it’s clearly and fully intrinsically motivated for the students involved, is everyone reading the same paper or is everyone reading different papers and just kind of like reporting and talking about what they read.
[03:10] Brian: Yeah, we, what we do is we pick volunteers who are interested in presenting. So you basically your first couple of times you show up and the first time we actually say, “Don’t even read the paper. I want you to come and just listen and experience.” Then afterwards, we say, here’s the paper we’re reading this week. Next week we will discuss it, and there will be usually two students assigned to have lead the discussion on the paper. So they’ve read it, they’ve synthesized it. They are sort of presenting it back to the group, and some people in the room have skimmed the paper. Some people are desperately trying to flip through it during the actual presentation, and some people have actually read it deeply enough. And then for the people who show up regularly and then show me, yeah, you, you’ve read the paper, you have interesting things to discuss. OK, well, next week or next semester, you can be one of the presenters of the paper.
[04:07] Kristin: OK. So, it’s clearly like a scaffolded process of like slowly integrating into the reading group, and as a group, they kind of decide what the paper is going to be the following week. Is there a particular theme, and what kind of papers everyone’s interested in?
[05:22] Brian: So we started off as a CS education reading group partially because that’s my wheelhouse, partially because these are papers that are usually accessible to students. But I am always say I’m open to whatever you’re interested in. I care more about the experience of doing it than the topic. But I will put some guardrails on, you know, students come to me and say, I really want to present this like deep learning neural network paper, and if I look at it, you’re like, there is way too much math here that people in the room will not even be able to begin to tackle. I’ll put the kibosh on that. But, you know, we’ve had papers on all sorts of things, just whatever sort of interesting to the student presenting, and also I think might be interesting, interested to the people in the group.
[05:10] Kristin: Got it. OK. So, we’ve covered the context as well as kind of the follow-up until the students actually get into the literature mapping. So, now let’s cover the tip itself, like how does the literature mapping work?
[05:23] Brian: OK, so the idea here is If I want to do a literature review, whether it’s a full literature review paper or just I need to do a literature review for something I’m doing, I realized there’s a lot of, I don’t want to call it busy work, but there’s a lot of relatively mundane work that just needs eyes on papers. And I realized that to seasoned academics, this is not the fun part of doing the research, but actually, to students, this is maybe a way that they can get into this area and get some access into this. So what I sort of realized is, OK, well, if we can find a way to scaffold it, it’s actually sort of a naturally scaffolded position.
[06:04] Brian: So I had a couple of things I wanted to do. So number (1) is I want to get the literature, not the full literature review, but at least the sort of mapping of what papers exist and how they relate to each other. Number (2) is I want to give that, that scaffolded instruction where students aren’t just handed a paper and say, hey, go read this and understand it all. And number (3) is, I don’t want this to be a massive time sink to me, you know, this is something where I’m doing this regularly. I’m sometimes doing it with things that I would need to do anyway, but other times I’m just going, OK, let’s do this, and I like, I’m sure a lot of your listeners do not have a ton of time to spend on this.
[06:46] Kristin: (laugh) Yes.
[06:47] Brian: So I really want this to be something where I’m not spending my days chasing students down. I want this to be sort of a great, if you do it, fantastic. If at the end of the week or the 2 weeks or whatever the time frame, you haven’t done your work, that’s OK. It just moves on to the next stage, and the ones who make it through, make it through, you know, again, totally opt in. That’s a big part of what I’m doing there.
[07:11] Brian: So the way they actually works, so, we outline in the paper 7 steps. There’s really 4 that we care about. So, the first thing is we get together in a group, and we come up with a systematic query together. This is one is a little bit more of an art than a science. We decide, you know, what are our inclusion, exclusion criteria, what do we care about? What are we looking at, what venues, you know, this is sort of something that any undergraduate student can give a basic understanding of, and we can talk about, but it gets buy-in from that.
[07:46] Brian: So then what I do is I get, OK, here’s our queries. You want a couple 100 right? You can’t have 10,000, you can’t have 7, you got to have a reasonable number. And we put them into the spreadsheets. So I built out these spreadsheets where everything is double-coded. So you get tabs. I’m not being super fancy and like hiding them from students, but basically, your name is assigned to a certain number of papers, and you have a little pull-down menu that says inclusion or exclusion based on the following criteria, right? We have our basic criteria.
[08:19] Brian: And then the script automatically goes, OK, well, these are papers everyone agreed on. These are papers where you know, we had disagreement on, and the nice thing is this usually requires, I tell students, you know, 2 to 3 minutes per paper, right? You are usually only having to read the abstract. You may occasionally have to open the paper to like check, you know, if you say, oh, it has to be at least longer than 2 pages, you may have to actually check the paper to see that. But this is very, very low level, just getting them comfortable with this idea. But it has some checks and balances, right? If a student goes, accept, accept, accept, accept, well, you know, they’re, they’re randomly assigned to be double-coded, so you catch that pretty easily. So once you do that, and that may take a couple rounds, sometimes you sort of go, oh, we’re accepting too many, or we’re excluding too many, or you had different, understandings of it.
[09:14] Brian: So you move on to the next, which is, at the, you decide what categories you care about. So, after this point, they have a vague idea of what they’re doing. So I usually say, OK, what axes are we caring about? So, how they’re being evaluated, what they’re evaluating, the population, etc. etc. And you say, OK, well, there’s a bit of a clustering exercise, right now that you’ve got a sense of what populations do we care about? Are we gonna code them on, are they looking at it by gender, by race, by previous program experience, in terms of things that are being evaluated? Are you evaluating drop rates? Are you evaluating grades, etc. right? You kind of have a bit of a clustering exercise.
[09:57] Brian: And then they have to read the papers again, usually again, randomly reassigning. And they put into the spreadsheet, and again it’s just pull-down menus, or putting X’s in boxes where you say, OK, you have to read this paper, and you have to tell me, “What is the thing that’s being evaluated?” You put an X on, if they’re looking at grades, you put an X in grades. If you’re looking at gender, you put an X in gender. If you’re looking at, you know, whatever the survey, you’re putting an X in survey. And again, now they’re having to read the papers, but in a directed way, right? You know, you’re not sort of left to the wolves. You have the paper, and you say, OK, does this binary decision you have to make that.
[10:39] Brian: And again, that’s all double-coded. So then, you have this usually does have discrepancies. You will have students where, I thought this was about gender, and I thought it wasn’t. Well, they mentioned gender, but actually, they didn’t report by gender. OK, so you have to now, you know, clarify those sorts of things, so this usually is a couple rounds of iteration.
[11:01] Brian: But the nice thing is this forces them to discuss this, you know, they assign and the spreadsheet will say, OK, you know, Alice said yes to survey, and Bob said no to survey. Now, why do you disagree with this? So you have to have a meeting and discuss that. And so there’s usually also a, a bit of a stage of, OK, we have to figure out what category, you know, if you say survey, well, we had 80% of the papers had a survey, you might have to say, OK, are you doing affective survey versus, you know, usability survey, you have to split some of those, or you say, well, there was only one paper that looked at prior programming experience. OK, let’s figure out how we merge that to another category.
[11:43] Brian: But the nice thing is they’re not having to read the full, full paper in detail, but they are gonna have to do a guided reading. And then by the end of this, you wind up with a literature map, which is, you know, it’s not something that I think people are as familiar with as a full literature review, but it’s a mapping. It’s, you know, you’ve looked at all these papers, and you’ve got usually 3 or 4 axes, and you can find the papers that you care about. So you know, the spreadsheet automatically spits out a series of PDFs that have references in them.
[12:16] Brian: So the first one we did was COVID. We looked at all the papers in CSEd published that mentioned COVID anywhere for like the 2 years following. And so if you say, OK, I want to look at the impact of online exams on mental health of female-identifying students, you can look across the row for mental health, down the row for exam and across for female. And here’s the 7 or 8 or 12 or 4 or whatever papers, they’re in there. And the other thing it does is it produces heat maps, which I really like because it’s just the counts, but you can very quickly see, oh, tons and tons of people are looking at this, but no one’s actually looked at this. That might be an interesting thing to look at. And the nice thing is these are useful. We’ve published a bunch of these usually as a poster at something like SIGCSE or ITiCSE, but it’s also produces something that the students can point to and go, I helped build that in a, you know, real way that they take ownership.
[13:19] Kristin: Awesome. I love this on so many levels cause, so for, for my research students, I teach them the 3 pass method on papers. And pass one is like, you’re kind of skimming it. Like, your goal is to extract useful information in the smallest amount of time possible. While I don’t give them this directed, like, questions that they’re answering, like, does this involve gender or not. So, I love that this gives them, gives students a much more concrete thing they’re looking for, as opposed to when I’m kind of teaching my students this like pass one method or pass two. They are often already centered around the research topic that they’re interested in, and so they have to do a deep, deeper dive. So, I guess like for my students, I would call this like this is the pass 1.5 (Kristin meant 0.5 here), like you’re scanning through to figure out like, is this a useful paper for you and then you wanna chuck them as fast as you can cause it’s a waste of your time once you realize it doesn’t include it.
[14:13] Brian: Yeah, I think that’s something I really help or try to focus the students is. A large part of reading papers is not reading the papers. It’s figuring out the bits you don’t actually need to read, and, you know, it sounds sort of a silly thing, but a lot of students when they first get involved in research, it’s so intimidating where you say, oh, you have to read these 20 papers, and they feel like, I’m, I’ve spent 3 days and I’m 3 paragraphs into the first paper, forget it, this isn’t for me, whereas by the end of this exercise, by the end of the semester, like we sort of tracked at one point and the average student had read, well, had touched on like 200 papers. They had skimmed maybe 100 papers. They had answered like specific questions on maybe 50 papers, and then in the end, they’ve got those 20 papers. So by the time they need to actually read it. They’ve actually touched that paper 3 times at sort of successively deepening levels, and it doesn’t feel like work really. It doesn’t feel like you’re having that push thrust upon you.
[15:16] Kristin: Yeah, and when I’m first work with undergraduate researchers, it’s often me busting myths like, you should not be reading every word in the order it is presented to you of this paper. And that’s why one of the first meetings with them is me sitting down for 30 minutes and I’m like, I’m gonna read this paper in front of you, and I’m literally gonna be like mean and say, all right, read the first sentence of this first paragraph in the intro, like, all right, yes, I buy that. We don’t care about the rest of this text. And like, we skip down and go, like, first sentence of every paragraph until like, OK, intro section, we’re already buy into the motivation and what they did. Let’s skip to the related work. These subsections look good. Let’s keep going. Like, look, we’ve made it through 3 pages of the paper already. And it’s so funny where I think the students are always like shocked when they’re like, wait, we, what? We spent a lot of time on the chart and the tables, and we spent almost no time reading the text. And I’m like, yes, this is what you do in a pass one of a paper. Your goal is to extract information, and like focus on where the dense parts of information are, and you will slowly realize this also feeds into their writing, where I talk a lot about topic sentences of paragraphs. And often I’m like, you wrote your, like, it’s OK that you wrote your topic sentence at the end of the paragraph because you got it out of your brain. Now, I need you to take that sentence and put it to the top, and then rework the paragraph.
[16:39] Brian: Well, the other thing I find this really helps students with is something that a lot of undergraduate students have trouble with is, I spent the semester, I read all these papers, I have them all in my brain, and now at the end of the semester, I don’t remember which paper had that idea. So, students are very, and, and probably a lot of researchers are very bad at sort of, you know, whether you’re using a tool or whether you have having some way of indexing and finding and remembering your papers, and this sort of forces you to naturally do that because you wind up with a little like you have a field that can only fit 10 words of explaining why this hits this particular category. But if you don’t fill that in, then when you’re assigned to, you know, meet up to resolve that issue, you go, oh, now I have to reread that whole section to figure that out. Well, if you’ve done that, the 30th time you’ve done that, you go, “Oh, OK, maybe I should bother filling in that field.”
[17:36] Kristin: Yes, it’s, I love this like natural forcing function to, and also scaffolding to support the student to realize and reflect on, oh, this is why I need to do this thing. So yeah, that’s why when I read this paper, it’s like, I love this, this like, so much natural consequences and forcing a reflection of them understanding why they have to do certain things, because nothing beats, like, learning how to do a thing except experience, and to feel the pain of, like, screwing it up multiple times before you’re like, “OK, now I get why I need to do this thing.” I’m like, yes. So, yes, I love it. Let’s move on to where else do you think this would work and where would it not work? Like, what are your thoughts there?
[18:22] Brian: So, I’ve always done this in a very opt-in model. That’s been part of my MO on this, that I like this idea. That’s part of why I built this up is because, you know, that you sometimes have students who halfway through, they go, “Actually this isn’t for me. I’ve changed my mind. Something else has come up.” And this is all designed in this phase model where just, you know, I can say, all right, at the end of this step or at the end of the stage. Look, you don’t, you’re not interested in continuing. OK, thank you very much. We can just reassign, you know, the spreadsheet will just automatically reassign based on the names.
[19:01] Brian: So I’ve pushed away from it, from just my own mental health and time management, but I do think this is absolutely works as a course, right? So what I’ve really done is I’ve, for the students who are engaged and into it. I’ve actually retroactively a couple of them built this into a course credit where I’ve said like, hey, you worked really hard on this, you did this, we’re gonna do a readings course together and this work you did over the past semester counts as sort of not the whole course but a chunk of the course.
[19:35] Kristin: Yeah yeah.
[19:36] Brian: And so I think this works fairly well. It would, it would work fairly well to build into a research course. I don’t think that this is nothing CSEd specific. The only reason that we’re doing CSEd other than the fact it’s my area of expertise is it’s accessible to students. I do think you would struggle with, say, mathematics papers where students might like, well, I don’t even know how to approach this, like, you know, I can’t get through the math to get enough of a, of an assessment, but I think almost any papers we had someone in biology, they did one of this with their classes. We’ve had people in someone in city studies did their version of it.
[20:18] Brian: Anything where you sort of want to get students that practice of reading a large number of papers in a scaffolded way, I think this is a great way to just sort of low overhead, you know, deal with a lot of the, I don’t have to have a million meetings with students or I don’t have to keep track of the students doing this. I pull the spreadsheet, and I go, OK, well, this is the students who have done all of theirs. This is the students who have not. OK, I need to discuss with those students and decide whether they want to continue on. Obviously, if you were in a course, this translates to assessment, right? So, all right, I say, hey, you know, I can easily every week pull out you every week you’ve been doing your hitting your marks. You’ve had your meetings for resolutions, you’re having whatever, or you know, you’re constantly, I’m having to remind you, I’m constantly having to nudge you, or you’re not getting things done, right? So you can figure out how to translate that into a, a more structured assessment model.
[21:17] Kristin: Yeah. This is reminding me of two things. Like, I think some of it is, so, here at Duke University, we have these independent studies that you can do with students, and so, you’re like, retroactively creating like a little reading course or something for those students that could easily slot into that model for us. I don’t know how many other schools have something like that. This also though, reminds me of UC San Diego, I think they had a paper a couple years ago of having literature reading groups for undergraduate students. I don’t think it was semester-long, though. I think it was like the groups would form and they would stick together for like, a third or half of the term for them, and there maybe is like a grad student involved in kind of organizing and helping them pick papers, but otherwise, the group was forming to kind of focus on a particular field or topic, and maybe there was like a faculty who would list out, like, here’s the papers that you should read if you want to learn more about this field. Do you think merging those ideas could work in a course?
[22:26] Brian: Yeah, absolutely. So this, I, so I split this out to a 12-week program in the SIGCSE paper. It doesn’t have to be. I’ve had, in fact, I’m doing one right now that we are on, well, I guess if you’re counting the break, we’re on probably week about 30 of it, but basically we, for various other reasons, we kind of got started late in the semester, and then had a some, and we just went, all right, you know what, we’re gonna put this on pause because we’re getting into exams, and now we’re picking it back up, and that works fine.
[22:59] Brian: I think if it was part of a like structured course where you’re like, hey, you are committing to doing X number of hours a week, you could probably shrink it down. And I think the nice thing is this is the literature map is sort of the starting point, right, so you could easily kind of go, OK, we’ve developed this literature map. In fact, that was one of the papers we did was we basically did the literature map. And then said, OK, well, now let’s do a literature review. And the literature review is basically, OK, you pull out each cell of this n-dimensional matrix and go, OK, well, these 4 papers, what do they say about this? Now, these 6 papers, what do they say about this? Now you have to actually read and assess, so I think it, it’s a very sort of malleable approach to doing these sorts of things, and yeah, I think it’s more of a case of if you want this to be just a fun exercise you do with some students, totally fine. If you want this to be part of a more structured course, I think it’ll work as well.
[24:01] Kristin: Yeah, I think for, like, an undergraduate computing education research course, this seems like a great, like, first half of the semester to really help them hone in on finding literature they find interesting, and maybe a research question they want to investigate. Based on like, what data is available to them, and then they spend the second half kind of focusing on the papers and the cells that they care about, and then also doing some kind of research analysis or data analysis or something like that. So, that’s another reason why I love this idea. Is there a place where you don’t think it would work?
[24:38] Brian: So, I think the one thing that I, I haven’t put in is anything like caveats about gaming the system, so I, you know, I do use random assignments. So in theory, you are, you know, if you’re just, yeah, I’m just gonna throw in what, say I’m not actually gonna read the paper, and I’m gonna make stuff up. You know, I’m trusting that these students are, they want this literature map to be published with their name on it. They have, you know, a stake in sort of doing it. I think if it were part of like a formal assessment where you were saying, well, you know, the difference between an A and a B is whether or not you get these done this week. That I think you would need to think a little bit about how are you ensuring that they’re not going to just, you know, figure out a way to, I mean, the current system is just an open Google Doc, so you could very quickly kind of go, oh, I’m just gonna go and change someone else’s values and stuff. So that I think you would have to kind of work it out.
[25:40] Brian: I do think it does require a fair bit of access. Again, I try to make this, this is not a huge time sink on me, but I do think this nature does require a fair amount of access to me at unusual times because it will be sort of OK. We have to get this done over the week, as opposed to we’re going to sit down in this specific 2-hour window and get this done. Now, maybe if you were doing it, of course, you could structure it a bit more, but I think the nature of the, OK, you have to go and do, you know, 50 inclusion exclusions over the next week. You do have to sort of be constantly, we use Slack and just have a channel where people will constantly be, “hey, I couldn’t access this paper,” or “I don’t understand what this term means,” or whatever, and if it’s just me doing it, that can then be a hold up for people. So I think you need to, you know, make sure that you’ve got, now I’ve done this enough that I now have other people in those Slack channels that can kind of go, oh, I had this trouble when I was doing it, or, you know, be helping out.
[26:48] Kristin: Interesting. I hadn’t thought of the fact that you’re also in some ways building a community, especially if you’re, you have a Slack workspace that it sounds like you’re just adding students as more students are doing this thing, which in some ways I think is healthy and good to have that kind of community that you’re slowly building. I have my own research lab Slack, and some of my students who have went on to grad school and are now about to graduate are still active in my lab Slack.
[27:17] Brian: I actually am going to present at one of the first undergraduate students I started working with Angela, Angela Zavaleta. She is now faculty at McMaster, and I’m going to present for her group soon, but she is still available in that slack. She still answers questions. It, I’m a big fan of that idea of building the community we have. I don’t know, 300-ish students in that Slack channel. Sure, 200 of which probably don’t regularly look at it, but there’s sort of building up that your sense of you’re part of that community, and that’s why I like this idea of, hey, if you only showed up for the 1st 2 weeks and did a bit of inclusion exclusion with us and then got busy with life, that’s OK. You’re still part of this community. You’re still able to reach out and able to join in or be involved in the future.
[28:04] Kristin: That’s awesome. Yeah, I think doing the assessment piece would be harder for, if it’s an actual course. Like, how would you structure things to help students catch up and all of that, because we can all think of a student who is like, I’ll catch up right at the last minute, and then they like do the Hail Mary and they make it, but you’re kind of like sweating the whole time.
[28:25] Brian: The other thing is because of the nature of every step you do is double-coded, and the majority of the time is actually resolving the discrepancies. It forces you, and which is pros and cons, it forces you to, you can’t leave it all until the last minute because you’re then someone else or a bunch of other students because it’s random assignment are stuck waiting on you. So again, I think this is the nice thing of this is it has this natural, you kind of have to, it’s a group worky style stuff that you are going to have to manage this stuff. The downside is it’s all the same stuff with group work. If you have one person who is, you know, really not pulling their weight, they can be pulling the whole, delaying the whole project.
[29:12] Kristin: I guess for a class, the way that I can imagine structuring it is that you set aside half a class day to handle those discrepancies. And if a person hasn’t done their work, and so they’re not able to do that step, they basically, well, you get to sit here and start processing papers and look through them and start adding things. And that social pressure potentially would make them at least realize I need to start doing this more outside of class cause I can’t be the one sitting in the corner while everyone else is talking to each other.
[29:46] Brian: Yeah, and we do meet, normally what I do is I meet once a week with whoever’s doing the mappings, and I haven’t looked too deeply, but you do notice a whole lot of activity in the spreadsheet in like the few hours leading up to those meetings
[30:02] Kristin: As long as they get it done, though, you’re like.
[30:04] Brian: No, no, it’s, it’s the social pressure really, really helps, I think with that. And again, it’s not, it’s not the same as like you’re gonna fail this course if you don’t do it, but it’s, OK, well, I’m gonna be the one who has to sit there and look at people and go, oh, well, you can’t do your work yet cause I didn’t finish mine. I think that does help a little bit.
[30:22] Kristin: I think it helps a lot, yeah. So, as we’re closing out, what are your plans for the future of this practice?
[30:29] Brian: So, I really, I want people to steal this idea. I think we’ve been doing it for a while. I’m sort of happy with the model I’ve built around it, but I think there’s a lot of different models that this idea works in. I like literature maps as a resource. I think they are actually useful in the community on their own. I’ve, you know, say if you’re wanting to do a literature review on something, you can kind of go, oh, I just want to steal this column or this intersection of the paper. It’s already there. It’s already done that high-level assessment for you. So I’d love to have these, you know, that exist in the world and are there.
[31:09] Brian: But really the idea is. I like having communities of undergraduate researchers. I think it’s a fantastic opportunity for them, but I think the barriers are often too high because either you have research-focused people who are publish or perish and overwhelmed with research stuff, and they don’t want to take the risks necessarily of having the undergraduate students involved in that, or they don’t have the time for that, or you have the sort of teaching focused people where we’re overwhelmed with our teaching and we can’t do it, or we feel like, well, the research we’re doing isn’t at that sort of level of quote unquote, hard research.
[31:54] Brian: And so there, but I think this is a very nice balance to that. I really want other people to start using this as an excuse to build up those communities. Whether you publish it or whether you build on them for future papers or whatever. The main thing I think is this gets a group of students doing something they otherwise wouldn’t. Getting to feel what research might be like and just getting that first level of exposure. So, I’ve made all those resources available. All the spreadsheets are there, we tried to make the, you know, easily accessible, the scripts for like OK, you exported a CSV and hit this, and it makes nice latex templates that you can just drag and drop into whatever you’re doing.
[32:40] Brian: What I’d eventually like this to be is, you know, undergraduate research group in a box, right? You know, you’re at an institution where you just don’t have this that exists. Hey, here is, you know, here is the step-by-step guide to going from 0 research students up to a half dozen, up to a dozen, and then it starts to gain its own momentum. So that really is what I hope people take away from this. And I’d love people to reach out and say, hey, I wanna do this, and I’d be more than happy to help you get set up and get running with these things.
[33:13] Kristin: I’m like kind of leaning in two directions. One is that I have a computing education research undergrad course, and in some ways, that course is, has too much in it, but in other ways, I feel like I need to rethink the learning objectives, and learning how to read a paper really should be an embedded learning objective in there than it is now. But another thought that I had is this is a way for early, for grad students to start out in their early mentoring journey. Does that make sense?
[33:47] Brian: Absolutely, yeah, so it’s, it’s also you don’t need a huge amount of experience to actually run these, you know, you need to be able to, it’s much more about the human resource management than it is about the technical management, right? So, I think this is would be great for, you know, and in fact, we, I have tried once, where I gave this to a, it was actually a senior undergraduate student who said, I kind of want to try running this in my own little group of students, and you know, the, all the problems they had, they, they did eventually produce one. All the problems that she had were more about, you know, oh well, you know, when I tell them to do stuff, they don’t necessarily listen to me as much as they listen to you when you tell them to do stuff.
[34:33] Kristin: Yup.
[34:34] Brian: But you know, I think for a grad student or for an early career researcher or, you know, for all sorts of things, I think this would be, again, I, it’s less, I think, in a CSEd research course or maybe just an introduction to the practice of research, right, for sort of anyone who wants to dip their toes in research. I think it’s a useful tool.
[34:56] Kristin: Yeah, definitely. All right. So, let’s move on to too long didn’t listen. What would you say is the main takeaways that you’d like someone to get out of this?
[35:08] Brian: So the main takeaway I think is literature mapping is a great way of getting students in a group that they generate one of these maps, they have to do the search, they have to do the inclusion/exclusion, they have to do categorization mapping. Everything is double-coded, runs on its own spreadsheet and script so that it’s fairly plug-and-play to do. But the secret is that this is a scaffolded exercise that it lets students get progressively deeper into the pages or deeper into the papers and read them with more and more depth and more and more focus. And it helps you build a community, and hopefully, other people will enjoy it cause it’s fairly easy to run and the students enjoy it.
[35:50] Kristin: Awesome. Thank you so much for joining us, Brian.
[35:52] Brian: All right. Well, thank you very much. It was great to be here.
[35:55] Kristin: And thank you for listening. Know someone who might enjoy this teaching practice byte? Share the episode or give us a shout-out online. That is a great way to support the podcast. And if you’re looking for other ways to help us out, join us on Patreon at patreon.com/csedpodcast, particular shoutout to patrons Roberto Hoyle, Adam Smith, and Owen Astrachan for helping to keep the podcast ad-free and supporting production. For past episodes, transcripts, and links, visit csedpodcast.org and don’t forget to subscribe so you never miss an episode! And otherwise, this was the CS-Ed Podcast. I’m your host, Kristin Stephens-Martinez, and our producer is Chris Martinez. And remember, teaching computer science is more than just knowing computer science. I hope you found something useful for your teaching today.