Episode 1: Online collaborative exams with Dr. Mariel Miller

Download: 01-Dr.Miller-Transcript

 

REBECCA

Welcome to the Teach Anywhere Podcast: EdTech Stories from Real Educators, where we interview faculty and instructors about how they use educational technology in their courses at UVic.  My name is Rebecca Edwards, and my pronouns are she/her.

 

THIAGO

I’m your co-host, Thiago Hinkel, and my pronouns are he and him. Before we begin, we acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands and the Songhees, Esquimalt and W̱SÁNEĆ peoples whose historical relationships with the land continue to this day.

 

REBECCA

So with us today is Dr. Mariel Miller. Dr. Miller is an assistant professor in the Department of Educational Psychology and Leadership Studies at UVic. Her research and professional interests include the design and use of online environments to support regulation of learning as a dynamic and social process. Hello, Mariel. Welcome to the podcast.

 

MARIEL

Thanks for having me.

 

THIAGO

Thank you for joining us today. Before we start chatting about your course, we were hoping that you could introduce yourself. Perhaps you could start by telling us about your journey here at UVic. What did you do before you started your current role?

 

MARIEL

Sure. Before my current role, I was the Director of Technology Integrated Learning at the LTSI. So I did that for quite a few years. And before that I was a learning experience designer. So that’s really what started my journey towards my current role.

 

THIAGO

And would you like to tell us a bit about your research interests and maybe briefly about the courses you teach?

 

MARIEL

Sure. So my research focuses on the intersection of learning and technology. So how instructors learn to integrate technology into their teaching and also how learners go about collaborating in digital environments and how they can be supported. And I’m also quite interested in how people can be supported by advanced technologies like learning analytics and AI and how people make sense of these tools too. So at the moment, I’ve been teaching ED-D 101, which is Strategies for University Success. And I also teach in the graduate program in EPLS.  So I teach 570, which is Self-regulated Learning, 508, Learning Theories, and 590, which is Psychological Foundations of Instructional Design.

 

REBECCA

Gosh, so it sounds like you work with undergraduate students, like all the way to graduate students. So you get a wide range of students in your, in your courses.

 

MARIEL

Yes, definitely.

 

REBECCA

So, before we started the recording today, you said that you wanted to tell us a little bit about a specific course and kind of some of the EdTtech work that you’ve done in that course. And I think that course was ED-D 101?  Is that, am I correct about that? Yeah, that’s right. Ok. And so I wanted to ask you a little bit more about that course specifically, and I think you’ve kind of already answered some of these questions, but I’ll go ahead and ask. So, is this the first time that you’ve taught ED-D 101?

 

MARIEL

It’s not. I’ve been teaching ED-D 101 in different varieties since I think 2009 or something like that. But I took quite a, quite a break in teaching it. And so last year was the first time I’ve taught it in probably about five years.

 

REBECCA

And I called it ED-D 101 because that’s what you had called it previously. But the full title is ED-D 101… Learning Strategies for University Success.

 

MARIEL

OK, perfect.

 

REBECCA

And how many students are in the course? And kind of what does this typical student look like in that course?

 

MARIEL

Sure. There’s… I think last term we had about 135 in one section, sometimes there are two sections that run at once, and it’s a first year undergraduate course. It’s an elective. So it’s open to almost any student on campus who can take it, and the goals are really to support students in um learning the theories of self-regulated learning and how to apply that in their own learning in ways that work for them as individual learners.

 

REBECCA

When you teach this course, what kind of instructional strategies do you like to focus on? Like what’s your kind of pedagogical approach to this course?

 

MARIEL

This course helps students to take the theory and concepts around self-regulated learning and think about those and then practice applying them in different ways to their own learning. So as a student throughout the course, they’re supported to almost collect data about their own learning and think about how these concepts and theories apply to them in different ways and how, how they might not. So it has a lot of really practical and applied activities where students work on their own learning in other classes.

 

REBECCA

So it sounds like it’s quite process-based for students and very like student-centered in the sense that students are able to approach the material in the way that best suits them.

 

MARIEL

Yes, that’s exactly right.

 

REBECCA

I have one more question about the course that you’ve taught. So you mentioned that it is typically first year students, and most of them are new to the university. Are they taking the course online? Are they taking the course in a blended format or is it a face-to-face course?

 

MARIEL

The course runs in almost a multi access way in the way that the lecture section is asynchronous online or has been in the last year. And then there’s a lecture section where students are practicing applying the concepts to their own learning. And students can enroll in either an online version of that or an in person version, whichever they prefer.

 

REBECCA

So it’s really student choice as to how they choose to kind of engage. That’s awesome.

 

THIAGO

So one of the objectives of our conversation today is to learn a bit more of educational technologies, digital technology that you use to teach this course. So I was wondering if you could tell us about one way or one educational technology that you use and you would like to discuss today? Is there a specific problem you were trying to solve when adopting a specific technology?

 

MARIEL

Yeah. Well, one thing that I was focused on was assessment in a an online format or ways that technology can support assessment really.  So I was thinking about how we can think of assessment slightly differently using technologies that are available now, and the technologies that I use were primarily Teams and Brightspace.

 

THIAGO

Yeah. So when deciding to use both Microsoft Teams and Brightspace, is there a specific tool or feature in those platforms or in those technologies that you adopted and are using?

 

MARIEL

So what I was trying to do is deliver an online collaborative exam. So I wanted to use the technologies in ways that would support that process. So I used Brightspace to help students become familiar with the expectations for the exam and to provide like advanced material and allowed them to get a big picture idea of what would be happening in this exam. So that was our course management system. And then I use Teams as the actual environment where they met to collaborate. So on the day of the exam, that’s where they came to meet as a whole group and then they split off into their individual small groups to write the exam.

 

REBECCA

So you were using two different kind of learning environments kind of together in order to provide the exam. So Brightspace kind of for some pieces of the exam. And then you were using Microsoft Teams for kind of the communication aspects. It sounds like they were using the chat, potentially video call? Did they do video calls?

 

MARIEL

It was video, yeah.

 

REBECCA

Oh, it was video! Ok. So you were using Teams for kind of the synchronous video-based discussion. And so students had these two environments to navigate. Did you have any problems with students navigating those two environments? Did any challenges pop up on the exam day?

 

MARIEL

So to make sure that students didn’t run into too many difficulties, we delivered this exam in three phases. So because the course is about self-regulated learning, we use sort of theories of self-regulated learning to frame the exam. So we had a preparation phase, where students in Brightspace got access to all the exam instructions and a checklist of all of the things that they would need in order to write an online exam, an online synchronous exam. So things like a quiet place to write the exam, a pretty stable internet connection, equipment with a webcam. And we wanted to make sure that students had all of those things and knew that those things would be important. Most of the students had those already because they were… because it was an online course. However, some students were writing, had a… so some students had an in-person lab, so they didn’t necessarily meet with others online regularly. So by making sure that they knew what the checklist was, they could let us know if they didn’t have any of those things available to them. And we had a physical space available on campus where they could come and write the exam. So the exam was equally accessible to all students in the class. The other thing that was in the preparation phase is a practice exam. So in one of the labs before class, before the exam class, we had them break into small groups and do essentially a dry run of what it would be like to write this exam. So it was a smaller problem. It wasn’t the exam content, but it kind of got all of those technical pieces out of the way. So that on the exam day, they could really focus on their learning and their collaboration. It was important to support the students in understanding what was expected and getting all of the sort of technical pieces in place so that those didn’t interfere in their learning during the exam.

 

REBECCA

Absolutely. So it sounds like you were able to use technology to help students, and when I say technology, you gave them a checklist. So it was a digital checklist that they kind of like worked through. So you were able to create digital learning environment that supported them to be prepared for the exam. And then you also provided opportunities to practice with the tools in advance. So it wasn’t that the exam was successful because all the technical pieces worked perfectly together. It’s rather you set up the environment and created scaffolds and supports for your students to help them be successful in that environment.

 

MARIEL

Yeah, exactly. And one of the learning outcomes for the exam became developing competencies for collaborating online. So because the online environment offered us the opportunity to support students in that, that became part of the learning in the course.

 

THIAGO

Yeah. And I was wondering how exactly did students collaborate in this exam? Right? Did each person write their own exam? Was it like one exam per group? Could they talk about the questions and, you know, come to a conclusion together and then each person would answer their own exam?

 

MARIEL

Yeah, great question. There’s lots of different ways to go about a collaborative exam. So, my goal in this case was to consider assessment, I mean, we often consider it as an individual endeavour, that it’s a point in time where we assess what students know. But I wanted to reframe it in this course to be assessment for learning so that students had an opportunity to take this assessment experience and to treat it as a learning experience so they could interact with peers and get a chance to discuss these topics in depth so that they could take this exam as a learning experience rather than of learning. So to support students with going about collaborating in an exam, what happened is that they all came into a room together at the beginning of the exam, and it didn’t look that different at that point, you know, you have, you know, your guidelines for what an exam might look like, the timing and what happens and how to access help and all those kinds of things. And then students broke off into small groups that we had set up beforehand. So we had set up groups based on like diversity and course performance to date. And we wanted to make sure the groups were a mix of different students. They may not have worked with those students before. And so they broke off into small groups with their student to meet in their own Teams meeting. So it was a synchronous video chat and they had access to the links for the exam. It was a quiz in Brightspace. So they shared their screen and worked through the questions together and then they submitted a group project or a group product at the end of that. So that they only got one grade, it was a group level assignment.

 

REBECCA

And this is kind of a really specific question, you mentioned that you had a physical space on campus for students that maybe didn’t have all the required equipment in their homes. And you also mentioned that they were on video calls. So I’m curious how you managed to have kind of people in a physical space together on video calls. This is just a logistical question. Did it go ok?

 

MARIEL

 It did go ok surprisingly.  We didn’t have that many students show up and the students who did show up actually seemed like they showed up because they just didn’t remember that that week was the online exam week. So that was their regular classroom. So we kept all of our regular classrooms for people who came to lab online. So it was more that the people who showed up were… just forgot that it was the online week that week despite all of our preparation experience. And we had some separation in the class. And because they were listening to their other group members talking, they had headphones so that they could cut down on the noise in the room. And then they weren’t all talking at one time. So there wasn’t a lot of noise in the room.

 

REBECCA

Yeah. Yeah. So it did work out, kind of, it wasn’t too bad. That’s great. So, I’m curious, you’ve talked a little bit about how the actual exam played out and how, like, the logistics of how students met and how they interacted. And I’m curious how your students reacted.

 

MARIEL

Yeah. Well, it’s interesting because in the third phase of the exam, we had them reflect on how things went. So we guided them to… it was the week after, so part of their work that week was to consider how things went, what challenges they encountered, what they would do differently, and what they wanted us to know about this collaborative exam. And the comments in that reflection were overwhelmingly positive. So the students really liked having the opportunity to work together on an exam. So they just mentioned that they like being able to bounce ideas off other people. There’s obviously some people who mentioned they didn’t… they were a little bit anxious about it going into it. They didn’t know what to expect. So I think having that preparation phase was really important. So we did hear those two things, they weren’t quite sure what to expect, but they really enjoyed the experience overall with the over… with the general feedback.

 

REBECCA

So you’ve talked a little bit about how your students kind of reacted to the exam. And I’m curious what was the impact on their learning or what did you feel the impact was on their learning?

 

MARIEL

Another thing that came up in that reflection was that students noted that having the chance to connect with each other in this synchronous exam, they really value that in terms of making connections with other students. So one thing that student said is that having the chance to talk with others and and solve a problem together helped them sort of feel a sense of community in the course. And so I think that was an impact of the exam we weren’t necessarily looking to build in, but students did note that that happened. And then another one I think is it did help them to have some experience with what it is like to collaborate online. So how to meet online, how to collaboratively edit a document, how to make sure everyone has a chance to speak, how to resolve it when there are differences of opinion and what the answer might be to a particular question. So having those experiences in class in your learning and being supported to reflect on them, I think was helpful for the group. Knowing that there are a lot of remote teams and remote collaboration opportunities happening once graduation occurs.

 

REBECCA

Yeah. So it felt really practical to your students and it was very tangible. And so based on that, you kind of talked a little bit about the ways that they interacted, like collaboratively editing, and then also like video chat. And I’m curious like some of these tools that you’re mentioning, they’ve obviously got better and better and better in recent years. And it sounds like you’ve actually at various times taught this course. And I’m curious, like, did you do this type of exam previously? And how has it improved or changed over the years?

 

MARIEL

Yeah, we did do this type of exam previously when the course was face-to-face in pre-pandemic and before video conferencing tools were available, then we would book a huge computer lab and the students would all come to the computer lab and write the exam via chat. And now that the technology has evolved, I think that having the chance to meet over video conferencing is much more authentic to what they’re doing. It also allows for different kinds of conversations to happen. So I think what I noticed is that the depth of conversation that happened in video conferencing was different than might occur in a chat. So I think it was more successful in that way. That being said the technology doesn’t, it’s not as easy as it could be to set up. So because we wanted the students to be able to look back on their collaboration, we had to individually set up every single Teams meeting for every single group and then add the students individually to each. So Teams and Brightspace don’t necessarily talk to each other, so there wasn’t an easy way to do that.

 

REBECCA

So there was a lot of like upfront work for the instructional team to prepare the environment for students.

 

MARIEL

Yeah. And had we been doing a collaborative project that would have lasted over time, it probably could have made it easier or ask students to kind of gather themselves into groups. But this was a collaborative exam, so we wanted to have a little bit more control over the environment than we might otherwise.

 

THIAGO

Yeah, and I’m just curious too because this is a first year course, right? And maybe this might not be something students would expect like to take an exam in a group, right, which is usually the opposite, we want just like to separate people and make sure they do not communicate, right? So was it at all surprising to students that they would have the chance to do that? And how was it like to sell this idea to them? Were they like eager to buy in and say like, yeah, let’s do this.

 

MARIEL

I think that we often spend so much time thinking of exams as these individual events where there’s a need to kind of control what students have access to and to make sure they don’t talk to each other, and that becomes particularly challenging when exams happen online because you don’t have as much control over the environment. But I think what was really powerful about this particular experience is that by delivering a collaborative exam, the point of the exam is to promote interaction among students to help them build on each other’s ideas. And I think that’s what we saw happen. So I think the students positive reactions to the exam really showed that they appreciated having the chance to to talk about exam answers and sort of co-construct one answer for the group. One thing that might have helped is that we did allow students to drop the midterm exam grade and replace it with the final. So there might have been a subset of students that felt really strongly that their performance should reflect only their performance; they wouldn’t want others to impact it. So for students who might have felt like that dropping it and replacing it with the final was an option. But that being said, you know, we didn’t hear many, many concerns about it.

 

REBECCA

Yeah. So you were kind of creating some flexibility for learners so that they could find their own comfort level.

 

MARIEL

Yeah. And I think this exam overall aimed to provide flexibility. So there was flexibility in terms of where they could write it, they could write it at home or they could write it in a room campus. And there was flexibility in terms of being able to access the materials, so that we gave them the scenario ahead of time. So the goal wasn’t to test what they could do in a certain time limit and it was open book. So there was flexibility in terms of how they kind of put together their answers or went about an exam. Whereas I think exams tend to be very instructor controlled otherwise. And in some cases that makes sense, you know, perhaps there are programs with accreditation and there’s particular reasons for having those kinds of exams. But in this context, this type of assessment matched the learning outcomes for the course and for this kind of midterm moment in the course.

 

THIAGO

So I was wondering, is there something that didn’t go maybe as expected? Were there lessons learned for you and the instructional team? And would you adapt your approach in the future?

 

MARIEL

I think I would definitely adapt the approach in the future. One thing that we did this time was we did opt for a very simple collaborative exam where students came into the exam knowing what to expect hopefully and then wrote the exam as a group, and they didn’t necessarily know their group first. So in the future, I think it would be interesting to make it so that students collaborate with the same group for longer over time. So there’s less time spent getting through the initial getting to know you part of collaboration during an exam context. So that’s probably something I might change in the future. And there’s also different types of collaborative exams. So there are sort of two step exams where you might collaborate first and then do an individual second part to the exam. So, you know, I could see looking into that in the future, but I think we did learn some lessons this way. But it was overall a pretty good experience.

 

REBECCA

Ok, thank you so much for sharing the story of the collaborative exam in your course with us. And before we say goodbye, I’d like to ask you one final question. And that is what is one question you wish I’d asked you and how would you have answered it?

 

MARIEL

One thing that I wish I had said, and it’s not that you didn’t ask the question, was that when we think of assessment, sometimes we spend so much time and energy trying to stop students from doing things we don’t want them to do. When the focus is on that, we miss these opportunities to come up with new ways of assessing students or providing new assessment opportunities that use digital technology that actually supports their learning more than it might in a traditional exam context. And then maybe there’s one other thing. The one other thing I didn’t mention is that when we’re designing this assessment, one thing that we paid particular attention to was accessibility and accommodations. So it could have been that there were accommodations in this class that made it, made it so that student would need an accommodation from this type of exam and that would have been provided. But there were a lot of accommodations that would have been needed if we weren’t providing this kind of exam as well. So I think that when you try and design assessments that are accessible overall, it is a better experience for students.

 

REBECCA

Yeah, creating a more accessible learning experience, a better learning experience for all students. Absolutely. All right.

 

THIAGO

Yeah. Wonderful. Thanks again for joining us today, Mariel. That was a great conversation, and we wish you all our best.

 

MARIEL

Thanks for having me.

Hosts: Rebecca Edwards & Thiago Hinkel 

Guest: Dr. Mariel Miller 

Technical Production: Thiago Hinkel

Transcript Preparation: Thiago Hinkel 

Theme music: “freesound1.wav” by freezound5 (https://freesound.org/people/freezound5/sounds/588258/). Available for use under the CC0 1.0 Public Domain Dedication at freesound.org.