Moving face-to-face exams to online TRAs as a result of the COVID-19 pandemic
Ana Costa Pereira (Module Lead, Department of Surgery and Cancer), Manuela Mura (Deputy Module Lead, Faculty Education Office) and Robert White (Deputy Module Lead, Department of Infectious Diseases)
Background to the module
Immunology & Inflammation (IMI) is a second-year elective module worth 7.5 ECTS and part of BSc Medical Biosciences (BMB) and BSc Medical Biosciences with Management (BMBM) programmes. The module benefits from a fully blended pedagogical approach, with theoretical, factual, information taught online via 20 eModules and more cognitive challenging concepts face-to-face in 20 interactive sessions.
As a result, IMI students engage with an eModule (which can take up to 4 hours depending on each individual student) focussing on an immunology key topic before the F2F session (2 hours), where they work in groups of 5. The IMI module wraps up with two 2 h consolidation sessions for which the content is decided based on perceived gaps and feedback/ requests logged on the module’s Padlet by the students.
Due to the complexity of the material, the IMI Team place greater emphasis on the bigger picture, focussing on explaining concepts and cellular mechanisms, as opposed to requiring students to learn all the names of cell markers and cytokines. It is not important to know exactly what this means: suffice to say that it requires much memory and rote learning and, as we have found out, is not needed for our undergraduates to obtain a sound foundation on the key principles of immunology.
Exam format
The original exam was designed to be a 3 h closed-book exam. It comprised:
- Part A. 20 single best answer questions (SABQs) (25% of the total marks) (25 min)
- Part B. 15 assertion reason questions (ARQs) (30% of the total marks) (30 min)
- Part C. 2 multi-part data interpretation questions (45% of the total marks) (1 hour and 30 min)
Questions in Section A and B are carefully written to avoid recall and simple memorisation. All questions are double checked by two additional academics who will challenge the author and thus serve both as quality control and gate keepers who ensure that students must have, at the very least, a grasp of immunology to be able to tackle each of the questions. These are automatically scored.
In part C, students are presented with data from scientific papers previously published and listed on PubMed. Sometimes, the IMI Team re-labels images or omit some figure panels for the sake of simplicity. In this section, students are asked to interpret the data and integrate it with the knowledge they have developed throughout the IMI module.
How was the exam adapted to a time-restricted assessment (TRA) online?
Because the original exam already focussed on testing high-order thinking skills, the online exam was identical to the original exam and it was set up on Blackboard by the Faculty Education Office (FEO) BMB Team. Although students were instructed to sit it as a closed-book exam, we changed any details provided in Section C that would make the original articles, from which the data had been taken, identifiable. This included code naming molecules, and relabelling graph axes. Further checks of questions in Sections A and B reassured the team that the questions could not be easily answered by simply resorting to memory and they thus remained untouched. We also did not change the duration of the exam because it has always been an IMI Team core belief that IMI students should be tested for knowledge rather than speed with which they can answer questions, and the original exam time set was longer than it could be.
Students were advised against consulting notes and the internet, as this would likely cost precious time which would be better used interpreting the questions. They were also reminded about the importance of being honest.
Exam outcome
Results distribution was similar to the previous academic year. Students who did best in the ICA also did better in the exam, and the exam top marks were attained by students that had best performed throughout the running of the IMI F2F sessions. Analysis of SBAQs and ARQs also suggested that questions were not easily searchable, with some straightforward questions still requiring understanding, and being missed by students whose performance was weaker. We did not detect serious misconduct in the IMI exam but we had no means of looking systematically for plagiarism, as BlackBoard exams were not scanned for plagiarism.
Lessons Learnt
- The way the exam is presented and organised can make a world of difference to students’ stress levels and it is something that requires a lot of thought. For example, in Section C, our students lost a lot of time scrolling the script up and down examining the images and writing down the answers. This was the result of images being presented sequentially, one after the other; on balance, it may have been better for the students to have Section C figures intertwined with the questions
- The type of questions must probe beyond factual recall or rote learning, which is ‘googleable’ and thus fraught with temptation to cheat. For example, if data handling uses research papers as a data source, codification and relabelling is really important
- Students appear to need more time for online exams, so be prepared to either reduce number/ scope/ complexity of questions
- Online platforms that are specially designed for exams are badly needed. Within Blackboard Exams, no plagiarism detection was available, no suitable marking scheme could be set (in theory it is but it is so awkward the team ended up not using it) and blind double marking was not possible as the system randomised examiners and made it impossible for the three examiners to be sure which scripts each had corrected. The system also crashed on several occasions.
- Because we had anticipated that scrolling up and down to look at images and write down the answers would be time-consuming, we had provided the students with a PDF with the images as the exam was released. However, Blackboard would not allow the students to access it on separate devices (e.g. laptop and iPad), or to have two windows open, which made said PDF of little use to students.
Advantages of online assessing
- Extremely easy to mark online which enabled better annotation of scripts
- Quicker marking as all answers were typed and no time was wasted deciphering more difficult handwriting styles
- Prompt exchange of scripts between examiners and an easier moderation process as everything was done digitally
- More environmentally friendly – no paper waste!
- No fear of losing scripts
Take home message
On balance the IMI academics favour online TRAs versus presential TRAs but we believe that careful exam design is absolutely crucial with regards to the type of questions, time and layout.