Abstract
Multiple choice questions (MCQs) as a tool of assessment is used at various levels of education because of its validity and reliability. The objectives of the current study was to compare the differences in MCQs flaws rate, difficulty factor (DF), discrimination index (DI), and reliability (R) of the test and to compare the non functionality of MCQs distracters (NFD) before and after the implementation of medical education department guidelines, College of Medicine, King Khalid University, Saudi Arabia. The cross sectional longitudinal and pretest-posttest design was adopted for this study for the pathology final MCQs exam for the years 2012, 2013, 2014 (where the guidelines were not implemented yet) and 2015, 2016, 2017(where the guidelines were implemented). The flaws adopted by the Department of Medical Education (DME) were the ones derived from the national board of examiners (NBME) and the Saudi Commission for Health Specialties (SCFHS) which include (grammatical, logical clues, extra details, absolute terms, negative stem, unfocussed stem and extra details in correct options). Acceptable questions difficulty, discrimination index, nonfunctioning distractors, and reliability is ranged from 30% - 85%, 0.20-0.24 or negative, <5% and > 0.90 respectively. Flaws were reported as 66.7%, 50%, 53.3% (for exam 1, 2 and 3) respectively before and 10%, 8.3%, and 13.3% (for exam 4, 5 and 6) respectively after implementation of the MCQs guidelines, with almost disappearance of the unfocused questions. Acceptable difficulties of the questions were 34%, 57% and 71% (for exams1, 2, 3) before guidelines implantation to 69%, 77% and 73% (for exam 4, 5, 6) after guidelines implementation. Changes were observed in the discrimination index: where questions with negative and zero discrimination index, dropped from 16.9%, 11% and 12% (for exam 1, 2 and 3 respectively) before application of the guidelines to 10%, 6%, 4.5% (for exam 4, 5 and 6 respectively) after application of the guidelines. Although the tendency to make all distracters functional was observed, no changes were found in reliability before and after the implementations of the guidelines. To conclude, dramatic improvement was observed in the quality of MCQs after the adoption of guidelines by Medical Education Department reflected in: reduction in MCQs flaws, improvement in difficulty factors, advancement in discrimination index and functionality of MCQs distractors, however no progress in the reliability of the MCQs exam.