Urgh, exams. The epic ‘true-false-no idea’ multiple choicers of my undergraduate days are not a distant enough memory for me. The whole ‘get it right, get 1 point’, ‘get it wrong, lose 1 point’ approach always seemed horrendously unfair, regardless of the statistical basis for the strategy (i.e. examiners don’t want to reward people that guess true or false on every question, as in theory they’ll get the right answer 50% of the time).
According to researchers at University of Nottingham, my personal MCQ fear is well founded. Shona Kelly and Reg Dennick have found that male medical students are much more likely to do well on an exam with some ‘true-false-abstain’ questions than are female students.
The authors of this study looked at seven years worth of results from medical school course assessments – which included course work, essays, in-class assessments, lab studies, Objective Structured Clinical Examinations (OSCEs), short answer tests, single phrase tests, spotter quizzes, single word answer exams, true-false-abstain questions, and Vivas.
Among the 359 course assessments, there was a statistically significant difference in the marks between the genders in a third of the courses in any given year. Univariate analyses indicated that women did better in assessments that included some in-class assessment and some short answer questions, but struggled with exams that included true-false-abstain questions. Men did better on assessments with some true-false-abstain questions and were at a disadvantage in those that included some short answer questions.
The female advantage seen with in-class assessment disappeared in multivariate analyses that took into account the subject area/content of the assessment and calendar year, however, and the advantage that females seemed to have in exams with short answer questions was very small (odds ratio 1.03).
On the other hand, the association between true-false-abstain and male advantage wasn’t affected in multivariate analyses. In fact, males were 16.7 times more likely to score higher than females if at least some true-false-abstain questions were in the assessment. This difference could be down to the fact that women were more likely to pick the ‘abstain’ option in this format exam.
So there’s the answer. My difficulty with university exams clearly had nothing to do with my preparation and was obviously down to the inherent gender bias in the format used for biomedical exams. Cough.
————————————————————————————————-
Kelly S & Dennick R (2009) Evidence of gender bias in True-False-Abstain medical examinations. BMC Medical Education 9(1). DOI: 10.1186/1472-6920-9-32
6 Comments
Take the multiple choice qualification test. (No thinking required)
Which degree do you want:
a: Medicine
b: Science
c: Economics
d: Law
e: I don’t know
Congratulations, if you chose (a), (b), (c) or (d) you will receive that degree.
If you picked (e) a randomly allocated degree will be awarded you.
Multiple choice degrees are worthless.
No wonder professional standards are so low.
We’ve compared year 1 and year 2 student performances in the biosciences in MCQ tests (5 options given) with two different answering styles, ‘single best answer’ and ‘elimination’. We did not see any significant difference in gender in either type of answering format of MCQ in either of the years of study. Even single best answer did not show gender bias in performance!
Students liked the elimination over single answer as they could show partial knowledge and were not made to guess if they had 2 or 3 options left after discarding answers they knew were incorrect. As the cohort of bioscience students is not too different from medical students (some of our students go on to study graduate entry medicine), I’m not sure how to explain the differences seen. It must be said though that multiple choice with 5 options is a different type of test than True-False-Abstain…
Re subject specificity in gender bias in mcq: there is data that this is culturally based, not subject based… an mcq on maths in the US did result in gender bias whereas in China it did not show a gender bias. In western society girls were made to think that maths is difficult whereas other cultures girls are not raised in this fashion and deal with it. Perhaps campaigns as Women in STEM subjects are finally paying off and girls indeed think they can study whatever subject they want as confident as boys!? Let’s hope so!
i disagree with test with multiple choice
i think it just about luck
if we have a good luck we can get a good value
very bad test
Thanks for the comments guys. The whole ‘men take more risks’ aspect of this research is quite interesting, although these authors’ results can’t really add anything to that particular debate. They’d need to do some kind of nested psychometric study to determine the risk-taking characteristics of the sample before they could draw any conclusions I reckon.
I’m less sure about the possibility that girls don’t take risks in biomedical MCQ exams because they’ve been conditioned to believe they’re hopeless at the subject. It would be interesting to see if the pattern is similar in a different discipline to see whether the confidence of women in their answers is affected by society’s view of the academic subject, although I’m guessing female-dominated subjects like the arts have essay-based rather than MCQ-based exams.
hmm…. I’ve always done extraordinarily well on “multiple choice/abstain” tests, and I’m female, HOWEVER, I was homeschooled as a child and actually never encountered the “girls aren’t as good as boys at math” theory until well into my teens. So I was never afraid to guess, because by that time my self-confidence in my abilities was already settled.
But girls who have been subtly told they “aren’t as good” at a particular field are DEFINITELY going to feel a bit more hesitant on whether or not their answers are correct… and thus more likely to “abstain.”
So I don’t think this is really at all a basic gender divide but rather the logical response to being told you aren’t expected to do as well…
Interesting result if it can be replicated. There have been studies before concluding that in similar circumstances men appear to be more confident/risktaking. This seems to be the case here, since the women abstained more and this drove the result.
If one wanted to keep the format and insisted on statistical equality one could presumably increase the punishment for answering wrong to equalize the result. I found the following quote from the paper amusing: “The explanation for this phenomenon is not clear but the common argument used is that it is associated with the greater risk taking behaviour of males. However, it could be equally due to the more cautious behaviour of females or different problem solving strategies.”
Apparently males being more risk taking is a different theory than females being more cautious.