Hi, welcome back. Today we're going to talk about multiple choice question writing. But before we talk about assessments, I want to just review for a moment the idea that we need to put the whole picture together. So our intended learning outcomes; what we're trying to teach and how we're actually going to assess it. So just as a reminder I want to talk, just for a moment about how we should blueprint our curriculum. And then switch to an assessment. Specifically multiple-choice questions. The rest of this module, we'll talk about other forms of assessments and those will be in separate videos. And then the challenge will be to construct multiple-choice questions to assess higher-order learning. So those are the goals of this session for you. So people always say, so you want to teach. and it, it can be a very [LAUGH] daunting task case in point, right here. but really, when you're trying to teach something, there's lots of questions that you need to ask. So you need to ask, what are you going to teach? And where might you teach it? Whom is the learner, how are they going to learn, but most importantly, after you've done all of the instruction and everything else. How will you know that they actually learned what it is you set them out to learn. So one way to try to know where you're heading is to create something called a curriculum blueprint. Just like you might do a blueprint for a building you're trying to build. So I always say to start backwards. So you define your intended learning outcomes before you start. And that's why you'll see in these videos I always start with ILOs, as well as online, the sessions always start with the ILOs. So we can always keep track of where we're headed. Okay. Once you know where you're headed, then you can figure out why you're doing it, and what to do. So the first question is, well, really who cares what you're trying to teach. And so, you may want to consider doing a needs assessment. Is this something that people want to learn or need to learn? And then the important thing is to also define your learner. Are we talking about novices? Are we talking about experts? Are we talking about a global audience? that helps define which way you want your curriculum to go. And then again the most important piece for this session and that what we're doing in this whole course is what teaching tools will you use. So my favorite, sort of illiteration, is to pick a pedagogy and you want that pedagogy to match your intended outcomes. But first you have to think about well, okay, I'm going to teach something. And I'm going to try and measure their success, but how do I do that? So while, we will I promise get to pedagogies and specifically what you can use in which situation, we need to talk just a minute about assessments. Which is why this module is going to be about multiple choice questions. So the first obvious question or statement is that there's no such thing as a perfect assessment. You're trying to get an approximation. And so you'll hear terms like reliability, meaning that the results you get when you administer an examination or a test of some kind, are reproduceable. They can be relied upon, that People who have learned this information to this degree will reproduce the same results in a variety of tests, situations. So, if you gave a similar test to the one you just gave. If you were having a reliable assessment, you would expect that the results on a similar test would be the same to the one that you're giving. Now the challenge with reliability is that it's going to be limited by sampling errors. So if you don't have enough items on your test, then you may actually not be able to capture all of the information that your learner has learned. Or if you have different examiners especially in a standardized test situation, or an essay, or an oral presentation. If there's variation between the examiners, there may not be reliability between examiner A and examiner B. So examiner B may grade tougher, or more rigorously, than examiner A. And so one way to sort of mitigate that is to have your examiners actually grade the same tests. So that you can make sure that there's standardization between how they look at an exam. So for instance, you may have a 100 papers that you need to grade and you have three people who are grading. So those three people, you could have them actually grade the same 10 items. Sorry, the same 10 exams, and then see how closely they each graded. That exam to the same level and once you know that they can grade the same way then you have something called, sort of, inter rater reliability. So that you know no matter who gets the exam paper, that person has been trained and standardized to score the exam in the same way. Another challenge is something called validity which is a little hard to sort of get your head around a little bit. But really what validity means is that you're actually measuring what you think you're measuring. So basically you can have two, there's lots of different types of validity. and you can have an entire course on just these two issues of reliability and validity but I just want to give an overview. So you can have content validity, which basically says, the content on the exam is what I either taught or what I think is crucial for people to learn. And if you don't sample enough of the curriculum, then you may not have enough content validity. So, an example that many people us is, that if you're giving an exam on pulmonology or diseases of the lungs. You wouldn't want all of your questions to be about pneumonia. You may want to have other questions about pneumonia and asthma and reactivate airway disease, etc. The other piece that's hard to understand is something called construct validity. So with construct validity You're basically measuring something that is a theoretical thing. So, for instance, how can you measure intelligence, right? If, you many have a theoretical construct of what intelligence is. And so then you create an exam to try to measure that. So if your exam actually doesn't match up with the construct, then you're not measuring what you think you're measuring. So you have to just be really careful in terms of assessments to make sure that the results that you get are reproducible. Since you can rely on the exam to test what it is you think you're testing. And also that your results are going to be valid. 'Kay, that's a lot. But another way to think about it, another way to organize the structures, is to think, okay, what are my intended learning outcomes. And you can go back to doing taxonomy and look at those learning verbs depending on the level that you want people to attain. So if you want them to be able to synthesize, you might use synthesize verbs Etcetera. Or to work at Miller's pyramid. And think about the domain that you're trying to teach. Is it a knowledge, is it a skill, or an attitude? Again we want to match that pedagogy to our intended learning outcome as well as to the type of domain, and the assessment should match. So let's flip it just a little bit differently. Let's say, all right, we're talking about assessment types and we're talking about the variety of domains. So we can talk about knowledge domains and we can talk about how you would evaluate someone's knowledge. You could use multiple-choice questions. You could use short answers or problem-solving sets. You could have them do essays. You could have them do oral exams. You can see how each one of those assessments will match or map back differently to which ever level on Bloom's Taxonomy. Or Miller's pyramid you're intending to get to at the end of your educational intervention. the other way to think about it is okay, how do I evaluate skills? And many of us are familiar with OSCEs, or Objective Standardized Clinical Examinations. Or OSATS, which are Objective Standardized Assessments of Topical Skills. Or in the United Kingdom, they're also called DOPS, which is Directed Observation of Procedural Skills. Now the hardest domain to measure are attitudes or attributes that people have. And you can use essays, portfolios, art, either written or performed, and literature as well. So again, I'm just giving a basic overview of why thinking about ILOs And thinking about how you're going to assess what it is you're trying to get your learner to, will then help you pick your pedagogy. So it's a sort of circular, you know, which came first, the chicken or the egg. but it's something that I wanted to make sure we mentioned. So let's get really to the task at hand, which is talking about multiple choice questions. So, the biggest question is, what's a multiple choice question? Is it A, a way to assess factual knowledge? B, an efficient way to assess individuals? C, a way to assess higher order understanding of information? D, a question with only one right answer. Or E, all of the above. Okay, let's go through first the types of multiple choice questions. So you can have a multiple choice question that is basically a true, false item format. The important piece about a true, false item Is that the choices must be unambiguous. And so you can't have things that could somehow be true or might possibly be false. There have to be clearly false answers and clearly right answers. So, for instance, you could not say that the sky is purple Or blue-gray or blue, because depending on where you are and what time of day and the light, et cetera. and the location, it may be that the sky could be all of those things. So the trick with true-false items and especially to keep yourself out of trouble after you've administered exams. To make sure that those choices are truly true and the false choices are absolutely false. The challenge with these types of multiple-choice questions, though, is that they lend themselves towards the recall of an isolated fact. And so if you're trying to asses higher order skills, you may not sort of pick that up by doing a true/false item. So the other item format is single best option format and that allows for the best matched answer to be produced by your learner. And you can incorporate multiple cognitive jumps within the question. Meaning, you can ask a straight forward knowledge question like the capital of Iowa is? or you can ask other questions that require the learner to sort of make a few cognitive jumps. So you could ask a question the city where London Bridge is located? Is which of the following, so that, that makes the person sort of jump to the answer. Okay. so let's go through sort of the anatomy of a multiple choice question. But before we do that I just want to place out a little disclaimer there which is that. I have the opportunity to write multiple choice questions for the National Board of Medical Examiner in Philadelphia, PA. And most of what I'm talking about I learned from them and I write questions for them as well. So let's look at the anatomy of a multiple-choice question. So the first part. An instructor's teaching a module on designing assessments for distance-learning. There are more than 5000 students. She wants to confirm that students learn the components of a multiple-choice question. Which of the following assessment types would be most suited to her purpose. A Essays, B Group projects, C Video-taped submissions, D Multiple choice question examinations. Okay before you panic, we're actually just [LAUGH] going to go through the different components of this question. So the beginning, or the first paragraph is what you call the item stem; sort of sets the, the scenario that you want people to think about. The next piece is the question that you're asking. So it's the lead-in question. And then the remaining pieces, of course, are the answer, the one correct answer, and distractors. Okay, so the item stem should really be focused on a concept and should not be based on memorization. Many of us are familiar with the item stem being in the form of a clinical vignette. And what you want your items stem to do is have enough information that you can formulate an answer to the question without ever looking at the options. So here's an example. A 35-year-old woman complaints of nausea and vomiting for the past three weeks. She also notes urinary frequency without burning. Her last menses was seven weeks ago. On examination, she is afebrile. Notable finding on exam includes a mildy enlarged pelvic mass. Her Bhcg is 6, 582. Now hopefully most of you with multiple cues within that item stem, know the answer before I even give you the options. So the next piece of the question, the multiple choice question's actually the lead-in. And the lead in should be specific to the vignette. So after that vignette which I've included on the slide, the next question is which of the following is the most likely diagnosis? The important thing about the lead-in question is that it can't be answered simply by reading the lead-in alone. Because otherwise you have no need for the vignette. So the lead-in should really direct the examinee towards a thought process or what, of what it is you're trying to ask them. So what is the [INAUDIBLE] of the question basically. Okay, now the answers and distractors. Again I've included the same stem, and I've included the same lead in questions. And now I've given you some answers and distractors. A, Leiomyoma, B pregnancy, C small bowel obstruction, D Ovarian cancer, or E Colon cancer. Now, important things to remember is that answers and the distractors should be similar types. So there's all diagnoses that I've listed and all diagnoses of organs within the pelvis. I did try to alphabetize [LAUGH] them, but I had failed in that effort. one of standardization is just to alphabetize them, since you can avoid people sort of being test wise and always picking answer C. The other thing you want to do is avoid something called queuing where if you include only two gynecology answers. And everything else is different that may cue. Or if you include similar words in the distractors that may cue people towards choosing between those two. And then they just have a 50% chance of getting The question right, wh, rather than actually answering the question. sometimes your correct answer will be about a mile long. And that actually cues people that that's the right answer. The other thing to remember is that all of the choices that you put as distractors should be plausible to the vignette. So in this case, I wouldn't put lung cancer, because that's just implausible given the vignette as it's written. 'Kay, so here's my own example of poorly-written stem. The instructor is teaching a module on designing assessments for distance learning There are more than 5,000 students. She wants to confirm that students learn from the session. Which of the following assessment types? Now, the problem with the stem is that, what are students supposed to learn from the session? Right, if you don't know what the intended outcome is. If we don't know what they're supposed to learn, how can we pick from the variety of assessments at all. So that's actually an example of a poorly written stem. This is an example of a poorly written lead-in. So it's sort of the same stem, changed a little bit, as that she wants to confirm that her students learn the components of a multiple-choice question And then the lead-in says, which of the following assessment types is best for testing memory of the specific components of a multiple-choice question? Again, you don't need the vignette. You can put your hands over the vignette and just read the lead in question. So that's poorly written. To continue on this journey of poorly written questions. These distractors are actually also not the best. And the reason why is, they're kind of vague. One of them is very, very long and that may sort of move people towards the right answer because D is the longest answer. Even if they don't know that an essay could be a good way to understand the components of a multiple choice question. Because in theory I could ask you to write an essay about, why do you need to have good distractors or what would make a good distractor. So many of these other options could fit. So as I mentioned I actually write questions for the National Board of Medical Examiners. And they have a really helpful resource online that's a complete item writing manual that talks about many of the things we've talked about today. But if you can write a really good multiple choice question, you actually can assess learners in a really efficient and cost effective. and pretty reliable and valid way. Alright, see you again soon. Thanks.
