0 comments on "Designing Effective Multiple Choice Assessments"

Designing Effective Multiple Choice Assessments

Audience:

  • Instructor

Overview

Multiple choice assessments are a convenient assessment tool for large enrollment courses, survey courses, and introductory courses. Many instructors consider them a necessary, but flawed assessment technique. However, by incorporating some of the ideas presented here, instructors may be able to work towards a more authentic assessment of student knowledge.

General Strategies [ref]Designing Multiple-Choice Questions. (n.d.). Retrieved from https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions [/ref][ref]Designing Quality Multiple Choice Questions. (n.d.). Retrieved from https://poorvucenter.yale.edu/MultipleChoiceQuestions[/ref]

There are two parts to every multiple choice question, a stem and possible answers. The stem identifies the problem or question. The answer options consist of a correct answer and distractors. Answer options that are not correct should all be distractors; avoid one mostly correct answer and a slightly more correct answer.

Well-written stems are clear, succinct, and avoid negatives and exceptions. You should be able to answer the stem without looking at the answer options.

Well-written distractors target common student misconceptions, are clear and succinct, are free from hints to rule them out, and do not include inclusive or exclusive catch-alls like “all of the above” or “none of the above.”

Recommendations [ref]Designing Multiple-Choice Questions. (n.d.). Retrieved from https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions [/ref][ref]Designing Quality Multiple Choice Questions. (n.d.). Retrieved from https://poorvucenter.yale.edu/MultipleChoiceQuestions[/ref] [ref]Senzaki, S. (2018, January). Using Multiple-Choice Assessments Well. UWGB Center for the Advancement of Teaching and Learning presentation.[/ref]

  • Align questions with course and unit learning objectives
  • When creating questions, be aware of potential accessibility issues. Do images have alt-text? Is an image required to answer the question? Is a sound clip required to answer the question? Etc.
  • Practice makes perfect. Provide students with opportunities to take low/no-stakes practice quizzes or exams. This will increase their access to the material as well as your particular question authorship style.
  • Consider different taxonomies of learning when writing questions (Dee Fink, Benjamin Bloom)
  • Ask students to select the best answer, since distractors may include correct information not necessarily related to the question
  • Avoid trick questions; you’re testing knowledge
  • Use a full statement in the question​
  • Write the correct response before other options
  • Limit the number of responses (3-5)​
  • Avoid negative wording​
  • Avoid ”all (or none) of the above”​
  • Make the choices grammatically consistent with the stem​
  • 5% of questions very easy (90% accuracy)
  • 5% of questions difficult (10% accuracy)
  • Other questions: average of 50% accuracy​

Feedback is Important [ref]Roediger III, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in cognitive sciences, 15(1), 20-27. Retrieved from https://www.sciencedirect.com/science/article/pii/S1364661310002081 [/ref][ref]Kleeman, J. (2010, November). Multiple choice quizzes help learning, especially with feedback. Retrieved March 4, 2019, from https://blog.questionmark.com/multiple-choice-tests-help-learning-especially-with-feedback[/ref]

Whether you choose to provide feedback during class time, through individual comments, or automatically through the learning management system (i.e. D2L and Canvas), research shows that feedback improves long-term knowledge retention. Roediger and Butler found that any feedback can have a positive impact on retention, but delayed (not immediate) feedback is most effective.

graph of testing and feedback impact on knowledge retention
Roediger III, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in cognitive sciences, 15(1), 23. Retrieved from https://www.sciencedirect.com/science/article/pii/S1364661310002081

Choices in Multiple Choice Assessment Modalities [ref]Senzaki, S. (2018, January). Using Multiple-Choice Assessments Well. UWGB Center for the Advancement of Teaching and Learning presentation.[/ref]

Scantron vs. learning management system, i.e. D2L/Canvas

Learning management system pros:

  • Students can take it outside of the classroom​
  • Can be a better environment than some classrooms​
  • Less stressful for the students​
  • Frees up a class time​
  • Better feedback​
  • Paper free, no cost​
  • Randomization (different blocks)​

Scantron pros:

  • Detailed item analysis
  • Discrimination index
    • Ranges -1.0 to +1.0
    • .30 or higher, better
    • 50+ respondents

Who to contact for help:

  • CATL
  • References

 

Comments are closed, but trackbacks and pingbacks are open.