Exam Statistics: Discrimination Factor

Question

Hello -

We are in the process of implementing a resource for our faculty to dig deep into student MCQ exam performance to evaluate the effectiveness of our questions on high stakes exams. We are currently using Classic Quizzes due to the inability to download statistics from New Quizzes in a usable format.

I noticed that the Discrimination index listed in Canvas quizzes is not accurate - it is displaying the point biserial correlation. Is this an anomaly on my end, or is this true for everyone? I'm confused because the actual calculation for the discrimination index is listed correctly in the instructor guide - but I do not see that it is even reported in the item analysis report?

It would be helpful to know the underlying calculations for each metric to understand what I can trust/not trust from the automatically reported data set.

Thanks for any advice - we are trying to adhere to best practices for assessment to increase the validity of our internal exam scores and correlation with the certification exam that they will take after graduation.

ErinRumpke · Answer

For anyone who needs to know - I worked with someone through the Canvas help desk to get this question answered - I confirmed that Canvas is using the point biserial instead of the discrimination index, despite the discrimination index being defined differently in the instructor guide. I was told that they do not plan to fix the inaccurate info in the instructor guide because classic quizzes is being phased out. The person who was helping me did not know if the values reported by new quizzes were accurate and the report does not provide the raw data for you to calculate it yourself easily from the item analysis report, so you would need to do student analysis report to get the raw data and recalculate that way.  Hope this helps someone!

ErinRumpke · Answer

Thank you - I meant to ask about the Discrimination Index, not the difficulty index - my error! Nonetheless, the Discrimination Index shown is not correct. There is no discrimination index column in the item analysis report, and the exam results webpage is showing the point biserial correlation of the correct response option as the index. These seems strange because the canvas instructor guide defines the calculation for this measure correctly, it just doesn't seem to actually use it?

Canvas Instructor Guide for Exam Statistics

As an easy to look at example, you can compare the value in column S for the first question (Point biserial correct) with the Discrimination index that appears on the webpage in the image below. In this instance, the Discrimination Index as calculated using the formula is 0.75, while the point biserial is 0.54. Obviously, there isn't too many takers for this exam, but I see the same spread for my courses with larger numbers of students and when looking at multicohort data - because the calculations are using two different parts of the spread. Obviously, both are helpful when assessing the quality of a question, but since I'm asking my faculty to compare their stats to established/published best practice criteria I was hoping to understand if this is a choice canvas has made or if it's something just on my end?

Thanks for your help!

james_whalley · Answer

@ErinRumpke When I just ran the Item Analysis, I see both point biserial correlation (Columns S - Y) and Difficulty Index (Column Q). If you are not seeing that, can you share a copy of what you are getting from the Item Analysis?