In the early part of my dissertation work, I was interested in comparing student performance on concept inventories. How things have changed!

Matt Kohlmyer, Michael Schatz and I wrote a paper comparing introductory E&M curricula using the Brief E&M Assessment (BEMA) in the early part of my career. You can find it here. If you want to review the BEMA, there’s a nice article here. Contact me directly if you’d like a copy of the BEMA.

In that article, we found that students taking Matter and Interactions (M&I) E&M performed better overall and on individual topics than students taking a pedagogically reformed, but traditionally sequenced E&M course at several different institutions. This led us to conclude that students taking Matter and Interactions were better prepared in E&M (as measured by the BEMA). The BEMA was specifically designed as the lowest common denominator concept inventory for E&M; hence, our conclusions were fair and justified.

In a new article that we have submitted to AJP, we performed a very similar comparison between mechanics courses: Matter and Interactions mechanics and the equivalent traditionally sequenced course. In this work, we used the Force Concept Inventory (FCI). Why? Because we had anecdotal evidence that Matter and Interactions students underperform on the FCI. This article is likely to be published in AJP which is not an open-access journal. I’m posting the preprint of the article in this post. But, it will eventually be on the arXiv.

Preprint of FCI paper

We found that students in M&I mechanics do underperform compared to students taking a traditionally sequenced course. However, in this article we did not conclude that traditional students are better prepared in mechanics. Why?

The FCI was not designed to compare curricula (or pedagogy, although it’s often used for this purpose). Moreover, the FCI is not aligned with the content and goals of the M&I course. One of the first things I learned about educational reform is the alignment of instruction and assessment. Clearly, the instruction in this case is not aligned with the FCI. In fact, the creators of M&I have clearly stated that this curriculum has fundamentally different goals (e.g., a 20th century introduction to mechanics) from a traditional course (e.g., 17th-18th century). We attempted to communicate these ideas in an earlier draft of the article.

When the preprint (and my thesis work) was originally picked up by the blog-o-sphere, our message was lost. Mark Guzdial, a member of of my thesis committee and eminent CS-ed blogger, concluded that “Computation + X doesn’t necessarily mean better learning in X“. While this might be true, it’s a terribly hard thing to measure. Nor do I believe my work in this area can make that conclusion. The whole course, not just the Python programming, is under consideration. Chad Orzel picked up an interesting thread, “practice matters.” That is quite true, but what is also important is the choice of evaluation instrument. To be fair, Chad makes a number of interesting remarks concerning the purpose of the introductory course. Is it to improve FCI scores? I think not.

In the above version of the paper, we have carefully laid out our message: concept inventories (in general) are limited. Concept inventories that are not designed with the content and goals of the course in question in mind are useful tools, but do not provide a complete picture of the course or what the students have learned. It’s very important that the evaluation match the instruction, otherwise, what are you measuring? Moreover, concept inventories are only able to compare students on concepts, methods, and tools which curricula both treat with equal intensity (like our BEMA work). Otherwise, there are caveats on the results.

Concept inventories also miss what are likely the most interesting aspect of curricular reform, the effect on new content, goals, and methods. How do we value these new effects? How are they weighed? Can we measure this? Such questions are important for those considering new ideas in their courses. And these are not easy questions are to answer. Nor are the answers fixed, but dynamically changing as we start to understand the purpose of our introductory courses.

In light of the underperformance by M&I students, is anything changing? Sure. My Georgia Tech collaborators are trying work within the curriculum to shore up conceptual difficulties. But, major changes are not planned. This is because we value the new concepts, methods, and tools of the M&I curriculum over a single measure of student performance.