EDAC and The Item Writing Workshop
As you know, CHD is developing an exam to assess individuals’ abilty to employ evidence-based design in their projects. On August 13 and 14, volunteers convened at our office to begin the process of writing examination questions. We were all surprised to learn about all the mechanics that are involved.
Our group was fearlessly led by psychometrician Robert Shaw of Applied Measurement Professionals (AMP). According to Rob, each test item should only measure one ability. By doing this, you have the best chance of measuring a candidate’s knowledge of one situation.
For half the day, Rob explained to us the process of item writing. For example, we learned that when writing the “wrong” answers, called distractors, these have to be neutral and must not cue the candidate to any one choice.
By the end of day one, I think all of us had a pretty good understanding of the item writing process. However, actually developing our own items was quite a different challenge. After Rob let us each select our assignments, everyone started writing. For most, the first item we each wrote took over an hour. I started to wonder if we were going to complete the required 300 items by the end of day two!
Well, we didn’t. We finished about 70 items. However, considering how difficult the task is, I am still extremely impressed with this number. Not only did the group write all these items, but they also started to review a few of them. This turned out to be a huge process in itself.
Because the group needs to unanimously support the terminology used to express an item as well as the available multiple choice answers, reviewing one item could take up to 30 minutes. (And this is only because Rob made us stop discussing the item.) There is no doubt that our wonderful group of EDAC volunteers is dedicated to meticulously approving the right content.
In the next few weeks, volunteers will continue to write examination items. Our goal is to produce at least 300 items by Labor Day. In September, AMP and CHD volunteers will re-convene to edit and approve many of these items. Concurrently, we are working on creating study materials to ensure that we are educating candidates to appropriately prepare them for the exam. If anything, learning the item writing process has renewed my appreciation for all of the tests I’ve taken throughout the years.