How do you create a good test?

How do you create a good test?

Five pitfalls to keep in mind

Testing is widespread: in funded and private education, within corporate training programs, by examination institutes and by professional associations. But does that always go correctly? Who doesn’t know the unpleasant feeling that can creep up on you after an exam. Commonly heard comments “They asked for nonsensical details” or “It was unclear what was meant by some of the questions.” Stepping into a trap is easy. In this blog, we highlight five pitfalls when it comes to assessment development. Keep them in mind and create a good key!

1. The assessment is too short;
2. The assessment is about “insignificant details”.
3. There is discussion about the answer;
4. The scoring is unfair or insufficiently clear;
5. The assessment contains too much text.

The assessment is too short

If a assessment is (too) short, you can’t properly demonstrate (enough) what knowledge and understanding you have. Moreover, the score you get on a assessment that is too short may be due to chance. If you happen to get questions about the part you master well, you’ll have a great result. But it could just as easily turn out the other way around. Thereby, the reliability of a assessment that is too short is low. Moreover, an assessment that is too short is also usually not valid. After all, based on too small a number of questions, you cannot tell to what degree someone has actually mastered the material because you simply are not measuring what you intend to measure.

The size of a test strongly depends on the question types used, the purpose of the test(formative or summative), the value attached to the test, the scope of the subject matter and the available exam time. In general, the more questions the more reliable the assessment.

The assessment is about insignificant details

What is actually being asked and why does one question fit the test and another not? To do so, always look at the learning objectives underlying the test and ask questions about the core of those learning objectives. For example, if you want to test candidates’ understanding of the different stages involved in building a house, it makes no sense to ask a question about the properties of tropical hardwoods. Or worse, asking about when tropical hardwoods were first used in construction.

How do you avoid such “off topic” questions? Have test questions determineby a subject matter expert other than the assessment developer. This determiner looks at the questions with fresh eyes and is therefore good at filtering out irrelevant questions.

There is discussion about the answer

A question can be so catchy or good, but something is wrong if there is discussion among experts about the correct answer. Always perform a check on this:

• Is the question
  specifically
  enough? If the question is asked too broadly, it is not clear which answer is being referred to. Often several answers are then correct.
• In practice, is there
  agreement
  on standards to be questioned? With questions about issues that are not yet crystallized in the industry, there can be disagreement about what is a good or bad answer. In tests, that presents problems. For example: a question about how a mediator should act in a particular situation. As long as it is not against the professional code, a course of action is not immediately wrong.
• Is a question unambiguous? Or do different (experienced) people interpret the question differently?

Also true for this trick question: determination by one or more subject matter experts is necessary to avoid discussion after sampling.

The scoring is unfair or insufficiently clear

You sometimes see beautiful exam assignments that get to the heart of the learning objective or even the subject and where candidate are expected to give multiple answers. How do you handle the scoring in such a case? Suppose ten items have to be completed, do you get ten points? That is fair, but then this part weighs relatively heavily. And what if you award three points, when do you get one, two or three points?

Or: You can get 10 points for each question, even if only one answer is required. This can then lead to differences in point assignments between evaluators as each person acts from their own perspective.

In short, by leaving room in the answer model, assessment often does not proceed with complete purity. Therefore, in the answer template, record for which answer how many points are awarded. By having clear guidelines, you keep the scoring as objective and
fair
possible.

The assessment contains too much text

The assessment developer assumes, sometimes erroneously, that candidates should “just read well. Be careful not to test reading comprehension, which is generally not a learning objective. Set the questions in so
simple
possible language. Do not use narrative case studies to make the assessment more fun. An assessment doesn’t have to be fun. Therefore, always check a question for redundant information.

Conclusion

Keep these five common pitfalls in mind as you develop your assessment. However, there is much more to putting together a quality assessment. If you want to learn more about this topic or about assessment quality in general, visit our downloads page.