Five pitfalls to bear in mind
Tests are carried out on a large scale: in public and private education, in company training programmes, by examination institutes and by professional associations. But is this always done properly? Who has not experienced that unpleasant feeling after an exam? Giving rise to remarks such as “irrelevant details were asked” or “It was unclear what was meant by some questions”. It is easy to step into a pitfall. In this blog we highlight five things to avoid when it comes to developing tests. Keep them in mind and create a good test!
If a test is (too) short, you cannot demonstrate your knowledge and insights well (enough). The score you obtain on a test which is too short may, moreover, be the result of chance. If you happen to get questions about the part you master well, you will have a good result. But it could just as well be the other way round. Thus, a test that is too short has low reliability. Moreover, a test that is too short is usually not valid either. After all, on the basis of too few questions you cannot see to what extent someone has actually mastered the subject matter, because you simply do not measure what you intend to measure.
The size of a test strongly depends on the types of questions used, the purpose of the test (formative or summative), the value attached to the test, the scope of the subject matter and the available examination time. In general, the more questions the more reliable the test.
What is actually being asked and why does one question fit into the test and the other not? Always look at the learning objectives which form the basis of the test and ask questions about the core of those learning objectives. If, for example, you want to test whether candidates have insight into the different phases which go through the construction of a house, it makes no sense to ask a question about the properties of tropical hardwood. Or worse: asking when tropical hardwood was first used in construction.
How do you avoid such ‘off topic’ questions? Have test questions validated by a subject expert other than the test developer. This validator looks at the questions with a fresh view and is therefore well able to filter out irrelevant questions.
Regardless of how good or compelling a question is, something is wrong if there is discussion among experts about the right answer. Always carry out a check on:
The same rule applies to this fallacy: it must be validated by one or more subject experts in order to prevent a discussion after the question has been answered.
Sometimes you see beautiful exam assignments, which go to the heart of the learning objective or even of the subject and where several answers are expected from the candidate. How do you deal with the scoring in such a case? Suppose ten items have to be completed, do you get ten points? That is fair, but then this part weighs relatively heavily. And what if you award three points, when do you get one, two or three points?
Or: You can get ten points for each question, even if there is only one answer to be given. This can then lead to differences in the awarding of points between assessors because everyone is acting from their own perspective.
In short, by leaving room in the answer model, the assessment is often not entirely accurate. Therefore, specify in the model which answer will receive how many points. Clear guidelines will help keep the scoring as objective and fair as possible.
The test developer assumes, sometimes wrongly, that candidates ‘just have to read well’. Be careful not to test reading comprehension, which is usually not a learning goal. Ask the questions in as simple language as possible. Do not use narrative cases to make the test more fun. A test does not have to be fun. Always check a question for unnecessary information.
Keep these five common pitfalls in mind when developing your test. However, there is a lot more to putting together a good quality test. If you want to know more about this subject or about the quality of tests in general, leave your details here and receive our book ‘The Power of Digital Testing’.