How do you create a good test?

Five pitfalls to bear in mind

Tests are carried out on a large scale: in public and private education, in company training programmes, by examination institutes and by professional associations. But is this always done properly? Who has not experienced that unpleasant feeling after an exam? Giving rise to remarks such as “irrelevant details were asked” or “It was unclear what was meant by some questions”. It is easy to step into a pitfall. In this blog we highlight five things to avoid when it comes to developing tests. Keep them in mind and create a good test!

1. The test is too short
2. The test focuses on ‘peripheral issues’
3. There is discussion about the answer
4. The scoring is unfair or not clear enough
5. The test contains too much text
6. Conclusion

1. The test is too short

If a test is (too) short, you cannot demonstrate your knowledge and insights well (enough). The score you obtain on a test which is too short may, moreover, be the result of chance. If you happen to get questions about the part you master well, you will have a good result. But it could just as well be the other way round. Thus, a test that is too short has low reliability. Moreover, a test that is too short is usually not valid either. After all, on the basis of too few questions you cannot see to what extent someone has actually mastered the subject matter, because you simply do not measure what you intend to measure.

The size of a test strongly depends on the types of questions used, the purpose of the test (formative or summative), the value attached to the test, the scope of the subject matter and the available examination time. In general, the more questions the more reliable the test.

2. The test focuses on ‘peripheral issues’

What is actually being asked and why does one question fit into the test and the other not? Always look at the learning objectives which form the basis of the test and ask questions about the core of those learning objectives. If, for example, you want to test whether candidates have insight into the different phases which go through the construction of a house, it makes no sense to ask a question about the properties of tropical hardwood. Or worse: asking when tropical hardwood was first used in construction.

How do you avoid such ‘off topic’ questions? Have test questions validated by a subject expert other than the test developer. This validator looks at the questions with a fresh view and is therefore well able to filter out irrelevant questions.

3. There is discussion about the answer

Regardless of how good or compelling a question is, something is wrong if there is discussion among experts about the right answer. Always carry out a check on:

• Is the question specific enough? If the question is phrased too broadly, it is not clear which answer is meant. Often, more than one answer is correct.
• Is there agreement in practice on the standards to be tested? When questions are asked about issues that have not yet crystallised in the sector, disagreement can arise about what constitutes a good or bad answer. This causes problems in tests. For example: a question about how a mediator should act in a certain situation. As long as it is not against the professional code, a way of acting is not immediately wrong.
• Is a question unambiguous? Or do different (experienced) people interpret the question differently?

The same rule applies to this fallacy: it must be validated by one or more subject experts in order to prevent a discussion after the question has been answered.

4. The scoring is unfair or not clear enough

Sometimes you see beautiful exam assignments, which go to the heart of the learning objective or even of the subject and where several answers are expected from the candidate. How do you deal with the scoring in such a case? Suppose ten items have to be completed, do you get ten points? That is fair, but then this part weighs relatively heavily. And what if you award three points, when do you get one, two or three points?

Or: You can get ten points for each question, even if there is only one answer to be given. This can then lead to differences in the awarding of points between assessors because everyone is acting from their own perspective.

In short, by leaving room in the answer model, the assessment is often not entirely accurate. Therefore, specify in the model which answer will receive how many points. Clear guidelines will help keep the scoring as objective and fair as possible.

5. The test contains too much text

The test developer assumes, sometimes wrongly, that candidates ‘just have to read well’. Be careful not to test reading comprehension, which is usually not a learning goal. Ask the questions in as simple language as possible. Do not use narrative cases to make the test more fun. A test does not have to be fun. Always check a question for unnecessary information.

Conclusion – How do you create a good test?

Keep these five common pitfalls in mind when developing your test. However, there is a lot more to putting together a good quality test. If you want to know more about this subject or about the quality of tests in general, leave your details here and receive our book ‘The Power of Digital Testing’.