Home > News > The pitfalls in validating an examination

The pitfalls in validating an examination

and tips on how to avoid them.

The validation of examination is a topic that often comes up in quality assurance systems, quality marks and legislation. Sometimes it is also laid down who does this and what must be done during this validation. It therefore appears to be a formality. Why, apart from the rules, validate an examination? And can it not just be done more quickly and more cheaply? We provide a number of practical tips for validating an examination and discuss the pitfalls involved. Don’t fall for them and read on!

What is validation?

Validating an examination is the step between developing and conducting the examination. This involves assessing and registering whether an examination or examination question meets the quality requirements.

When a developer or development group has done their work, the examination is often already checked from a technical point of view. So what is the use of validation? Apart from the established rules, it is a message to the outside world that exams are checked before they are used. “There is always a control step” and “the examination is always seen by more than one pair of eyes” are frequently heard statements.

In addition to perception, validating an examination also has a number of formal characteristics:

It serves as an official approval by the examination board or a branch organisation.
It is the confirmation that an examination measures what it is supposed to measure, so it is relevant to the examination requirements and therefore to professional practice.
Validation prevents errors both in testing and content.

Validating an examination is therefore an important part of the examination development process. But it also comes with its own specific snags.

In the absence of specialists, don’t take the easy way

We all know, of course, that a validator is not the same person as the developer, after all, you don’t mark your own homework. But what do you do in the absence of specialists? The higher and more specialised the level, the more difficult it is to find enthusiasts who can do this work well and who do not charge a top rate for it. An alternative is to have developers validate each other’s work, or better still: consider a higher budget or other financial motives. Perhaps validators would like to enter into pleasant cooperation, expand their network, develop on a personal level or gain a bond with your institute.

Do not draw negative conclusions about the quality of the developer too quickly

The temptation is strong to have doubts about the developer’s knowledge after part of the examination has been rejected. But is that justified?

Be aware that developing an examination requires a different thought process than validating it. The validator has the challenge and the ‘luxury’ of being able to look from a distance, while the developer is in the middle of the process and by definition can look less objectively. In addition, developers still have room to grow in their profession. Conclusion: Provide feedback on the findings and give recommendations for improvement. In this way we can all become a little better and we continue to develop ourselves.

“Let’s validate by email, that is a lot more efficient”

Let’s be honest, who has never thought this after a long validation meeting?

Validating by email, however, often leads to the following problems:

In the absence of discussion, the validator is less sharp;
In the event of a difference of opinion, it is no longer clear how the decision-making process is going and whether it is still balanced. On the contrary, this can lead to a longer process. How, then, will the decision be made in the end?

Changes? Be alert!

It’s great when a validator comes up with immediate improvements to the questions submitted. Especially when the development is under time pressure. This considerably reduces the percentage of rejected questions. If the direction of the question changes too much, a new question may have to be developed without a separate validation. The advantage of viewing the questions objectively as validator then lapses. This makes mistakes more likely. So be alert.

Does the examination meet the examination requirements?

Nothing has such major consequences as examinations that do not meet the examination requirements. If the level is too high or if it contains issues that do not meet the examination requirements, students and other interested parties will lose trust. They no longer consider the examination to be an honest or valid measurement. Focus not only on the technical and substantive details of the examination, but also on the examination as a whole. In the case of creating an item bank, which has been developed per item and not per examination: check the examination by means of a sample draw. Is the examination relevant and is it correct in terms of ‘look and feel’?

Conclusion

The importance given to validation in the creation of examinations is justified. In order not to fall into the ‘pitfalls’, we give you the following practical tips:

Separate development and validation; after all, you don’t check your own homework.
Do not draw negative conclusions about the quality of the developer too quickly. Validation is simply something else than development.
Validation by email may seem advisable, but is not recommended.
Make sure that the examination questions do not change direction too much. Otherwise, a new question will have to be developed, without separate validation.
Don’t just focus on the technical and substantive details of the examination, but assess it as a whole.

The pitfalls in validating an examination

What is validation?

In the absence of specialists, don’t take the easy way

Do not draw negative conclusions about the quality of the developer too quickly

“Let’s validate by email, that is a lot more efficient”

Changes? Be alert!

Does the examination meet the examination requirements?

Conclusion

Would you like to know more?

Products

Services

About us