The pitfalls of setting an examination

and tips on how to avoid falling for it.  

The setting of examinations is a topic that often recurs in examination regulations, quality assurance systems, quality marks and legislation. Sometimes, it is also specified who does this and what needs to be done during this assessment. It therefore appears to be a mere formality. Why, apart from the rules, should an examination be assessed? And can't it be done a little faster and cheaper? We provide a number of practical tips for assessing an examination and discuss the pitfalls. Don't fall into them and read on!

What is determination?

Setting an examination is the step between construction and administration. This involves assessing and recording whether an examination or examination question meets the quality requirements.

When a developer or construction group Once they have done their work, the examination has often already been checked for technical accuracy. So what is the point of verification? Apart from the established rules, it sends a message to the outside world that examinations are checked before they are used.“There is always a verification step” and “the examination is always reviewed by more than one pair of eyes” are frequently heard statements.

In addition to perception, setting an examination also has a number of formal characteristics:

• It is an official agreement of the examination board or a trade association;
• It is the validation that an examination measures what it is supposed to measure, so that it is relevant to the examination requirements and therefore to professional practice;
• Verification prevents technical and content-related errors.

Setting an examination is therefore an important part of the examination development process. However, there are also some snags.

In the absence of specialists, do not take the easy route.

We all know, of course, that an assessor is not the same person as the designer; after all, a butcher does not inspect his own meat. But what do you do when there is a shortage of specialists? The higher and more specialised the level, the more difficult it is to find enthusiasts who are good at this work and who do not charge the usual top rates. An alternative is to have designers assess each other's work, or better yet, look into budget increases or other financial incentives. Perhaps assessors would like to enter into a pleasant collaboration, expand their network, develop themselves on a personal level, or form a bond with your institution.

Do not jump to negative conclusions about the quality of the constructor.

After part of the examination has been rejected, it is tempting to question the designer's knowledge. But is that justified?

Be aware that developing an examination requires a different thought process than setting it. The examiner has the challenge and the ‘luxury’ of being able to view things from a distance, while the developer is in the middle of the process and, by definition, cannot view things as objectively. In addition, constructors still need to grow in terms of testing techniques.

Conclusion: Share your findings and give advice on how to improve. That way, we can all get a bit better and keep developing ourselves.

‘We will confirm by email, that is much more efficient.‘

Be honest, who has never thought this after a long assessment meeting?

However, confirming by email often leads to the following problems:

• The assessor is less critical due to a lack of discussion.
• In the event of a disagreement, it is no longer clear how the decision-making process will proceed and whether it will still be balanced. This may actually result in a more protracted process. How will the final decision be made?

Adjustment? Be alert!

It is helpful if an assessor makes improvements to the submitted questions on the spot, especially when development is under time pressure. This significantly reduces the percentage of rejected questions. If the direction of the question changes too much, a new question may need to be composed without separate assessment. This negates the advantage of viewing the questions objectively as an assessor. This can lead to mistakes more easily. So be alert.

Does the examination meet the examination requirements?

Nothing has such a significant impact as examinations that do not meet the examination requirements. If the level is too high or if there are elements that do not belong in the examination requirements, the confidence of students and other stakeholders will decline. They will no longer consider the examination to be a fair or valid assessment. Do not focus solely on the technical and content-related details, but also on the examination as a whole. When setting up an item bank, developed per item rather than per examination: Check the examination by means of a random sample. Is the examination relevant and does it have the right look and feel?

Conclusion

The role that assessment plays in the creation of examinations is certainly justified. To avoid falling into the ‘pitfalls’, we offer the following practical tips:

• Separate construction and determination; a butcher does not inspect his own meat;
• Do not jump to negative conclusions about the quality of the constructor. Determining is simply different from constructing;
• Confirmation by email seems advisable, but is not recommended;
• Ensure that the examination questions do not change direction too much. Otherwise, a new question must be developed without separate determination.;
• Do not focus solely on the technical and content-related details of the examination, but assess it as a whole.

Does the above resonate with you, or would you like to discuss the structure of the examination process? Please leave your details and we will be happy to tell you more.