In our system, we provide a learning tool in the form of tutorials. Each tutorial encompasses a batch of sentence pairs, which have been manually selected for a particular language.
The main objective of the tutorial is to evaluate the annotator's understanding of the annotation scale, as well as their annotation proficiency. This is achieved by comparing the user's own annotations to a set of gold standard annotations, and computing a Spearman correlation coefficient to measure the degree of alignment. If the coefficient surpasses a certain threshold (currently 0.6), implying a strong agreement, DURel designates the tutorial as duly completed by the user and updates their records to reflect this achievement. Annotators that have completed a study gain access to the annotation features of DURel. If the computed correlation is weak (coefficient falls below the threshold limit), the tutorial is not deemed as completed. In this case, the annotator will not gain access to the annotation, but they can retry to complete the tutorial.