Introduction
Your task is to rate the degree of semantic relatedness between two uses of a word. For instance, presented with a sentence pair as in (1), you are asked to rate the semantic relatedness between the two uses of grasp in (1a) and (1b).
-
- He continued to grasp, between forefinger and thumb, the edge of the cloth I had been sewing.
- For just a moment he didn’t grasp the import of what the old man had said.
Preparation
After registering on DUREL, you must complete the tutorial. Once you have passed the tutorial, you can annotate existing projects or upload your own project. The judgment in the tutorial work analogously to the later annotations.
Task structure
In regular annotation, you select the word for which you want to annotate sentences in the main menu. By clicking the 'Start annotation' button, you will get two sentences displayed next to each other, as you can see in Example A. The target word is marked in bold in the respective sentence. Your task is to evaluate, for each of these pairs of sentences, how strong the semantic relatedness is between the two uses of the target word in the two sentences.
Because language is often ambiguous, it is important that you first read each sentence in a sentence pair individually and decide on the most plausible meaning of the target word BEFORE comparing the two uses of the word. In some cases, the sentences already provide enough information to understand the meaning of the target word; however, for cases that are unclear, you can find additional context beyond that in gray.
The judgment scale
The scale that you will be using for your judgment ranges from 1 (the two uses of the word have completely unrelated meanings) to 4 (the two uses of the word have identical meanings). This four-point scale is shown in Table 1.
- 4. Identical
- 3. Closely related
- 2. Distantly related
- 1. Unrelated
- - Cannot decide
Table 1: Four-level scale of semantic relatedness.
Please try to ignore differences between the uses that do not impact their meaning. For example, eat and ate can express the same meaning, even though one is in present tense, and the other is in past tense. Also, distinctions between singular and plural (as in carrot vs. carrots) are typically irrelevant for the meaning.
Note that there are no right or wrong answers in this task, so please provide your subjective opinion. However, please try to be consistent in your judgment.
Annotation Examples
We now consider some evaluation examples to illustrate the different degrees of semantic relatedness you might encounter in annotation. Please note, as mentioned above, that these are only examples, and you should always give your subjective opinion.
The two instances of eat in Example A are judged identical in meaning (rating: 4), because both uses refer to the physical act of consuming food.
In contrast, the two uses of child in Example B are judged closely related but not identical (rating: 3), because the meaning of child(ren) in sentence 1 may be paraphrased as ‘young person’, while the meaning in sentence 2 is closer to ‘offspring’.
Annotation
Sentence 1
Speaking of bread and butter reminds me that we'd better eat ours before the coffee get quite cold.
Sentence 2
When the meal was over and they had finished their tea after they ate, Wang the Second took the trusty man to his elder brother's gate.
Example A: Judgment 4 (Identical).
Annotation
Sentence 1
He agreed and began practicing his sleightofhand tricks to the great pleasure of some children, the same ones, I suspect, who had plagued me when I was a child.
Sentence 2
The daylight had long faded; her child lay camly sleeping by her side; a candle was burning dimly on the stand.
Example B: Judgment 3 (Closely related).
In Example C, the two uses of the word crossroad art related, but more distantly (rating: 2): Unlike the child example above, the two uses of crossroad in this example have different meanings, which are yet related by a figurative similarity, on the level that both involve some kind of decision.
Annotation
Sentence 1
He came to a crossroad and read the signs; to the south, Kenniston, 20 m.
Sentence 2
As a result we are at a crossroad; either school integration efforts will be abandoned in the South, or they will be pursued in the Nort as well.
Example C: Judgment 2 (Distantly related).
A rating of 1 is used for two uses of a word that are completely unrelated in their meaning, as it is the case for bank in Example D. Note that this pair of uses is semantically more distant than the two uses of crossroad above. River banks and financial banks are not semantically related to each other.
Annotation
Sentence 1
His parents had left a lot of money in the bank and now it was all Measle's, but a judge had said that Measle was too young to get it.
Sentence 2
Sherrell, it is said, was sitting on the bank of the river close by, and as soon as the men had disappeared from sight he jumped on board the schooner.
Example D: Judgment 1 (Unrelated).
Finally, you have the option to annotate ”Can't decide”. Please use this option only if absolutely necessary, i.e., if you cannot make a decision about the degree of semantic relatedness between the two words marked in bold. This may be the case, for example, if you find a sentence too flawed to understand, the use of the target word is ambiguous, or the two uses of the target word do not match (i.e., do not have the same lemma).
Historical language data
The sentences provided for the annotation task were gathered from historical corpora. Sentences may occur more than once in the table.
As language changes over time, words might be used differently from what you are familiar with. If you are unsure about the meaning of a word or construction in a sentence, try to infer it from the meaning of the context.
The sentences may be very short or very long and some may seem ungrammatical. Also, words may be spelled in a different way than you are used to.
Try to ignore these issues; focus only on the meaning of the target words in their contexts. If you find that a sentence is too flawed to understand it, or the meaning of the target word is ambiguous, or the two instances of the target word do not match (i.e., they do not have the same lemma), please provide a comment to this effect.
Questions?
If you have any questions, please do not hesitate to ask them to durel@ims.uni-stuttgart.de.