Semantic annotation, the process of iden tifying key phrases in texts and linking them to concepts in a knowledge base, is an important basis for seman tic information retrieval and the semantic web uptake. Despite the emergence of semantic annotati on systems, very few comparative studies have been published on their performance. In this paper, we provide an evaluation of the performance of existing system s over three tasks: full semantic annotation, named entity recognition, and ke yword detection. More specifi cally, the spotting capability (recognition of relevant surface forms in text) is evaluated for all three tasks, whereas the disambiguation (correctly associating an entity from Wikipedia or DBpedia to the spotted surface forms) is evaluated only for the first two tasks. We use logistic regression to identify significant performance differences. Although some of the annot ators are specifically targeted at some task (NE, SA, KW), our results show that they do not necessarily obtain the best performance on those tasks. In fact, systems identified as full semantic annotators beat all other systems on all data sets. We also show that there is still much room for improvement for the identification of the most relevant entities described in a text

Keywords