Evaluating Semantic Search Tools using the SEALS platform

Evaluating Semantic Search Tools using the SEALS platform

inproceedings
Abraham Bernstein, Stuart N. Wrigley, Khadija Elbedweihy, Dorothee Reinhard, Fabio Ciravegna
In common with many state of the art semantic technologies, there is a lack of comprehensive, established evaluation mechanisms for semantic search tools. In this paper, we describe a new evaluation and benchmarking approach for semantic search tools using the infrastructure under development within the SEALS initiative. To our knowledge, it is the first effort to present a comprehensive evaluation methodology for semantic search tools. The paper describes the evaluation methodology including our two-phase approach in which tools are evaluated both in a fully automated fashion as well as within a user-based study. We also present and discuss preliminary results from the first SEALS evaluation campaign together with a discussion of some of the key findings.
Evaluating Semantic Search Tools using the SEALS platform
2010
Proceedings of the International Workshop on Evaluation of Semantic Technologies (IWEST 2010) Workshop
ddis