Analyzer of Question Answering
This analyzer assesses the ability of search engines to find answers to question queries. The question queries include obvious quesitons with a question word ([when did CSKA win UEFA cup], [where is Lhasa located]) as well as other queries that imply an obvious short answer ([currency of Brasil], [nitric acid formula]).
The users typing such queries are most likely looking for the answer to their question. The quicker they get the answer, the better. Ideally, the snippet of the first page in search results should include the answer.
This analyzer has several tabs. In each of the tabs the total score of a search engine is the sum of its scores for each query.
1. Answer position in top 10 snippets
Here a search engine gets a score from 0 to 1 for each query, reflecting the highest position of a snippet, that contained the answer, in top 10 results. For example, if the answer was found in the first snippet, the search engine gets 1 for the query. If it was found in the second place, the score is 0.9 etc. If the answer was not found in the snippets of top 10 results, the score is 0.
2. Answer presence in snippets
Here a search engine gets 1 for a query if at least one of the top 10 snippets contained an answer to the question. 0 is assigned otherwise.
3. Position of answer web pages in top 10
A search engine gets a score from 0 to 1 for each query, reflecting the highest position of a web page, that contained the answer, in top 10 results. The score is 1 if the first page found contained an answer, 0.9 if the answer was present on the second page and 0 if none of the web pages found contained the answer.
4. Answer presence on web pages found
Here a search engine gets 1 for a query if at least one of the top 10 web pages contained an answer to the question. 0 is assigned otherwise.
A question query may have multiple possible answers. For example, the possible answers for [Where are Maldives located] are "Indian ocean" and "Asia".
 90−100%- 80−90%
- 60−80%
- 40−60%
- 20−40%
- 0−20%
|
|
