What are the Search Quality Analyzers?
The quality of web search is important for everyone. If the search engines (SEs) do their job well, the users save time and find what they need fast.
But how do you assess search quality? One cannot rely on individual opinions here since every user has his/her own search habits and preferred query types. Google may work well for some users, but others will prefer say Yahoo or something else just because they usually search for different things in a different way.
The popularity of a SE does not directly reflect the search quality either, because the popularity is highly influenced by marketing and PR.
In order to independently assess the search quality, we developed a set of analyzers, one for each type of search queries. For all of these analyzers we use special sets of sample queries and sample sites. We measure the quality of navigational and informational search, the percentage of pornography among the pages found by a SE etc.
We hope that our tests are (or eventually will be) an objective and reliable source of information on search quality.
Enjoy.
How do the Analyzers work?
To estimate the search quality for various types of queries, we use special sets of test queries and analyze the pages returned for these queries. For example, here is how we test the quality of navigational searh that is the queries aimed at finding a particular web page. We use approximately 500 sample queries and specify the corresponding set of ‘test sites’ (the sites that would be good responses for these queries).
Thus if the user inputs "CNN" , (s)he probably wants to see www.cnn.com as the first result. Cnn.com is listed as an organic result for the query 'CNN'.
In order to prevent the Analyzers from being compromised by search engine developers, we use a different set of queries every day. We constantly replenish and refine the pool of queries from which each day’s set is randomly selected.
You can find a description of the methods used in a particular analyzer on the page where the analyzer data is shown.
We highly appreciate any corrections and welcome any criticism. Please feel free to send us the errors you find, suggest new sample queries, criticize the method etc.
|
|
Analyzer of nagivational search
|
|
|
Analyzer of correct hints
|
|
|
Typo resistance analyzer
Human are not machines, they make mistakes. This includes the mistakes while typing in a search query: a typo, next button pressed by accident ("quety" instead of "query"), a double character or a missed one ("qury" or "queery"), after all, the user can type the word 'by ear' not knowing the correct spelling ("yandax" instead of "yandex").
Show →
In this case, the search engine can adhere to one of the following strategies:
1) no processing: search with exact spelling only
2) recognize the typo, but still search for the entered query with an additional hint: "perhaps you were looking for [correct spelling]?"
3) recognize the typo and search for the correct spelling immediately
Depending on the chosen strategy, the user either remains unaware of the fact that (s)he is mistaken, or notices it and makes an extra click (up to the user), or gets the correct results without ever noticing his own mistake.
This analyzer compares the search results of the "correct query" and several forms of its possible mistypings. The similarity of results to those of a "correct" query is evaluated.
Apart from deliberate typo correction, matches can arise in four cases:
1) accidentally
3) the page contains both the correct and mistyped spelling
4) incorrect reaction of the engine's morphology (e.g., the unknown word "mushroomz" which is a typo of "mushrooms" is corrected to "mushroom")
5) promotion of the same websites both for correct and incorrect spelling of queries
All of these cases produce noise in this analyzer: an accidental match of results.
The similarity is evaluated in the same way as for the update analyzer but with a different set of queries.
The more matching results are registered, the higher is the index of the search engine for this analyzer. This determines the order of search engines in the informer of the analyzer.
In future, a rotation of query sets with typos from a wide array will be introduced.
Hide ↑
[ Link to article ]
|
|
|
Quotation search quality analyzer
|
|
|
Analyzer of 'adult sites' presence in the search results
|
|
|
Recall analyzer
The recall analyzer estimates the relative size of indices of the Internet search engines.
Show →
The total number of documents indexed, as reported by the SE itself cannot be used for comparison because different SEs have different methods of document count. For example, some of them include duplicate documents in the count while others do not. Counting the duplicate documents may double the reported index size or increase it even more.
Additionally, the size of the index is a very PR-sensitive issue as it is one of the very few simple notions in SE area easily understandable by journalists. This means that the bigger index database size you report, the better press you get.
The number of documents found for a particular query does not always reflect the real number of documents indexed by a SE. Almost every frequent query will return tens of thousands results in all search engines. But the user will never be allowed to see them all: the search session will be interrupted after browsing through first hundreds of pages. Thus the exact number of web pages found can be verified only in the case when the number of possible results is very small, that is for queries containing very rare words.
For multiple-word queries, certain SEs show in the search results not only the documents where all the words comprising the query are found, but also the documents containing single words from the query. These "tail" documents usually irrelevant to the query, but counting them can increase the total number of pages found.
In order to obtain independent and reliable data on the relative index size of the popular SEs, we developed a simple automatic method, based on a set of sample queries. We gathered a set of very rare words, all of which occur several tens of times on the web. Once a day, we count how many of these occurences are found by each search engine.
To make the data steadier, we use a different set of sample queries from the whole query pool every day.
The set of sample queries is constantly replenished by our linguists. If you have some rare words and want to help us cover the 'faraway' areas of the Net, please send us these words, and we will consider including them into the sample queries list.
Hide ↑
[ Link to article ]
|
|
|
Update analyzer
‘Update’ refers to the process of search results renewal. When the results are updated, some sites may make it to the top 10, some other sites may "sink". Every search engine has its own update style which becomes clear in this analyzer. Every day the search engine update analyzer monitors the top ten responses to 140 queries in order to assess the number of sites that changed their positions, and how much the positions have changed.
Show →
Let Di be the change in position for the page that appeared i-th in top 10 search results on day 1. For example, if the fifth page from the first day top 10 appeared third or seventh on the second day, D5=2. If the second day top 10 did not contain a certain page which was present on the first day, then we will assume that Di=10 for that page.
The update indicator is calculated using the formula:
10
∑ Di/100
i=1
Consider a couple of examples:
Example 1
On Day 1, a certain query has the following Top 10:
C1, C2, C3, C4, C5, C6, C7, C8, C9, C10.
On Day 2, the same query has this Top 10:
Cn, C1, C2, C3, C4, C5, C6, C7, C8, C9.
In this case the update indicator is calculated as follows:
((2-1)+(3-2)+(4-3)+(10-9)+10)/100 = 0.19 (19%)
Example 2
For Day 1, a certain query has the following Top 10:
C1, C2, C3, C4, C5, C6, C7, C8, C9, C10.
For Day 2, the same query has this Top 10:
Cn1, Cn2, Cn3, Cn4, Cn5, Cn6, Cn7, Cn8, Cn9, Cn10.
In this case the update indicator equals:
10*10/100 = 1.00 (100%)
The analyzer also calculates the additional parameters: the number of sites which disappeared from the search results and the number of sites which changed their positions.
This analyzer has no valuation. The results can be interpreted in two ways: a search engine that has frequent large updates could be considered more up-to-date; a search engine with rare updates can be considered more stable and predictable. The informer of this analyzer sorts the search engines in the ascending order of update level.
Hide ↑
[ Link to article ]
|
