Estimating the Query Difficulty for Information Retrieval

Many information retrieval (IR) systems suffer from a radical variance in performance when responding to users' queries. Even for systems that succeed very well on average, the quality of results returned for some of the queries is poor. Thus, it is desirable that IR systems will be able to identify "difficult" queries so they can be handled properly. Understanding why some queries are inherently more difficult than others is essential for IR, and a good answer to this important question will help search engines to reduce the variance in performance, hence better servicing their customer needs. Estimating the query difficulty is an attempt to quantify the quality of search results retrieved for a query from a given collection of documents. This book discusses the reasons that cause search engines to fail for some of the queries, and then reviews recent approaches for estimating query difficulty in the IR field. It then describes a common methodology for evaluating the prediction quality of those estimators, and experiments with some of the predictors applied by various IR methods over several TREC benchmarks. Finally, it discusses potential applications that can utilize query difficulty estimators by handling each query individually and selectively, based upon its estimated difficulty. Table of Contents: Introduction - The Robustness Problem of Information Retrieval / Basic Concepts / Query Performance Prediction Methods / Pre-Retrieval Prediction Methods / Post-Retrieval Prediction Methods / Combining Predictors / A General Model for Query Difficulty / Applications of Query Difficulty Estimation / Summary and Conclusions
About the Author: David Carmel is a Research Staff Member at the Information Retrieval group at IBM Research Lab at Haifa. David earned his PhD in Computer Science from the Technion, Israel Institute of Technology in 1997. David's research is focused on search in the enterprise, query performance prediction, social search, and text mining. For several years David taught the Introduction to IR course at the CS department at Haifa university. At IBM, David is a key contributor to IBM enterprise search offerings. David is a co-founder of the Juru search engine which provides integrated search capabilities to several IBM products, and was used as a search platform for several studies in TREC conferences. David has published more than 60 papers in Information retrieval and Web journals and conferences, and he serves in the Program Committee of many conferences (SIGIR, WWW, WSDM, CIKM, ECIR), journals (IR Journal), and workshops. Elad Yom-Tov is a Research Staff Member at the Data Analytics department at IBM Research Lab at Haifa, Israel. The main focus of his work is research into methods for large-scale machine learning, with a recent focus on social analytics. Prior to his current position he worked at Rafael Inc., where he applied machine learning to image processing. Elad is a graduate of Tel-Aviv University (B.Sc.) and the Technion, Haifa (M.Sc. and Ph.D). He is the author (with David Stork) of the Computer Manual to accompany Pattern Classification, a book and a Matlab toolbox on pattern classification. He has published over 40 papers on Machine Learning and its applications. Elad's work in Information Retrieval includes query difficulty estimation, social tagging, and novelty detection.