Evaluating information retrieval and access tasks: NTCIR's legacy of research impact

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life. Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants. This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one.

Chapter 1. Graded Relevance -- Chapter 2. Experiments on Cross-Language Information Retrieval using Comparable Corpora of Chinese, Japanese, and Korean Languages -- Chapter 3. Text Summarization Challenge -- Chapter 4. Challenges in Patent Information Retrieval -- Chapter 5. Multi-Modal Summarization -- Chapter 6. Opinion Analysis Corpora Across Languages -- Chapter 7. Patent Translation -- Chapter 8. Component-Based Evaluation for Question Answering -- Chapter 9. Temporal Information Access -- Chapter 10. SogouQ -- Chapter 11. Evaluation of Information Access with Smartphones -- Chapter 12. Mathematical Information Retrieval -- Chapter 13. Experiments in Lifelog Organisation and Retrieval at NTCIR -- Chapter 14. The Future of Information Retrieval Evaluation.