Learning from Implicit Feedback Through Online Experimentation

Project Goals

The goal of the project is to harness the information contained in users' interactions with information systems (e.g. query reformulations, clicks, dwell time) to train those systems to better serve their users' information needs. The key challenge lies in properly interpreting this implicit feedback and collecting it in a way that provides valid training data. Moving beyond existing passive data collection methods, the project draws on multi-armed bandit algorithms, experiment design, and machine learning to actively collect implicit feedback data. Developing these interactive experimentation methods goes hand-in-hand with developing machine learning algorithms that can use the resulting training data, and empirical evaluations that validate the models of user behavior assumed by the algorithms.

This research will improve retrieval quality for important applications like intranet search and desktop search. Additionally, the project will provide an operational full-text search engine for the Physics E-Print ArXiv and potentially other digital libraries, thus forming a test-bed for the research while also providing a valuable service and dissemination tool to the academic community beyond computer science. Including an REU Supplement, the project provides interesting and motivating research opportunities to undergrads and international exchange students, and the PI's will include relevant material into the undergraduate and graduate curriculum. Finally, the project will provide easy-to-use software that enables research and teaching, via this project website.

People

• Thorsten Joachims (PI)
• Robert Kleinberg (Co-PI)
• Pannaga Shivaswamy (Postdoc)
• Yisong Yue (Ph.D. Student)
• Yue Gao (Ph.D. Student)
• Karthik Raman (Ph.D. Student)
• Joshua Moore (Ph.D. Student)
• Shuo Chen (Ph.D. Student)
• Adith Swaminathan (Ph.D. Student)
• Ruben Sipos (Ph.D. Student)
• Diego Accame (Ugrad Student)
• Jacob Banks (Ugrad Student)
• Christina Brandt (Ugrad/Master Student)
• Tobias Schnabel (Visiting Undergraduate Student)
• Andrew Hoelscher (Ugrad Student)
• David Kupiec (Ugrad Student)
• Hooyeon Haden Lee (Ugrad Student / Research Assistant)
• Tobias Schnabel (Visiting Student)

Related Publications

Related Software

• Dynamic Ranking - software for Dynamic Ranked Retrieval.
• OSMOT Search Engine - open-source search engine for learning ranked retrieval functions and evaluating the learned rankings.
• SFOUR - Learning Submodular Models - software for learning submodular summarization and retrieval models.

Related Courses

• T. Joachims, Advanced Topics in Machine Learning: "Learning with Humans in the Loop", Spring 2010.

Related Presentations

• T. Joachims, Keynote Address: Learning with Humans in the Loop, European Conference on Machine Learning (ECML), 2013.
• K. Raman, Stable Coactive Learning via Perturbation, International Conference on Machine Learning (ICML), 2013.
• Shuo Chen, Multi-space Probabilistic Sequence Modeling, ACM Conference on Knowledge Discovery and Data Mining (KDD), 2013.
• R. Sipos, Generating Comparative Summaries from Reviews, poster, Conference on Information and Knowledge Management (CIKM), 2013.
• K. Raman, Learning Socially Optimal Information Systems from Egoistic Users, European Conference on Machine Learning (ECML), 2013.
• K. Raman, Online Learning to Diversify from Implicit Feedback. ACM Conference on Knowledge Discovery and Data Mining (KDD), 2012.
• T. Joachims, Online Structured Prediction via Coactive Learning, International Conference on Machine Learning (ICML), 2012.
• T. Joachims, Online Learning with Implicit User Preferences. Invited Talk, IPAM Workshop on Multimedia Search, 2011.
• T. Joachims, Online Learning with Implicit User Preferences. Invited Talk, NIPS Workshop on Choice Models and Preference Learning, 2011.
• T. Joachims, Online Learning with Implicit User Preferences. Invited Talk, Netflix, 2012.
• P. Shivaswamy, Multi-armed Bandit Problems with History. Conferenceon Artificial Intelligence and Statistics (AISTATS), 2012.
• K. Raman, Learning to Diversify from Implicit Feedback. WSDM Workshop on Diversity in Document Retrieval, 2012.
• T. Joachims, Learning with Humans in the Loop. Invited Talk, Purdue University, 2011.
• T. Joachims, The Value of User Feedback. Invited Keynote Talk, European Conference on Information Retrieval, 2011.
• K. Raman, Structured Learning of Two-Level Dynamic Rankings. Conference on Information and Knowledge Management (CIKM), 2011.
• P. Shivaswamy, Multi-armed Bandit Problems with History, The Learning Workshop, 2011.
• Yisong Yue, Beat the Mean Bandit. International Conference on Machine Learning, 2011.
• T. Joachims, Dynamic Ranked Retrieval, Conference on Web Search and Data Mining (WSDM), 2011.
• T. Joachims, Learning with Humans in the Loop. Invited Talk, Machine Learning Distinguished Lecture Series, Carnegie Mellon University, 2010.
• T. Joachims, Optimizing Search Engines as a Dueling Bandits Problem. Invited Talk, ICML Workshop on Reinforcement Learning and Search in Very Large Spaces, Haifa, Israel, 2010.
• Yisong Yue, An Interactive Learning Approach to Optimizing Information Retrieval Systems.
  • Carnegie Mellon University, Pittsburgh, PA, 2010.
  • Yahoo! Research, Santa Clara, CA, 2010.
  • Google Zurich, Zurich, Switzerland, 2010.
  • Microsoft Research Asia, Beijing, China, 2010.
• Yisong Yue, New Learning Frameworks for Information Retrieval.
  • Microsoft Research, Redmond, WA, 2010. (video)
  • Google, Mountain View, CA, 2010.
  • Johns Hopkins University, Baltimore, MD, 2010.
  • Yahoo! Research, Sunnyvale, CA, 2010.
  • Carnegie Mellon University, Pittsburgh, PA, 2010.
  • Cornell University, Ithaca, NY, 2010.
  • IBM TJ Watson, Hawthorne, NY, 2009.
• T. Joachims, Evaluating and Optimizing Search Engines through Interactive Experiments. Invited Talk, Microsoft Research, Cambridge, UK, 2009.
• T. Joachims, Measuring Result Utility through Interactive Experiments. Invited Talk, Seminar on Interactive Information Retrieval, Dagstuhl, Germany, 2009.
• Yisong Yue, Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem.
  • New York Academy of Sciences Machine Learning Symposium, New York, NY, 2009.
  • Cornell University, Ithaca, NY, 2009.
• T. Joachims, How does Clickthrough Data Reflect Retrieval Quality? Invited Talk, University of Glasgow, Glasgow, UK, 2009.
• Yisong Yue, Towards Interactive Approaches to Learning to Rank. Invited Talk. SIGIR 2009 Workshop on Learning to Rank, Boston, MA, 2009.
• T. Joachims, Search Engines that Learn. Invited Talk, Fraunhofer IAIS, Sankt Augustin, Germany.
• R. Kleinberg, Some Vignettes From Learning Theory. Invited Talk, Microsoft Faculty Summit, Redmond, WA, 2009.

Acknowledgement and Disclaimer

This material is based upon work supported by the National Science Foundation under Award IIS-0905467. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation (NSF).

Last change: 12/20/2013