LMSS @ Cornell Tech: Jason Baldridge, Google

Language Lifts Experience (via Zoom)

Abstract: Human knowledge and use of language is inextricably connected to perception, action and the organization of the brain. The availability of large multimodal datasets and high-fidelity simulations of the real world is enabling new research on learning grounded representations for complex utterances across the world's languages. Learning methods based on neural networks have not only achieved impressive empirical gains on practical tasks, but are also producing remarkable representations of language while using only indirect supervision. That said, current work tends to be remarkably naive about linguistic diversity, complexity and generalization---in ways that are both productive and limiting. 

In this context, I'll discuss my team's work on vision-and-language navigation, with a focus on our recently released multilingual Room-across-Room (RxR) dataset and contextualizing it with respect to existing datasets that combine vision, language and action. I'll then shift to a discussion of broader questions both of meaning in language and of language's role in structuring our experience -- and how work on computational language grounding presents new opportunities to enhance and advance our scientific understanding of language and its fundamental role in human intelligence.

Bio: Jason is a research scientist at Google, where he works on natural language understanding. He was previously an Associate Professor of Computational Linguistics at the University of Texas at Austin. His main research interests include categorial grammars, parsing, semi-supervised learning for NLP, reference resolution and text geolocation. He has long been active in the creation and promotion of open source software for natural language processing, including co-creating the Apache OpenNLP Toolkit and OpenCCG. Jason received his Ph.D. from the University of Edinburgh in 2002, where his doctoral dissertation on Multimodal Combinatory Categorial Grammar was awarded the 2003 Beth Dissertation Prize from the European Association for Logic, Language and Information.