IC Spring Seminar Series with Guest Speaker Jesse Thomason

Embracing Language as Grounded Communication

Abstract

Language is not text data, it is a human medium for communication. The larger part of the natural language processing (NLP) community has doubled down on treating digital text as a sufficient approximation of language, scaling datasets and corresponding models to fit that text. I have argued that experience in the world grounds language, tying it to objects, actions, and concepts. In fact, I believe that language carries meaning only when considered alongside that world, and that the zeitgeist in NLP research currently misses the mark on truly interesting questions at the intersection of human language and machine computation. In this talk, I’ll highlight some of the ways my lab enables agents and robots to better understand and respond to human communication by considering the grounded context in which that communication occurs, including neurosymbolic multimodal reasoning, natural language dialogue and interaction for lifelong learning, and utilizing NLP technologies on non-text communication. 

Bio

I am an Assistant Professor at the University of Southern California where I lead the Grounding Language in Actions, Multimodal Observations, and Robots (GLAMOR) Lab. Our research enables agents and robots to better understand and respond to human language by considering the grounded context in which that language occurs by considering three threads: 1) We jointly learning models with language, world perception, and physical action to enable end-to-end agent behavior and improve continual learning; 2) We investigate ways to take advantage of the extra-textual visual world and embodied context in which language is uttered to improve reasoning in language-and-vision and language-guided robotics tasks; and 3) We work to improve speech and sign recognition by leveraging contextual and structural information, as well as to apply language technologies to accessibility and health applications.