Robot Learning from Massive Human Videos

Title: Robot Learning from Massive Human Videos
 
Abstract: Robotics has made remarkable strides, driven by advances in machine learning, optimal control, and hardware innovation. However, unlike conventional machine learning domains, data collection in robotics poses unique challenges: it is neither straightforward nor inherently scalable. To address this gap, our work focuses on leveraging in-the-wild human videos to enable learning for manipulation and agile robotics.
First, I will present "RAM," our recent effort to achieve zero-shot robotic manipulation by retrieval with foundation models. Next, I will delve into "UH-1," a project that enables humanoid control by learning from vast datasets of human motion videos.
 
Bio: Yue Wang is an assistant professor at the University of Southern California and a faculty scientist at Nvidia Research. His lab is focusing on three major directions: 1) neural scene representations for robotics; 2) real-to-simulation-to-real transfer for robotics; 3) and robotic manipulation. He worked on 3D geometric deep learning during his PhD. His paper "Dynamic Graph CNN" has been widely adopted in 3D visual computing and beyond. He received the Powell Faculty Research Award, Toyota Young Faculty Researcher award, Nvidia Fellowship, the best paper award in geometric computing and graphics at the inaugural international congress of basic science, and the best paper nomination at the CVPR 2021 workshop on autonomous driving. He was also named the first place recipient of the William A. Martin Master’s Thesis Award for 2021. Yue received his bachelor from Zhejiang University, master from UCSD, and PhD from MIT. He has spent time at Nvidia Research, Google Research and Salesforce Research