Wei-Chiu Ma 馬惟九

Assistant Professor @ Cornell CS

Email / Google scholar / Twitter / Misc. / Pro Bono

About Me

I am an Assistant Professor of Computer Science at Cornell University.

My research lies at the intersection of 3D/4D computer vision and robotics. I am interested in building AI systems that can understand, reconstruct, and re-simulate our dynamic world, leveraging these capabilities to enable more robust autonomous systems or advance entertainment applications.

Prior to joining Cornell, I was a Young Investigator/Postdoc at AI2/University of Washington. I received my Ph.D. from MIT, where I worked with Antonio Torralba (aka the Great Torralba) and Raquel Urtasun. Previously, I was a Senior Research Scientist at Uber ATG R&D and Waabi working on self-driving vehicles. I completed my M.S. in Robotics at Carnegie Mellon University (CMU), where I was advised by Kris M. Kitani.

Prospective students: I am always looking for motivated and talented students! If you are interested in collaborating or joining my group as a PhD/MS/Undergrad student or intern, please read this.

Research Projects

(show selected / show by date)

	Beyond the Frame: Generating 360° Panoramic Videos from Perspective Videos Rundong Luo, Matthew Wallingford, Ali Farhadi, Noah Snavely, Wei-Chiu Ma arXiv 2025 / project page / arXiv / code (coming soon)
	DRAWER: Digital Reconstruction and Articulation With Environment Realism Hongchi Xia, Entong Su, Marius Memmel, Arhan Jain, Raymond Yu, Numfor Mbiziwo-Tiapo, Ali Farhadi, Abhishek Gupta, Shenlong Wang, Wei-Chiu Ma CVPR 2025 / project page / arXiv (coming soon) / code (coming soon)
	Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation Shivam Duggal, Yushi Hu, Oscar Michel, Aniruddha Kembhavi, William T. Freeman, Noah A. Smith, Ranjay Krishna, Antonio Torralba, Ali Farhadi, Wei-Chiu Ma CVPR 2025 / arXiv (coming soon)
	Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering Cheng Sun, Jaesung Choe, Charles Loop, Wei-Chiu Ma, Yu-Chiang Frank Wang CVPR 2025 / arXiv / code
	Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model Benlin Liu, Yuhao Dong, Yiqin Wang, Zixian Ma, Yansong Tang, Luming Tang, Yongming Rao, Wei-Chiu Ma, Ranjay Krishna CVPR 2025 / project page / arXiv
	From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos Matthew Wallingford, Anand Bhattad, Aditya Kusupati, Vivek Ramanujan, Matt Deitke, Sham Kakade, Aniruddha Kembhavi, Roozbeh Mottaghi, Wei-Chiu Ma, Ali Farhadi NeurIPS 2024 / project page / arXiv / huggingface
	Multilingual Diversity Improves Vision-Language Representations Thao Nguyen, Matthew Wallingford, Sebastin Santy, Wei-Chiu Ma, Sewoong Oh, Ludwig Schmidt, Pang Wei Koh, Ranjay Krishna NeurIPS 2024 / paper Spotlight presentation
	Task Me Anything Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, Ranjay Krishna NeurIPS 2024 Datasets and Benchmarks / project page / paper / huggingface / code
	BLINK: Multimodal Large Language Models Can See but Not Perceive Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma^†, Ranjay Krishna^† ECCV 2024 / project page / paper / dataset / Eval AI / code
	Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, Shenlong Wang CVPR 2024 / project page / paper / shooting demo / code
	ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models Meng-Li Shih, Wei-Chiu Ma, Lorenzo Boyice, Aleksander Holynski, Forrester Cole, Brian Curless, Janne Kontkanen CVPR 2024 / arXiv
	Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects Tianhang Cheng, Wei-Chiu Ma, Kaiyu Guan, Antonio Torralba, Shenlong Wang NeurIPS 2023 / project page / arXiv
	LightSim: Neural Lighting Simulation for Urban Scenes Ava Pun, Gary Sun, Jingkang Wang, Yun Chen, Ze Yang, Sivabalan Manivasagam, Wei-Chiu Ma, Raquel Urtasun NeurIPS 2023* / arXiv
	UniSim: A Neural Closed-Loop Sensor Simulator Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Joyce Anqi Yang, Raquel Urtasun CVPR 2023 / project page / paper / 4K demo / video (8 mins) Highlight presentation
	UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun CVPR 2023 / project page / paper / video (1 min)
	What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging Zitian Tang, Wenjie Yeh, Wei-Chiu Ma, Hang Zhao CVPR 2023 / arXiv / dataset
	SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping Yuan Shen, Wei-Chiu Ma, Shenlong Wang NeurIPS 2022 / project page / paper / code
	CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Realistic and Controllable Sensor Simulation Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Joyce Anqi Yang, Wei-Chiu Ma, Raquel Urtasun CoRL 2022 / project page / paper / video
	MIRA: Mental Imagery for Robotic Affordances Lin Yen-Chen, Pete Florence, Andy Zeng, Jonathan T. Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, Phillip Isola CoRL 2022 / project page / paper / video
	Virtual Correspondence: Humans as a Cue for Extreme-View Geometry Wei-Chiu Ma, Anqi Joyce Yang, Shenlong Wang, Raquel Urtasun, Antonio Torralba CVPR 2022 / project page / paper / video (1.5 mins) / video (5 mins) / MIT News / TechXplore
	NeurMiPs: Neural Mixture of Planar Experts for View Synthesis Zhi-Hao Lin, Wei-Chiu Ma, Hao-Yu Max Hsu, Yu-Chiang Frank Wang, Shenlong Wang CVPR 2022 / project page / paper / code / video
	Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild Shivam Duggal, Zihao Wang, Wei-Chiu Ma, Sivabalan Manivasagam, Justin Liang, Shenlong Wang, Raquel Urtasun WACV 2022 / arXiv / video
	BARF: Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey ICCV 2021 / project page / arXiv / code / video / news coverage (The Batch: DeepLearning.AI) Oral presentation
	S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling Ze Yang, Shenlong Wang, Sivabalan Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, Raquel Urtasun CVPR 2021 / arXiv / video (5 mins)
	Recovering and Simulating Pedestrians in the Wild Ze Yang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wei-Chiu Ma, Raquel Urtasun CoRL 2020 / arXiv / video Spotlight presentation
	Deep Feedback Inverse Problem Solver Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun ECCV 2020 / project page / arXiv / short video (1.5 mins) / long video (10 mins) Spotlight presentation
	Weakly-supervised 3D Shape Completion in the Wild Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang, Yuwen Xiong, Hao Su, Raquel Urtasun ECCV 2020 / arXiv Spotlight presentation
	LevelSet R-CNN: A Deep Variational Method for Instance Segmentation Namdar Homayounfar, Yuwen Xiong, Justin Liang, Wei-Chiu Ma, Raquel Urtasun ECCV 2020* / arXiv
	Conditional Entropy Coding for Efficient Video Compression Jerry Junkai Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, Raquel Urtasun ECCV 2020 / arXiv
	PolyTransform: Deep Polygon Transformer for Instance Segmentation Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, Raquel Urtasun CVPR 2020 / arXiv / supp / video / code (coming soon!) State-of-the-art performance on Cityscapes! Consistent improvements across all backbones!
	LidarSIM: Realistic LiDAR Simulation by Leveraging the Real World Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Bin Yang, Shuhan Tan, Mikita Sazanovich, Wei-Chiu Ma, Raquel Urtasun CVPR 2020 / paper Oral presentation
	Exploiting Sparse Semantic HD Maps for Self-Driving Vehicle Localization Wei-Chiu Ma, Ignacio Tartavull, Ioan Andrei Bârsan, Shenlong Wang, Min Bai, Gellert Mattyus, Namdar Homayounfar, Shrinidhi K. Lakshmikanth, Andrei Pokrovsky, Raquel Urtasun IROS 2019 / arXiv / video Oral presentation
	Deep Rigid Instance Scene Flow Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, Raquel Urtasun CVPR 2019 / project page / arXiv / paper + supp (uncompressed) / GN solver gif Deep structured scene flow model that rank 1st on KITTI Scene Flow Benchmark. Faster than prior art by 800 times.
	Convolutional Recurrent Network for Road Boundary Extraction Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Shenlong Wang, Raquel Urtasun CVPR 2019 / paper / supp
	DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, Raquel Urtasun ICCV 2019 / arXiv / code / differentiable PatchMatch module Real-time stereo estimation (62 ms) via Differentiable PatchMatch!
	The Sound of Motions Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba ICCV 2019 / arXiv / demon video
	DAG-Mapper: Learning to Map by Discovering Lane Topology Namdar Homayounfar, Wei-Chiu Ma, Justin Liang, Xinyu Wu, Jack Fan, Raquel Urtasun ICCV 2019 / paper / supp
	Single Image Intrinsic Decomposition without a Single Intrinsic Image Wei-Chiu Ma, Hang Chu, Bolei Zhou, Raquel Urtasun, Antonio Torralba ECCV 2018 / paper
	Deep Parametric Continuous Convolutional Neural Networks Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun CVPR 2018 / paper Spotlight presentation
	SurfConv: Bridging 3D and 2D Convolution for RGBD Images Hang Chu, Wei-Chiu Ma, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler CVPR 2018 / paper / code
	Hierarchical Recurrent Attention Networks for Structured Online Maps Namdar Homayounfar, Wei-Chiu Ma, Shrinidhi K. Lakshmikanth, Raquel Urtasun CVPR 2018 / paper / supp
	Activity Forecasting: An Invitation to Predictive Perception Kris M. Kitani, De-An Huang, Wei-Chiu Ma Group and Crowd Behavior for Computer Vision. Chapter 12, 2017 / link
	Find Your Way by Observing the Sun and Other Semantic Cues Wei-Chiu Ma, Shenlong Wang, Marcus A. Brubaker, Sanja Fidler, Raquel Urtasun ICRA 2017 / arXiv / demo video Oral presentation
	Forecasting Interactive Dynamics of Pedestrians with Fictitious Play Wei-Chiu Ma, De-An Huang, Namhoon Lee, Kris M. Kitani CVPR 2017 / arXiv
	How Do We Use Our Hands? Discovering a Diverse Set of Common Grasps De-An Huang, Wei-Chiu. Ma, Minghuan Ma, K. M. Kitani CVPR 2015 / paper
	Recognizing Hand-Object Interactions in Wearable Camera Videos Tatsuya Ishihara, Kris M. Kitani, Wei-Chiu Ma, Hironobu Takagi, Chieko Asakawa ICIP 2015 / paper
	Novel traffic signal timing adjustment strategy based on Genetic Algorithm Hsiao-Yu Tung, Wei-Chiu Ma, Tian-Li Yu CEC 2014 / paper Oral presentation
	TDTOS: T-Shirt Design and Try On System Chen-Yu Hsu, Chi-Hsien Yen, Wei-Chiu Ma, Shao-Yi Chien Asia-Pacific Workshop on FPGA Applications 2012* / demo video Oral presentation Best Application Award

Inspired by Prof. Kyunghyun Cho and Krishna Murthy, starting January 2021, I have decided to commit 1~2 hours every week to provide guidance, suggestions, and/or mentorships for students from underrepresented groups or whoever is in need. Please fill in this form if you are interested.

Wei-Chiu Ma 馬惟九

About Me

Recent News

Research Projects

Pro bono office hour

Misc.