Zhengyi (Zen) Luo

Hi there! My name is Zhengyi Luo (罗正宜) and I am a third year PhD student at Carnegie Mellon University’s Robotics Institute, School of Computer Science, advised by Prof. Kris Kitani. I earned my bachelor’s degree from University of Pennsylvania in 2019, where I worked with Prof. Kostas Daniilidis. My research has been supported by Qualcomm Innovation Fellowship and the Meta AI Mentorship Program.

My research interest lies at the intersection of vision, learning, and robotics. I am working on topcis including human pose estimation, human-object interaction, human motion modelling etc. Through my research, I want to create methods that effectively interpret spatial-temporal sensory input and build a representation of the 3D world to reason about the interactions between agents and the physical environment. On the application side, I am excited about humanoid robots and AR/VR.

Pittsburgh PA, 15213

news

Feb 27, 2024 SimXR (simulated humanoid from XR headsets) accepted to CVPR 2024 (highlight - top 11%)!
Feb 27, 2024 PACER+ (simulated humanoid for pedestrain animation) accepted to CVPR 2024!
Feb 27, 2024 Ego-Exo4D (large-scale multiview video dataset) accepted to CVPR 2024 (oral - top 3%)
Jan 24, 2024 PULSE (humanoid motion representation) accepted to ICLR 2024 (spotlight - top 5%)!
Jul 13, 2023 PHC (real-time humanoid control and imitation) accepted to ICCV 2023!
Jul 13, 2023 Pose estimation using simulated humanoid in driving scenes accepted to ICCV 2023!
Show more

Affiliations

Ph.D. in Robotics
School of Computer Science
Aug 2021-Present
Visting Researcher
Meta Reality Labs
PI: Weipeng Xu
Aug 2022-Aug 2024

Research Scientist Intern, Toronto AI Lab
PI: Sanja Fidler
May 2022-Aug 2022

M.S in Robotics
School of Computer Science
Aug 2019-Aug 2021

3D Software Engineer Intern, AR/VR
PI: Novaira Masood
May 2018/2019-August 2018/2019

B.S.E in CS
School of Engineering and Applied Science
Aug 2015-May 2019

Publications

  • Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
  • Tairan He*, Zhengyi Luo*, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi
  • In submission
  • * Equal contribution
  • [Paper] [Project]
  • Real-Time Simulated Avatar from Head-Mounted Sensors
  • Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
  • CVPR 2024 (Highlight — Top 11%)
  • [Paper] [Project]
  • PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios
  • Jingbo Wang*, Zhengyi Luo*, Ye Yuan, Yixuan Li, Bo Dai
  • CVPR 2024
  • * Equal contribution
  • [Paper]
  • Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
  • K Grauman et al.
  • CVPR 2024 (Oral — Top 3%)
  • [Paper] [Project]
  • PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction
  • Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, Lei Zhang
  • In Submission
  • [Paper] [Project]
  • Universal Humanoid Motion Representations for Physics-Based Control
  • Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
  • ICLR 2024 (Spotlight — Top 5%)
  • [Paper] [Project] [Code]
  • Perpetual Humanoid Control for Real-time Simulated Avatars
  • Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu
  • ICCV 2023
  • [Video] [Paper] [Project] [Code]
  • Learning Human Dynamics in Autonomous Driving Scenarios
  • Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, Sameh Khamis
  • ICCV 2023
  • [Paper]
  • Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
  • Davis Rempe*, Zhengyi Luo*, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany
  • CVPR 2023
  • * Equal contribution
  • [Video] [Paper] [Project]
  • Embodied Scene-aware Human Pose Estimation
  • Zhengyi Luo*, Shun Iwase*, Ye Yuan, Kris M. Kitani
  • NeurIPS 2022
  • * Equal contribution
  • [Video] [Paper] [Project] [Code]
  • From Universal Humanoid Control to Automatic Physically Valid Character Creation
  • Zhengyi Luo, Ye Yuan, Kris M. Kitani
  • ArXiv
  • [Paper] [Project] [Code]
  • Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
  • Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, Kris M. Kitani
  • ICLR 2022 (Oral Presentation — Top 1.6%)
  • [Paper] [Project] [Code]
  • Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
  • Zhengyi Luo, Ryo hachiuma, Ye Yuan, Kris M. Kitani
  • ICCVW (CV4ARVR) 2021, NeurIPS 2021
  • [Video] [Paper] [Project] [Code]
  • 3D Human Motion Estimation via Motion Compression and Refinement
  • Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani
  • ACCV 2020
  • Oral Presentation
  • [Video] [Paper] [Project] [Code]
  • Learning Shape Representations for Clothing Variations in Person Re-Identification
  • Yu-Jhe Li, Zhengyi Luo, Xinshuo Weng, Kris M. Kitani
  • arXiv, 2020
  • [Paper]
  • Cross-Domain 3D Equivariant Image Embeddings
  • Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, Ameesh Makadia
  • ICML 2019
  • [Paper]
  • Cloud Chaser: Real Time Deep Learning Computer Vision on Low Computing Power Devices
  • Zhengyi Luo, Austin Small, Liam Dugan, Stephen Lane
  • ICMV 2018
  • [Paper] [Project] [Code]
  • The rural–urban stress divide: Obtaining geographical insights through Twitter
  • Kokil Jaidka, Sharath Chandra Guntuku, Jane H Lee, Zhengyi Luo, Anneke Buffone, Lyle H Ungar
  • Computers in Human Behavior, 2020: 106544
  • [Paper] [Project]

---

Professional Services

Conference Reviewer: ICML (2020, 2021, 2022, 2023), ICLR (2022, 2023), NeurIPS (2021, 2022), CVPR (2022, 2023), ICCV (2021), ECCV (2022)

Journal Reviewer: IJCV, TMM