Zhengyi (Zen) Luo

Hi there! My name is Zhengyi Luo (罗正宜) and I am a third year PhD student at Carnegie Mellon University’s Robotics Institute, School of Computer Science, advised by Prof. Kris Kitani. I earned my bachelor’s degree from University of Pennsylvania in 2019, where I worked with Prof. Kostas Daniilidis. My research has been supported by Qualcomm Innovation Fellowship and the Meta AI Mentorship Program.

My research interest lies at the intersection of vision, learning, and robotics. I am working on topcis including human pose estimation, human-object interaction, human motion modelling etc. Through my research, I want to create methods that effectively interpret spatial-temporal sensory input and build a representation of the 3D world to reason about the interactions between agents and the physical environment. On the application side, I am excited about humanoid robots and AR/VR.

Pittsburgh PA, 15213

news

Jul 13, 2023 One paper on real-time humanoid control and imitation accepted to ICCV 2023!
Jul 13, 2023 One paper on pose estimation using simulated humanoid in driving accepted to ICCV 2023!
Feb 27, 2023 One paper on controllable pedestrain animation accepted to CVPR 2023!
Sep 14, 2022 One paper on embodied human pose estimation accpeted to NeurIPS 2022!
Sep 1, 2022 Joining Meta as a Visiting Research at Meta Reality Labs, Pittsburgh for a year!
Aug 30, 2022 Awarded Qualcomm Innovation Fellowship 2022!
Show more

Affiliations

Ph.D. in Robotics
School of Computer Science
Aug 2021-Present
Visting Researcher
Meta Reality Labs
PI: Weipeng Xu
Aug 2022-Aug 2024

Research Scientist Intern, Toronto AI Lab
PI: Sanja Fidler
May 2022-Aug 2022

M.S in Robotics
School of Computer Science
Aug 2019-Aug 2021

3D Software Engineer Intern, AR/VR
PI: Novaira Masood
May 2018/2019-August 2018/2019

B.S.E in CS
School of Engineering and Applied Science
Aug 2015-May 2019

Publications

  • Real-Time Simulated Avatar from Head-Mounted Sensors
  • Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
  • In submission
  • [Project]
  • Universal Humanoid Motion Representations for Physics-Based Control
  • Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
  • In submission
  • [Paper] [Project]
  • Perpetual Humanoid Control for Real-time Simulated Avatars
  • Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu
  • ICCV 2023
  • [Video] [Paper] [Project] [Code]
  • Learning Human Dynamics in Autonomous Driving Scenarios
  • Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, Sameh Khamis
  • ICCV 2023
  • [Paper]
  • Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
  • Davis Rempe*, Zhengyi Luo*, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany
  • CVPR 2023
  • * Equal contribution
  • [Video] [Paper] [Project]
  • Embodied Scene-aware Human Pose Estimation
  • Zhengyi Luo*, Shun Iwase*, Ye Yuan, Kris M. Kitani
  • NeurIPS 2022
  • * Equal contribution
  • [Video] [Paper] [Project] [Code]
  • From Universal Humanoid Control to Automatic Physically Valid Character Creation
  • Zhengyi Luo, Ye Yuan, Kris M. Kitani
  • ArXiv
  • [Paper] [Project] [Code]
  • Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
  • Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, Kris M. Kitani
  • ICLR 2022 (Oral Presentation — Top 1.6%)
  • [Paper] [Project] [Code]
  • Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
  • Zhengyi Luo, Ryo hachiuma, Ye Yuan, Kris M. Kitani
  • ICCVW (CV4ARVR) 2021, NeurIPS 2021
  • [Video] [Paper] [Project] [Code]
  • 3D Human Motion Estimation via Motion Compression and Refinement
  • Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani
  • ACCV 2020
  • Oral Presentation
  • [Video] [Paper] [Project] [Code]
  • Learning Shape Representations for Clothing Variations in Person Re-Identification
  • Yu-Jhe Li, Zhengyi Luo, Xinshuo Weng, Kris M. Kitani
  • arXiv, 2020
  • [Paper]
  • Cross-Domain 3D Equivariant Image Embeddings
  • Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, Ameesh Makadia
  • ICML 2019
  • [Paper]
  • Cloud Chaser: Real Time Deep Learning Computer Vision on Low Computing Power Devices
  • Zhengyi Luo, Austin Small, Liam Dugan, Stephen Lane
  • ICMV 2018
  • [Paper] [Project] [Code]
  • The rural–urban stress divide: Obtaining geographical insights through Twitter
  • Kokil Jaidka, Sharath Chandra Guntuku, Jane H Lee, Zhengyi Luo, Anneke Buffone, Lyle H Ungar
  • Computers in Human Behavior, 2020: 106544
  • [Paper] [Project]
  • Visual Analytics Approach to Vessel Behaviour Analysis
  • Liang Jin*, Zhengyi Luo*, Shu Gao
  • Journal of Navigation, 2018,71(5): 1195-1209
  • * indicates equal contribution

---

Professional Services

Conference Reviewer: ICML (2020, 2021, 2022, 2023), ICLR (2022, 2023), NeurIPS (2021, 2022), CVPR (2022, 2023), ICCV (2021), ECCV (2022)

Journal Reviewer: IJCV, TMM