Jiyao Zhang

I am a second-year Ph.D. candidate in the Center on Frontiers of Computing Studies (CFCS) at the School of Computer Science, Peking University, China, advised by Prof. Hao Dong. Before this, I was a research assistant at the EPIC Lab, advised by Prof. He Wang.

My research focuses on 3D computer vision and robotics, with an emphasis on embodied perception and manipulation. I aim to enable robots to autonomously perceive, understand, and interact with the world.

We are looking for interns in the areas of robot perception, synthetic data generation, and robot manipulation. If you are interested in our research, don't hesitate to contact me.

Email: jiyaozhang@stu.pku.edu.cn  /  Google Scholar

📣 News

📕 Publications( * : equal contribution)

Code

@article{zhang2024omni6dpose,
 title={Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking},
 author={Zhang, Jiyao and Huang, Weiyao and Peng, Bo and Wu, Mingdong and Hu, Fei and Chen, Zijian and Zhao, Bo and Dong, Hao},
 booktitle={European Conference on Computer Vision},
 year={2024},
 organization={Springer} }

We introduce Omni6DPose, a substantial dataset characterized by its diversity in object categories, large scale, and variety in object materials. To address the substantial variations and ambiguities of Omni6DPose, we introduce GenPose++, a SOTA category-level pose estimation framework.

@article{tian2024robokeygen,
 title={RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-based 3D Keypoint Generation},
 author={Tian, Yang and Zhang, Jiyao and Huang, Guowei and Wang, Bin and Wang, Ping and Pang, Jiangmiao and Dong, Hao},
 booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
 year={2024} }

We present a novel framework to predict robot pose and joint angles, bifurcating the high-dimensional prediction task into two manageable subtasks: 2D keypoints detection and lifting 2D keypoints to 3D.

@article{liu2024rgbgrasp,
 title={RGBGrasp: Image-based Object Grasping by Capturing Multiple Views during Robot Arm Movement with Neural Radiance Fields},
 author={Liu, Chang and Shi, Kejian and Zhou, Kaichen and Wang, Haoxiao and Zhang, Jiyao and Dong, Hao},
 journal={IEEE Robotics and Automation Letters},
 year={2024},
 publisher={IEEE} }

We introduce a pioneering approach called RGBGrasp. This method depends on a limited set of RGB views to perceive the 3D surroundings containing transparent and specular objects and achieve accurate grasping.

@article{zeng2023distilling,
 title={Distilling Functional Rearrangement Priors from Large Models},
 author={Zeng, Yiming and Wu, Mingdong and Yang, Long and Zhang, Jiyao and Ding, Hao and Cheng, Hui and Dong, Hao},
 journal={IEEE Robotics and Automation Letters},
 year={2024},
 publisher={IEEE} }

We propose a novel approach that leverages large models to distill functional rearrangement priors.

Code

@article{zhang2023genpose,
 title = {GenPose: Generative Category-level Object Pose Estimation via Diffusion Models},
 author = {Zhang, Jiyao and Wu, Mingdong and Dong, Hao},
 journal = {Advances in Neural Information Processing Systems},
 year = {2023} }

We explore a pure generative approach to tackle the multi-hypothesis issue in 6D Category-level Object Pose Estimation. The key idea is to generate pose candidates using a score-based diffusion model and aggregate poses using an energy-based diffusion model. By aggregating the remaining candidates, we can obtain a robust and high-quality output pose.

@article{wu2023learning,
 title = {Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping},
 author = {Wu, Tianhao and Wu, Mingdong and Zhang, Jiyao and Gan, Yunchong and Dong, Hao},
 journal = {Advances in Neural Information Processing Systems},
 year = {2023} }

We propose a novel task called human-assisting dexterous grasping that aims to train a policy for controlling a robotic hand's fingers to assist users in grasping objects.

@inproceedings{tian2023robot,
 title={Robot Structure Prior Guided Temporal Attention for Camera-to-Robot Pose Estimation From Image Sequence},
 author={Tian, Yang and Zhang, Jiyao and Yin, Zekai and Dong, Hao},
 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
 pages={8917--8926},
 year={2023} }

We propose Structure Prior Guided Temporal Attention for online Camera-to-Robot Pose estimation (SGTAPose) from successive frames of an image sequence.

Code

@inproceedings{dai2022domain,
 title={Domain randomization-enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects},
 author={Dai, Qiyu and Zhang, Jiyao and Li, Qiwei and Wu, Tianhao and Dong, Hao and Liu, Ziyuan and Tan, Ping and Wang, He},
 booktitle={European Conference on Computer Vision},
 pages={374--391},
 year={2022},
 organization={Springer} }

We propose Domain Randomization Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and demonstrate that the proposed DREDS bridges the sim-to-real domain gap.

🏅 Honors

Template adapted from Jon Barron