Research

Pioneering Generative AI Research for VFX and Digital Humans

The recent breakthroughs in generative AI for image and video creation are poised to revolutionize the film and entertainment industry, changing the way we produce high-quality film content. Our mission embarks on embracing cutting-edge technologies to address VFX challenges that were once deemed either impossible or prohibitively expensive using traditional CG methods, while in the long term, aiming for more generalized production-level video generation and authoring capabilities.

In our relentless pursuit of staying at the forefront of innovation and maintaining a competitive edge, Pinscreen is committed to engaging actively in fundamental research. This commitment is evident in our exploration of new technological capabilities through scientific publications, especially in the areas of Computer Vision, Computer Graphics, and Machine Learning.

Since our inception, we have published 50 papers at top conferences (CVPR, ICCV, ECCV, SIGGRAPH, NeurIPS, etc.) and generated 7 patents, among others. Our groundbreaking work has earned prestigious recognitions, being showcased at the World Economic Forum in Davos and 8 presentations at SIGGRAPH Real-Time Live! Our collaborative efforts extend to partnerships with leading academic institutions (MBZUAI, UC Berkeley, ETH Zurich, USC, etc.), and we actively participate in government-funded research programs such as DARPA.

Key Publications

VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence

Phong Tran, Egor Zakharov, Long-Nhat Ho, Liwen Hu, Adilbek Karmanov, Aviral Agarwal, McLean Goldwhite, Ariana Bermudez Venegas, Anh Tuan Tran, Hao Li
Proceedings of the 17th ACM SIGGRAPH Conference and Exhibition in Asia 2024 – SIGGRAPH Asia 2024
[paper]

VOODOO 3D: VOlumetric POrtrait Disentanglement fOr One-Shot 3D Head Reenactment

Phong Tran, Egor Zakharov, Long Nhat Ho, Anh Tuan Tran, Liwen Hu, Hao Li
Proceedings of the 37th IEEE International Conference on Computer Vision and Pattern Recognition 2024 – CVPR 2024
[paper] [video] [project]

XMEM++: Production-Level Video Segmentation from Few Annotated Frames

Maksym Bekuzharov, Ariana Bermudez, Joon-Young Lee, Hao Li
Proceedings of the IEEE International Conference on Computer Vision 2023 – ICCV 2023
[paper] [video] [project]

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion

Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, Shiry Ginosar
Proceedings of the 35th IEEE International Conference on Computer Vision and Pattern Recognition 2022 – CVPR 2022
[paper] [video] [project]

Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement

Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, Shiry Ginosar
Proceedings of the 34th IEEE International Conference on Computer Vision and Pattern Recognition 2021 – CVPR 2021
[paper] [video] [project]

Deep Face Normalization

Koki Nagano, Huiwen Luo Zejian Wang, Jaewoo Seo, Jun Xing, Liwen Hu, Lingyu Wei, Hao Li
Proceedings of the 12th ACM SIGGRAPH Conference and Exhibition in Asia 2019 – SIGGRAPH Asia 2019
[paper] [video]

PaGAN: Real-Time Avatars Using Dynamic Textures

Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, Hao Li
Proceedings of the 11th ACM SIGGRAPH Conference and Exhibition in Asia 2018 – SIGGRAPH Asia 2018
[paper] [video]

Avatar Digitization from a Single Image for Real-Time Rendering

Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Carrie Sun, Yen-Chun Chen, Hao Li
Proceedings of the 10th ACM SIGGRAPH Conference and Exhibition in Asia 2017 – SIGGRAPH Asia 2017
[paper] [video]

FLAME: Learning a Model of Facial Shape and Expression from 4D Scans

Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, Javier Romero
Proceedings of the 10th ACM SIGGRAPH Conference and Exhibition in Asia 2017 – SIGGRAPH Asia 2017
[paper] [video] [project]

Realistic Dynamic Facial Textures from a Single Image using GANs

Kyle Olszewski, Zimo Li, Chao Yang, Yi Zhou, Ronald Yu, Zeng Huang, Sitao Xiang, Shunsuke Saito, Pushmeet Kohli, Hao Li
Proceedings of the IEEE International Conference on Computer Vision 2017 – ICCV 2017
[paper] [video]

Photorealistic Facial Texture Inference using Deep Neural Networks

Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, Hao Li
Proceedings of the 30th IEEE International Conference on Computer Vision and Pattern Recognition 2017 – CVPR 2017
[paper] [video]