Anyi Rao

Assistant Professor
Human AI for Creativity
Hong Kong University of Science and Technology
Associate Director of HKUST Media Intelligence Research Center
Email: anyirao [at] ust.hk

Bio

Anyi Rao is an Assistant Professor at the Hong Kong University of Science and Technology (HKUST). He leads the Multimedia Creativity Lab (MMLab@HKUST). He is the Associate Director of HKUST Media Intelligence Research Center. He studies human AI and agentic AI, focusing on the creation, editing and understanding of art, media and film, aiming to build human-AI collaborative intelligence and unleash human creativity and productivity. His works include ControlNet, AnimateDiff, MovieNet, Virtual Studio, and IC-Light, with a Marr Prize (ICCV best paper award). These works have been widely used in industry, including Amazon Prime Video, Netflix, Tencent, and more.

He was a Postdoctoral Scholar at Stanford with Maneesh Agrawala. He received the Ph.D. at MMLab, Chinese University of Hong Kong with Dahua Lin and Bolei Zhou. He has research experiences at Meta Reality Lab, Vector Institute, University of Toronto, Hong Kong University. He organized the SIGGRAPH/CVPR/ICCV/ECCV Creative Visual Content Workshop and the SIGGRAPH Generative Models Course, curated 2025 Hong Kong HKUST AI Film Festival and 2023 Paris ShortFest AI Film Festival. He also serves as a co-chair of MMSys26, UIST25, VINCI25, CVM25, UIST24 and area chair/TPC of CVPR26, ICLR26, SIGGRAPH Asia26, UIST26, SIGGRAPH Asia25.

He has hosted the Brown Media Innovation Research Fund, Amazon Video Research Fund, been featured in Forbes 30 Under 30 Asia 2025 List, and won the Rising Star Award at the World Artificial Intelligence Conference 2024. He gave keynote at the Golden Rooster Film Festival, the Shanghai Television Magnolia Festival, was featured by Shanghai TV Financial Channel, Hong Kong Cable Television.

Actively looking for highly motivated students to join the group. See openings for more details. Please fill out this 2026 form for 2026 intake and send an email to me if you are interested in.

News

2025-12: We are organizing the 8th Workshop on AI for Creative Visual Content Generation Editing and Understanding at SIGGRAPH Asia 2025. Please follow our Twitter for more information!

2025-07: CineVision is accepted to UIST 2025 and Light-A-Video is accepted to ICCV 2025.

2025-05: We are organizing the Seventh Workshop on AI for Creative Visual Content Generation Editing and Understanding at SIGGRAPH 2025. This is the first year that SIGGRAPH has a technical workshop program.

2025-04: Chair and curate the First Hong Kong HKUST AI Film Festival with CVM 2025 on April 19.

2025-02: IC-Light is accepted to ICLR 2025 as Oral. And one paper is accepted to CVPR 2025.

2024-08: Three paper are accepted to UIST 2024, ICML 2024 and ECCV 2024.

2024-02: Cinematic Behavior Transfer is accepted to CVPR 2024. Our efforts on Intelligent Cinematography 🎬 Virtual Film Studio include our SIGGRAPH Virtual Dynamic Storyboard, CVPR Cinematic Behavior Transfer and ECCV Multi-camera Editing.

2023-11: AnimateDiff is online and gets an update on SparseCtrl ability.

2023-10: 🧑‍🎨 ControlNet receives the 🏆 Best Paper Award (Marr Prize) at ICCV 2023. V1 V1.1 A1111 WebUI

2023-06: We are organizing 🍿 inaugural Paris ShortFest AI Film Festival, jointly with ICCV 2023 in Paris, France.

2025-02: We are organizing the Sixth Workshop on AI for Creative Visual Content Generation Editing and Understanding at CVPR 2025.

2024-08: We are organizing the Course on Generative Models for Visual Content Editing and Creation at SIGGRAPH 2024.

2024-10: Serve in ACM UIST 2025 organizing committee as a Registeration Co-Chair.

2024-02: Serve in ACM UIST 2024 organizing committee as a Registeration Co-Chair.

2024-01: We are organizing the Fourth Workshop on AI for Creative Visual Content Generation Editing and Understanding at CVPR 2024.

2023-08: Three papers are accepted to ICCV 2023 as Oral, UIST 2023, and ACM MM 2023.

2023-06: We are organizing the Third Workshop on AI for Creative Video Editing and Understanding at ICCV 2023 in Paris

2023-04: Two papers are accepted to AAAI 2023 as Oral and IJCAI 2023.

2022-10: We are organizing the Second Workshop on AI for Creative Video Editing and Understanding at ECCV 2022.

2022-07: Two papers on 👷 City-Super Research: CityNeRF and Shoot360 are accepted to ECCV 2022 and SIGGRAPH 2022.

2022-03: Two papers are accepted to CVPR 2022 and IEEE Transactions on Multimedia.

2021-09: We are organizing the First Workshop on AI for Creative Video Editing and Understanding during ICCV 2021.

2021-07: Two papers are accepted to ICCV 2021 and IEEE Transactions on Multimedia.

2021-05: Our CVPR 2020 work SceneSeg is set as the baseline for the ACM Multimedia 2021 Grand Challenge: Tencent Ads Algorithm Competition. Participate to win USD$100,000 for the first prize.

2020-07: MovieNet is online with an easy-to-use toolkit as a part of OpenMMLab.

2020-07: Three papers are accepted to ECCV 2020.

2020-02: One paper is accepted to CVPR 2020. Also appears at LUV 2020 (15-min talk) and Sight and Sound 2020 (5-min talk).

2020-01: HotFlip is included in AllenNLP and TextAttack

Selected Publication [Full List]

CineVision: An Interactive Pre-visualization Storyboard System for Director–Cinematographer Collaboration
Zheng Wei, Hongtao Wu, Lvmin Zhang, Xian Xu, Yefeng Zheng, Pan Hui, Maneesh Agrawala, Huamin Qu, Anyi Rao
User Interface Software and Technology (UIST), 2025
[Paper] [Webpage]

VideoRepainter: Keyframe-Guided Creative Video Inpainting
Yuwei Guo, Ceyuan Yang, Anyi Rao, Chenlin Meng, Omer Bar-Tal, Shuangrui Ding, Maneesh Agrawala, Dahua Lin, Bo Dai
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
[Paper] [Webpage]

IC-Light: Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
International Conference on Learning Representations (ICLR), 2025 (Oral)
[Paper] [Webpage] [Demo]

ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database
Anyi Rao, Jean-Peïc Chou, Maneesh Agrawala
User Interface Software and Technology (UIST), 2024
[Paper] [Webpage]

Cinematic Behavior Transfer via NeRF-based Differentiable Filming
Xuekun Jiang*, Anyi Rao* (co-first), Jingbo Wang, Dahua Lin, Bo Dai
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
[Paper] [Webpage]

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, Bo Dai
International Conference on Learning Representations (ICLR), 2024 (Spotlight)
[Paper] [Webpage]

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, Bo Dai
European Conference on Computer Vision (ECCV), 2024
[Paper] [Webpage]

ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
IEEE/CVF International Conference on Computer Vision (ICCV), 2023 Best Paper Award (Marr Prize)
[Paper] [Webpage] [Supplements] [V1] [V1.1] [A1111 WebUI]

Automated Conversion of Music Videos into Lyric Videos
Jiaju Ma, Anyi Rao, Li-Yi Wei, Rubaiat Habib Kazi, Hijung Valentina Shin, Maneesh Agrawala
User Interface Software and Technology (UIST), 2023
[Paper] [Webpage]

Dynamic Storyboard Generation in an Engine-based Virtual Environments for Video Production
Anyi Rao*, Xuekun Jiang*, Yuwei Guo, Linning Xu, Lei Yang, Libiao Jin, Dahua Lin, Bo Dai
ACM Special Interest Group on Computer Graphics and Interactive Techniques Conference (SIGGRAPH) Poster, 2023
[Paper] [Webpage]

Shoot360: Normal View Video Creation from City Panorama Footage
Anyi Rao, Linning Xu, Dahua Lin
ACM Special Interest Group on Computer Graphics and Interactive Techniques Conference (SIGGRAPH), 2022
[Paper] [Webpage]

A Coarse-to-Fine Framework for Automatic Video Unscreen
Anyi Rao, Linning Xu, Zhizhong Li, Qingqiu Huang, Zhanghui Kuang, Wayne Zhang, Dahua Lin
IEEE Transactions on Multimedia, (TMM), 2022
[Paper] [Webpage]

BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering
Also known as CityNeRF: Building NeRF at City Scale
Yuanbo Xiangli*, Linning Xu*, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, Dahua Lin
European Conference on Computer Vision (ECCV), 2022
[Paper] [Webpage]

A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, Dahua Lin
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
LUV 2020 and Sight and Sound 2020 workshops (Oral)
[Paper] [Webpage]

A Unified Framework for Shot Type Classification Based on Subject Centric Lens
Anyi Rao, Jiaze Wang, Linning Xu, Xuekun Jiang, Qingqiu Huang, Bolei Zhou, Dahua Lin
European Conference on Computer Vision (ECCV), 2020
Video Turing Test 2020 Workshop (Oral)
[Paper] [Webpage]

MovieNet: A Holistic Dataset for Movie Understanding
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, Dahua Lin
European Conference on Computer Vision (ECCV), 2020 (Spotlight)
[Paper] [Webpage]

Selected Awards and Grants

Forbes 30 Under 30 Asia	2025
Tencent Rhino-Bird Grant	2025
AIS Support Funding for Interdisciplinary Research Collaboration	2025
Bridge Gap Funding	2025
Rising Star by World Artificial Intelligence Conference (WAIC)	2024
Talent Gift Funding	2024
Art Tech Funding	2024
Best Paper Award (Marr Prize) by International Conference on Computer Vision (ICCV)	2023
Magic Grant by The Brown Institute for Media Innovation	2023
Amazon Prime Video Gift Funding	2023

Tencnet, Cybever, and Pika Grant for SIGGRAPH Workshop Organization	2025
Adobe and Pika Grant for CVPR Workshop Organization	2024
Pika and KAUST Grant for ICCV Workshop Organization	2023
KAUST Grant for ECCV Workshop Organization	2022
Adobe Grant for ICCV Workshop Organization	2021
Hong Kong PhD Fellowship	2021
Most Influential Paper by Paper Digest	2020
Nanjing University Top-Grade Scholarship,the highest honor in the university	2018
SenseTime Scholarship, awarded to 30 students out of all AI major undergraduate students in China	2017
Provincial Merit Student awarded by the Jiangsu Province	2017
National Scholarship awarded by the China Ministry of Education	2015
Gold Medal in Invitational National Mathematical Olympiad	2013
Nanjing University Outstanding Student Leader Award	2015
Nanjing University Outstanding Student Award	2016
Nanjing University Top Volunteer Excellence Award	2015
Zhenggang Scholarship, top 40 students in Nanjing University	2016
Zhenggang Jingying Scholarship	2017
Nanjing University People Scholarship	2016
Nanjing University People Scholarship	2017
World ranking 32nd in 2016 Calculus World Cup	2016
Meritorious winner prize in the 2016 National Mathematical Contest in Modeling	2016
Best paper in the 2014 University Electronics Design Contest	2014

Talks

From Intention to Attention to Manifestation

Hong Kong International AI Art Festival, 12/2025
SIGGRAPH Asia, 12/2025
World Cultural Forum, 11/2025
SIGGRAPH, 08/2025

Bridging the Representation Gap of Humans and Computers for Video Production

China Golden Rooster Film Festival, 11/2024
World Artificial Intelligence Conference (WAIC), 07/2024
Shanghai Television Magnolia Festival, 06/2024
University of Hong Kong (HKU), 05/2024
Hong Kong University of Science and Technology (HKUST), 05/2024
Stanford University, 04/2024
Hong Kong Shanghai AI Forum, 04/2024

Collaborative Intelligent Tools to Support Video Production

Symposium on AI Technologies and Their Implications, 03/2024
Adobe, 03/2024
CCF, 03/2024
Film School of HKBU, 05/2021

Creative Video Understanding, Editing and Generation

Art School of UTK, 02/2024
ICCV23 Workshop, Paris, France, 10/2022

Controllable Visual Content Generation to Unleash Creativity and Productivity

Netflix, 11/2023
Bay Area Vision Day (Stanford, Berkeley, Caltech), 09/2023

Temporal and Contextual Transformer for Multi-Camera Editing

ECCV22 Workshop, 10/2022

Multimodal Representation Learning

Meta, Redmond, WA, 12/2022
CVPR20 Workshop on Sight and Sound, 06/2020
CVPR20 Workshop on Learning from Unlabeled Videos, 06/2020

Cinematic Style Analysis Based on Subject Centric Lens

ECCV20 Workshop on Video Turing Test: Human-level Video Story Understanding, 08/2020

Press Coverage

Where AI Meets Humanity: A Scholar Democratizing Storytelling: HKUST

AI Film Production: i-CABLE News / HKUST

AI Film Festival: Associated Press / Yahoo / PR Newswire HKET Ta Kung Pao HK Economic Journal

Media/Entertainment Industry: China Golden Rooster Film Festival / Shanghai Television Magnolia Festival / China News

World AI Conference: Video Generation Forum / Rising Star

Art: Hidden Imagery In AI Art / Funky AI-generated Spiraling Medieval Village Captivates Social Media

Professional Activities

Area Chair for CVPR26

Area Chair for ICLR26

Associate Chair for UIST26

Co-Chair (Workshop) for MMSys26

Area Chair (TPC) for SIGGRAPH Asia26

Co-Chair (Registeration) for UIST25

Area Chair (TPC) for SIGGRAPH Asia25

Co-Chair (Film) for CVM25

Co-Chair (Media Gallery) for VINCI25

Co-Chair (Registeration) and Session Chair for UIST24

Leading/key Organizer: Workshop on Agentic AI for Visual Media at CVPR26

Leading/key Curator: 2025 Hong Kong HKUST AI Film Festival, 2023 Paris ShortFest AI Film Festival.

Leading/key Organizer: Workshop on AI for Creative Video Editing and Understanding at SIGGRAPH25, CVPR25, CVPR24, ICCV23, ECCV22, ICCV21

Conference Reviewer: CVPR, ICCV, ECCV, ACCV, SIGGRAPH, SIGGRAPH Asia, Eurographics, CHI, UIST, MM, NeurIPS, ICML, ICLR, AAAI, IJCAI

Journal Reviewer: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), IEEE Transactions on Multimedia (TMM), IEEE Transactions on Visualization and Computer Graphics (TVCG), IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), ACM Transactions on Graphics (TOG), Transactions on Machine Learning Research (TMLR), International Journal of Computer Vision (IJCV)

Judge: The 3rd International Artificial Intelligence Fair

Research Experiences

Research Intern at Meta Reality Lab

Research Intern at Shanghai Artificial Intelligence Laboratory

Research Intern at SenseTime Research

Visitor at the University of Toronto and Vector Institute

Research Assistant at the Advanced Integration and Mining Lab, Eugene, OR, USA

Research Intern at University of Hong Kong, Hong Kong S.A.R.

Teaching Experiences

HKUST AMCC 5140 AI for Visual Arts and Creativity (AVAC)
HKUST AMCC 5150 Visual Computing for Visual Arts and Creativity (VCVAC)
HKUST AMCC 5250 Filmmaking with AI Innovations (FilmAI)
HKUST AMCC 5000 Creative Convergence: Foundations of Arts and Machine Creativity
HKUST AMCC 6950A Special Projects in Arts and Machine Creativity
HKUST EMIA 6950F Independent Study
HKUST EMIA 6500K Visual Computing for Visual Content Creation (VCVCC)
HKUST EMIA 6500H AI for Visual Content Creation (AIVCC)
SIGGRAPH Asia 2025 Course on Generative Models for Visual Content Editing and Creation
SIGGRAPH 2024 Course on Generative Models for Visual Content Editing and Creation
CCF 2024 Advanced Disciplines Lectures

Patents

A Video Generation Method, CN202210699177.X
A Video Editing Method and Related Program Products, CN202210691662.2
A Video Editing Method, CN202010694551.1
A Video Classification Method, CN202010694811.1
An Image Processing Method and Related Products, CN202010450801.3
A Zero-shot Action Recognition Method, CN202110821209.4
A Layout Generation Method, CN202111128