Content Adaptive Gigapixel Videography via Multi-scale Gimbal Camera
Room 2407 (Lifts 17-18), 2/F Academic Building, HKUST

Thesis Examination Committee

Prof Jungwon SEO, MAE/HKUST (Chairperson)
Prof Ling SHI, ECE/HKUST (Thesis Supervisor)
Prof Lu FANG, ECE/HKUST (Thesis Co-supervisor)



Gigapixel videography, beyond the resolution of single camera and human visual perception, works to capture large-scale dynamic scene with extremely high resolution. Restricted by the spatial-temporal bandwidth product of optical system, gigapixel videography is faced with the challenges of expensive and complicated optical, electronic and mechanical design, laborious calibration, massive data processing etc. Aiming for the flexible, efficient and economized system, we propose a content adaptive gigapixel videography through the novel unstructured dual-scale camera array scheme, i.e., the global reference camera with wide-angle lens works to capture the global scene, the local gimbal camera with telephoto lens works to obtain the local high-resolution details. The adaptation of local gimbal camera is enabled by the regions-of-interest (ROI) detection in global reference camera. The insight behind lies in the high sparsity of natural scene in both spatial and temporal domains.

More specifically, to retain high-resolution dynamic contents, a large-scale, multi-person and long-term tracking framework based on hierarchical image pyramid is introduced to sequentially select ROI. Meanwhile, a B-splines based motion planning algorithm for local gimbal camera is adopted to refine the motion trajectory. For synthesis of high-resolution video, we adopt multi-scale iterative feature matching and video warping with the huge resolution gap. The remaining static contents without local gimbal camera capturing are generated by Generative Adversarial Networks (GAN) based super resolution scheme. Experimental results conducted on the wild scenes demonstrate the flexibility and effectiveness of the content adaptive gigapixel videography scheme, enabling potential applications in various areas such as surveillance, aerial robots, autonomous driving etc.

Room 2407 (Lifts 17-18), 2/F Academic Building, HKUST
Event Format
Speakers / Performers:
Xiongfei SU
Department of Electronic & Computer Engineering
Post an event
Campus organizations are invited to add their events to the calendar.