Title
Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network
Abstract
Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional frame-based image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a large-scale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance.
Year
DOI
Venue
2022
10.1007/s11633-022-1350-3
Machine Intelligence Research
Keywords
DocType
Volume
Event camera, multi-modal fusion, image de-occlusion, spiking neural network (SNN), image reconstruction
Journal
19
Issue
ISSN
Citations 
4
2731-538X
0
PageRank 
References 
Authors
0.34
10
3
Name
Order
Citations
PageRank
Si-Qi Li100.34
Yue Gao23259124.70
Qionghai Dai33904215.66