Paper Info
Title
Joint Semantic Segmentation And 3d Reconstruction From Monocular Video
Abstract
We present an approach for joint inference of 3D scene structure and semantic labeling for monocular video. Starting with monocular image stream, our framework produces a 3D volumetric semantic + occupancy map, which is much more useful than a series of 2D semantic label images or a sparse point cloud produced by traditional semantic segmentation and Structure from Motion(SfM) pipelines respectively. We derive a Conditional Random Field (CRF) model defined in the 3D space, that jointly infers the semantic category and occupancy for each voxel. Such a joint inference in the 3D CRF paves the way for more informed priors and constraints, which is otherwise not possible if solved separately in their traditional frameworks. We make use of class specific semantic cues that constrain the 3D structure in areas, where multiview constraints are weak. Our model comprises of higher order factors, which helps when the depth is unobservable. We also make use of class specific semantic cues to reduce either the degree of such higher order factors, or to approximately model them with unaries if possible. We demonstrate improved 3D structure and temporally consistent semantic segmentation for difficult, large scale, forward moving monocular image sequences.[GRAPHICS].
Year
DOI
Venue
2014
10.1007/978-3-319-10599-4_45
COMPUTER VISION - ECCV 2014, PT VI
Keywords
Field
DocType
Conditional Random Field,High Order Factor,Structure From Motion,Semantic Label,Conditional Random Field Model
Structure from motion,Conditional random field,Voxel,Computer vision,Inference,Computer science,Segmentation,Artificial intelligence,Prior probability,Point cloud,3D reconstruction
Conference
Volume
ISSN
Citations 
8694
0302-9743
63
PageRank 
References 
Authors
1.77
28
5
Name
Order
Citations
PageRank
Abhijit Kundu11356.37
Yin Li279735.85
Frank Dellaert35242438.33
Fuxin Li477252.53
James M. Rehg55259474.66