Name
Abstract | ||
|---|---|---|
The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR. |
Year | DOI | Venue |
|---|---|---|
2020 | 10.1109/TPAMI.2019.2961672 | IEEE Transactions on Pattern Analysis and Machine Intelligence |
Keywords | DocType | Volume |
weighted analysis sparse representation model,guided depth image enhancement,previous optimization-based models,WASR model,conduct guided depth reconstruction,dynamically changed stage-wise operations,guidance strategy,reconstruction quality,Gaussian RBF nonlinearity parameterization,CNN nonlinearity parameterization,DG-RBF,DG-CNN methods,optimal network parameters,paired training data,optimization-inspired network architectures,guided depth image super-resolution,realistic depth image reconstruction tasks,learned dynamic guidance,consumer depth sensors,insufficient quality,high resolution RGB camera,guided depth reconstruction problems | Journal | 42 |
Issue | ISSN | Citations |
10 | 0162-8828 | 4 |
PageRank | References | Authors |
0.38 | 27 | 7 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Shuhang Gu | 1 | 701 | 28.25 |
| Shi Guo | 2 | 49 | 3.41 |
| Wangmeng Zuo | 3 | 3833 | 173.11 |
| Yunjin Chen | 4 | 407 | 14.89 |
| Radu Timofte | 5 | 1880 | 118.45 |
| Luc Van Gool | 6 | 27566 | 1819.51 |
| Lei Zhang | 7 | 16326 | 543.99 |