SCIENCE CHINA Information Sciences, Volume 60, Issue 3: 032101(2017) https://doi.org/10.1007/s11432-015-0710-5

## Energy-based multi-view piecewise planar stereo

Wei WANG1,2,*, Lihua HU3,
• AcceptedFeb 3, 2016
• PublishedOct 8, 2016
Share
Rating

### Abstract

The piecewise planar model (PPM) is an effective means of approximating a complex scene by using planar patches to give a complete interpretation of the spatial points reconstructed from projected 2D images. The traditional piecewise planar stereo methods suffer from either a very restricted number of directions for plane detection or heavy reliance on the segmentation accuracy of superpixels. To address these issues, we propose a new multi-view piecewise planar stereo method in this paper. Our method formulates the problem of complete scene reconstruction as a multi-level energy minimization problem. To detect planes along principal directions, a novel energy formulation with pair-wise potentials is used to assign an optimal plane for each superpixel in an iterative manner, where reliable scene priors and geometric constraints are incorporated to enhance the modeling efficacy and inference efficiency. To detect non-principal-direction planes, we adopt a multi-direction plane sweeping with a restricted search space method to generate reliable candidate planes. To handle the multi-surface straddling problem of a single superpixel, a superpixel sub-segmenting scheme is proposed and a robust $P^n$ Potts model-like higher-order potential is introduced to refine the resulting depth map. Our method is a natural integration of pixel- and superpixel-level multi-view stereos under a unified energy minimization framework. Experimental results for standard data sets and our own data sets show that our proposed method can satisfactorily handle many challenging factors (e.g., slanted surfaces and poorly textured regions) and can obtain accurate piecewise planar depth maps.

### Funded by

National Natural Science Foundation of China(U1404622)

National Natural Science Foundation of China(U1404620)

Scientific Research Starting Foundation for Advanced Talents of Zhoukou Normal University(zknuc2015103)

National Natural Science Foundation of China(61273280)

Development Project of Henan Provincial Department of Science and Technology(152102310381)

National Natural Science Foundation of China(61333015)

National Natural Science Foundation of China(61421004)

National Natural Science Foundation of China(61103143)

### Acknowledgment

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant Nos. 61421004, 61333015, 61273280, 61103143, U1404620, U1404622), Development Project of Henan Provincial Department of Science and Technology (Grant No. 152102310381), and Scientific Research Starting Foundation for Advanced Talents of Zhoukou Normal University (Grant No. zknuc2015103).

### References

[1] Hong L, Chen G. Segment-based stereo matching using graph cuts. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, 2004. 74--81. Google Scholar

[2] Zitnick C L, Kang S B. Stereo for image-based rendering using image over-segmentation. Int J Comput Vision, 2007, 75: 49-65 CrossRef Google Scholar

[3] Klaus A, Sormann M, Karner K. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06), Hong Kong, 2006. 15--18. Google Scholar

[4] Wang Z F, Zheng Z G. A region based stereo matching algorithm using cooperative optimization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 2008. 1--8. Google Scholar

[5] Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell, 2001, 23: 1222-1239 CrossRef Google Scholar

[6] Bleyer M, Rhemann C, Rother C. PatchMatch stereo - stereo matching with slanted support windows. In: Proceedings of British Machine Vision Conference, Dundee, 2011. 1--11. Google Scholar

[7] \c{C}i\v{g}la C, Zabulis X, Alatan A A. Segment-based stereo-matching via plane and angle sweeping. In: Proceedings of 3DTV Conference, Kos Island, 2007. 1--4. Google Scholar

[8] Furukawa Y, Curless B, Seitz S, et al. Manhattan-world stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 1422--1429. Google Scholar

[9] Furukawa Y, Ponce J. Accurate, dense, and robust multi-view stereopsis. IEEE Trans Pattern Anal Mach Intell, 2009, 32: 1362-1376 Google Scholar

[10] Mičuš\'{\i}k B, Košecká J. Piecewise planar city 3D modeling from street view panoramic sequences. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 2906--2912. Google Scholar

[11] Mičuš\'{\i}k B, Košecká J. Multi-view superpixel stereo in urban environments. Int J Comput Vision, 2010, 89: 106-119 CrossRef Google Scholar

[12] Gallup D, Frahm J M, Mordohai P, et al. Real-time plane-sweeping stereo with multiple sweeping directions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007. 1--8. Google Scholar

[13] Gallup D, Frahm J M, Pollefeys M. Piecewise planar and non-planar stereo for urban scene reconstruction. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1418--1425. Google Scholar

[14] Sinha S N, Steedly D, Szeliski R. Piecewise planar stereo for image-based rendering. In: Proceedings of IEEE 12th International Conference on Computer Vision, Kyoto, 2009. 1881--1888. Google Scholar

[15] Kim H, Xiao H, Max N. Piecewise planar scene reconstruction and optimization for multi-view stereo. In: Proceedings of the 11th Asian Conference on Computer Vision. Berlin: Springer, 2012. 191--204. Google Scholar

[16] Chauve A L, Labatut P, Pons J P. Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, 2010. 1261--1268. Google Scholar

[17] Kowdle A, Sinha S N, Szeliski R. Multiple view object cosegmentation using appearance and stereo cues. In: Proceedings of the 12th European Conference on Computer Vision. Berlin: Springer, 2012. 789--803. Google Scholar

[18] Bodis-Szomoru A, Riemenschneider H, van Gool L. Fast, approximate piecewise-planar modeling based on sparse structure-from-motion and superpixel. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014. 469--476. Google Scholar

[19] Gould S, Fulton R, Koller D. Decomposing a scene into geometric and semantically consistent regions. In: Proceedings of IEEE 12th International Conference on Computer Vision, Kyoto, 2009. 1--8. Google Scholar

[20] Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell, 2002, 24: 603-619 CrossRef Google Scholar

[21] Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24: 381-395 CrossRef Google Scholar

[22] Isack H, Boykov Y. Energy-based geometric multi-model fitting. Int J Comput Vision, 2010, 92: 123-147 Google Scholar

[23] Delong A, Osokin A, Isack H N, et al. Fast approximate energy minimization with label costs. Int J Comput Vision, 2012, 96: 1-27 CrossRef Google Scholar

[24] Pham T T, Chin T J, Yu J, et al. The random cluster model for robust geometric fitting. IEEE Trans Pattern Anal Mach Intell, 2014, 36: 1658-1671 CrossRef Google Scholar

[25] Tola E, Strecha C, Fua P. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Mach Vision Appl, 2012, 23: 903-920 CrossRef Google Scholar

[26] Frey B J, Dueck D. Clustering by passing messages between data points. Science, 2007, 315: 972-976 CrossRef Google Scholar

[27] Kohli P, Ladicky L, Torr P H S. Robust higher order potentials for enforcing label consistency. Int J Comput Vision, 2009, 82: 302-324 CrossRef Google Scholar

Citations

• #### 0

Altmetric

Copyright 2019 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有