||Multi-Direction Hole-Filling Algorithm for Multi-View Stereoscopic Synthesis and Hardware Architecture
||Department of Electrical Engineering
In order to offer immersive view experience for viewer, 3D stereoscopic technology applications are more popular. One of the most widely discussed topics is (depth image-based rendering) DIBR. The DIBR system renders virtual views at different viewpoints using 3D warping process.
The DIBR calculates disparity value by depth value, and then moves the pixel of an object to generate new viewpoint image. During a 3D warping process, some parts of an occluded background region might be visible in a synthesized virtual image, i.e., a disoccluded region. It is referred to as a “hole”. The conventional way to solve this problem is using a smoothing filter to smooth depth map to reduce hole’s size before shifting, but it’s time-consuming. This thesis discards the traditional smoothing depth approach and proposes a novel depth map refinement scheme to remove blur pixels near the object boundary caused by restriction of camera and make the holes sit on the correct position. With the blur pixels removal, the hole-filling will become more reliable. The holes after warping process are filled by the proposed multi-direction hole-filling algorithms based on direction of edge. It can preserve the texture of background. The experimental results show that the visual quality of our proposal is competitive to others contemporary methods.
摘 要 I
誌 謝 III
List of Figures VI
List of Tables VIII
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Organization of the Thesis 2
Chapter 2 Background 3
2.1 Depth Image-based Rendering System 3
2.1.1 Pre-processing 4
2.1.2 Warping 5
2.1.3 Hole-filling 11
2.2 Smooth-depth-based Depth Image-based Rendering 12
2.2.1 Symmetric and asymmetric Gaussian smoothing filter 14
2.2.2 Parallax-map-based DIBR 15
2.2.3 Adaptive edge-oriented smoothing filter 15
Chapter 3 The proposed multi-view system 18
3.1 Overview of proposed multi-view system 18
3.2 Proposed M1 20
3.2.1 Boundary detection and refinement 20
3.2.2 Hybrid warping and crack hole-filling 24
3.2.3 Multi-direction hole-filling 30
3.3 Proposed M2 38
Chapter 4 Hardware implements 40
4.1 Overview of hardware architecture 40
4.2 Hardware architecture of the proposed method_1 43
4.3 Hardware architecture of the proposed method_2 46
Chapter 5 Experimental results 47
5.1 Objective quantitative results and subjective results 50
5.2 Hardware synthesis 63
Chapter 6 Conclusions 65
C. Fehn, K. Hopf, and B. Quante, ”Key technologies for an advanced 3D TV system,” in Proc. SPIE Three-Dimensional TV, Video, and Display III, pp. 66-80, Oct. 2004,
C. Fehn, “Depth-image-based-rendering (DIBR), compression and transmission for a new approach on 3D-TV,” in Proc. International Society for Optical Engineering Conf. Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, vol.5291, pp.93-104, Jan. 2004,.
K. J. Yoon and S. Kweon, “Adaptive support-weight approach for correspondence search” IEEE Trans. Pattern Anal. Mach. Intell., vol.28, pp.650-656, Apr. 2006,.
M. C. Yang, J. Y. Liu, Y. C. Yang, and K. H. Chen, “A qualty-improved stereo matching by using incrementally-averaging orthogonally-scanned edge detection,” in Proc. Int. Conf. 3DSA, pp. 489-491, June 2012.
K. Zhang, J. Lu, and G. Lafruit, “Cross-based local stereo matching using orthogonal integral images,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, pp. 1073-1079, July 2009.
D. Kim, D. Min, and K. Sohn, “A stereoscopic video generation method using stereoscopic display characterization and motion analysis,” IEEE Trans. Broadcast., vol. 54, pp. 188-197, June 2008.
J. Lee, S. Yoo, M. Kang, C. Chun, and C. Kim, “Depth estimation from a single image using salient objects detection,” in Proc. Int. Conf. 3DSA, pp. 264-267, June 2012.
M. T. Pourazad, P. Nasiopoulos, and R. K. Ward, “An H.264-based scheme for 2D to 3D video conversion,” IEEE Trans. Consum. Electron., vol. 55, pp. 742-748, May 2009.
Y. Feng, J. Ren, and J. Jiang, “Object-based 2D-to-3D video conversion for effective stereoscopic content generation in 3D-TV applications,” IEEE Trans. Broadcast., vol. 57, pp. 500-509, June 2011.
Y. C. Fan and T. C. Chi, “The novel non-hole-filling approach of depth image based rendering,” in Proc. IEEE 3DTV, pp. 325-328, May 2008.
C. Fehn, “A 3D-TV approach using depth-image-based rendering (DIBR),” in Proc. Vis., Imaging, and Image Process., pp. 482-487, Sept. 2003.
A. Redert, M. O. de Beeck, C. Fehn, W. Ijsselsteijn, M. Pollefeys, L. V. Gool, E. Ofek, I. Sexton, and P. Surman, “ATTEST-advanced three-dimensional television system technologies,” in Proc. IEEE 3DPVT, pp. 313-319, Jan. 2002.
W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, “Smoothing depth maps for improved stereoscopic image quality,” in Proc. SPIE Three-Dimensional TV, Video, and Display Ⅲ, pp. 162-172, Oct. 2004.
A. Woods, T. Docherty, and R. Koch, “Image distortions in stereoscopic video systems,” in Proc. SPIE Stereoscopic Displays and Appl., pp. 36-48, Feb. 1993.
W. A. IJsselsteijn, H. de Ridder, and J. Vliegen, “Effects of stereoscopic filming parameters and display duration on the subjective assessment of eye strain,” in Proc. SPIE Stereoscopic Displays and Virtual Reality Syst., pp. 12-22, Apr. 2000.
I. Daribo and H. Saito, “A novel inpainting-based layered depth video for 3DTV,” IEEE Trans. Broadcast., vol. 57, pp. 533-541, June 2011.
P. Ndjiki-Nya, M. Koppel, D. Doshkov, H. Lakshman, P. Merkle, K. Muller, and T. Wiegand, “Depth image-based rendering with advanced texture synthesis for 3-D video,” IEEE Trans. Multimedia, vol. 13, pp. 453-465, June 2011.
M. Solh and G. AlRegib, “Hierarchical hole-filling (HHF): depth image based rendering without depth map filtering for 3D-TV,” in Proc. IEEE MMSP, pp. 87-92, Oct. 2010.
M. S. Ko, D. W. Kim, D. L. Jones, J. Yoo, “A new common-hole filling algorithm for arbitrary view synthesis,” in Proc. Int. Conf. 3DSA, pp. 242-245, June 2012.
A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, pp. 1200-1212, Sept. 2004.
L. Zhang and W. J. Tam, “Stereoscopic image generation based on depth images for 3DTV,” IEEE Trans. Broadcast., vol. 51, pp. 191-199, June 2005.
W. Y. Chen, Y. L. Chang, S. F. Lin, L. F. Ding, and L. G. Chen, “Efficient depth image based rendering with edge dependent depth filter and interpolation,” in Proc. IEEE ICME, pp. 1314-1317, July 2005.
T. C. Lin, H. C. Huang, and Y. M. Huang, “Preserving depth resolution of synthesized images using parallax-map-based DIBR for 3D-TV,” IEEE Trans. Consum. Electron., vol. 56, pp. 720-727, May 2010.
H. Dong, S. Jianfei, and X. Ping, “Improvement of virtual view rendering based on depth image,” in Proc. IEEE ICIG, pp. 254-257, Aug. 2011.
P. J. Lee and Effendi, “Nongeometric distortion smoothing approach for depth map preprocessing,” IEEE Trans. Multimedia, vol. 13, pp. 246-254, Apr. 2011.
L. H. Wang, X. J. Huang, M. Xi, D. X. Li, and M. Zhang, “An asymmetric edge adaptive filter for depth generation and hole filling in 3DTV,” IEEE Trans. Broadcast., vol. 56, pp. 425-431, Sept. 2010.
T. C. Yang, P. C. Kuo, B. D. Liu and J. F. Yang, “Depth Image-Based Rendering with Edge-oriented Hole Filling for Multiview Synthesis," in Proc. IEEE ICCCAS, pp. 50-53, Nov 2013.
B. E. Bayer, “Color imaging array,” U.S. Patent 3 971 065, July 20, 1976.
W.-Y. Lee, “Improved Hole-filling Algorithms and Hardware Implementation for Virtual View Generation,” Master’s Thesis, Dept. of Electrical Engineering, National Tsing Hua University, July 2011.
S. J. Yao, P. F. Jin, H. Fu, L. H. Wang, D. X. Li, M. Zhang, “Real-Time 3DTV System for Autostereoscopic Displays,” in Proc. IEEE ICALIP, pp. 621-626, July 2012
H.-J. Chen, F.-H Lo, F.-C. Jan, and S.-D. Wu,“Real-Time Multiview Rendering Architecture for Autostereoscopic Displays,” in Proc. IEEE Intl, Symp. Circuits and Systems (ISCAS), pp. 1165-1168, May 2010.