||An Enhanced Recurrent Neural Network for Image Deraining
||Institute of Computer & Communication
rain streak removal
recurrent neural network
channel attention mechanism
Computer vision algorithms like object detection, image classification have changed our life significantly in various applications. However, these algorithms cannot usually have good performance in practical applications due to the fact that the unpredictable degradations often occur in realistic scene, for instance, noise, illumination, and severe weather conditions. Commonly seen weather conditions such as rain, snow, and sandstorm can adversely affect the performance of many computer vision tasks.
In this Thesis, a rain removal algorithm based on recurrent neural network is proposed to remove rain streak stage by stage. A Rain Streak Prediction Network is proposed to predict the rain streak part of a rainy image, providing more information to deraining. A residual dense block combining with channel attention mechanism, called RDCAB is used to enhance the ability of deraining. Experimental results show that the proposed method gets more nature textures and details compared with available methods.
List of Tables vi
List of Figures vii
Chapter 1 Introduction 1
Chapter 2 Background and Related Works 4
2.1 Rain Physical Properties & Rain Model 4
2.2 Related Work 6
2.3 Recurrent Neural Network 10
2.3.1 Simple Recurrent Neural Network 10
2.3.2 Long Short-Term Memory 11
2.3.3 Gate Recurrent Unit 13
Chapter 3 The Proposed Algorithm 15
3.1 Proposed Network Architecture 17
3.1.1 Rain Streak Prediction Network 18
3.1.2 Deraining Recurrent Neural Network 19
184.108.40.206 Convolutional Gated Recurrent Unit 20
220.127.116.11 Residual Dense Channel Attention Block 21
3.2 Loss Function 23
3.2.1 MSE Loss 23
3.2.2 SSIM Loss 24
3.2.3 Perceptual Loss 24
3.2.4 Total Loss Function 25
Chapter 4 Experimental Results 28
4.1 Experimental Dataset 28
4.2 Parameter and Experimental Setting 31
4.3 Experimental Results of Simulated Images 32
4.4 Ablation Experimental Results 41
4.5 Application 44
Chapter 5 Conclusion and Future Work 46
5.1 Conclusion 46
5.2 Future Work 46
 K. Garg and S. K. Nayar. Detection and removal of rain from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pages I–I,2004.
 Zhang, X., Li, H., Qi, Y., Leow, W.K., Ng, T.K. Rain removal in video by combining temporal and chromatic properties. In IEEE ICME, pages 461–464, 2006.
 P. Barnum, S. Narasimhan, T. Kanade. Analysis of rain and snow in frequency space. International Journal of Computer Vision 86 (2-3) (2010) 256–274.
 J. Bossu, N. Hautiere, and J.-P. Tarel. Rain or snow detection in image sequences through use of a histogram of orientation of streaks. IJCV, 93(3):348–367, 2011.
 C. Stauffer, W. Grimson, Learning patterns of activity using realtime tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (8) (2000) 747–757.
 Santhaseelan, V., Asari, V.K.: Utilizing local phase information to remove rain from video. International Journal of Computer Vision 112(1) (2015) 71–89.
 Li-Wei Kang, Chia-Wen Lin, Yu-Hsiang Fu. Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition. In IEEE Transactions on Image Processing, vol. 21, no. 4, 2013.
 Yi-Lei Chen, Chiou-Ting Hsu. A Generalized Low-Rank Appearance Model for Spatio-Temporally Correlated Rain Streaks. In IEEE International Conference on Computer Vision (ICCV), 2013.
 Y. Li, R. Tan, X. Guo, J. Lu, and M. Brown. 2016. Rain Streak Removal Using Layer Priors. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
 Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, John Paisley. Removing rain from single images via a deep detail network. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
 K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
 Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan. Deep Joint Rain Detection and Removal from a Single Image. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
 Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. arXiv preprint arXiv:1807.05698, resca2018.
 Fisher Yu, Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
 Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017.
 H. Zhang, V. A. Sindagi, and V. M. Patel. Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957, 2017.
 M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Neural Information Processing Systems (NIPS), 2014.
 K. Garg and S. K. Nayar. Vision and rain. International Journal of Computer Vision, 75(1):3–27, 2007.
 L. W. Kang, C. W. Lin, and Y. H. Fu, “Automatic singleimage-based rain streaks removal via image decomposition,” IEEE Trans. on Image Processing, vol. 21, no. 4, pp. 1742– 1755, April 2012.
 A. Tripathi and S. Mukhopadhyay, “Video post processing: low-latency spatiotemporal approach for detection and removal of rain,” IET image processing, vol. 6, no. 2, pp. 181–196, 2012.
 C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, ´ A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802, 2016.
 P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
 Z. Yi, H. Zhang, T. Gong, Tan, and M. Gong. Dualgan. Unsupervised dual learning for image-to-image translation. In IEEE International Conference on Computer Vision (ICCV), 2017.
 J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems (NIPS), 2017.
 D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
 Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. arXiv preprint arXiv:1804.07723, 2018.
 J. Wang, X. Li, and J. Yang. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
 R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, June 2018.
 X. Fu, B. Liang, Y. Huang, X. Ding, and J. Paisley, “Lightweight pyramid networks for image deraining,” IEEE Trans. on Neural Networks and Learning Systems, pp. 1–14, 2019
 X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6):2944–2956, 2017.
 S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997
 K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, ¨ F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014
 W. Yang, R. T. Tan, J. Feng, J. Liu, S. Yan, and Z. Guo, “Joint rain detection and removal from a single image with contextualized deep networks,” IEEE Trans. on Pattern Analysis and Machine Intelligence, pp. 1–1, 2019.
 R. Li, L.-F. Cheong, and R. T. Tan, “Heavy rain image restoration: Integrating physics model and conditional adversarial learning,” in Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, June 2019.
 X. Hu, C.-W. Fu, L. Zhu, and P.-A. Heng, “Depth-attentional features for single-image rain removal,” in Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, June 2019.
 Youzhao Yang and Hong Lu, "Single image deraining using a recurrent multi-scale aggregation and enhancement network", Proceedings of the IEEE International Conference on Multimedia and Expo, 2019.
 Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)
 G. Huang, Z. Liu, K. Q. Weinberger, and L. Maaten. Densely connected convolutional networks. In CVPR, 2017.
 K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
 J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. 2016.
 D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng, “Progressive image deraining networks: a better and simpler baseline,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3937–3946