Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion

GUAN Jin-chao; DING Ling; YANG Xu; LIU Peng-fei; WANG Hai-nian

doi:10.19818/j.cnki.1671-1637.2024.03.010

Volume 24 Issue 3

Jun. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Traffic and Transportation Engineering > 2024 > 24(3): 154-170

Next Previous

GUAN Jin-chao, DING Ling, YANG Xu, LIU Peng-fei, WANG Hai-nian. Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion[J]. Journal of Traffic and Transportation Engineering, 2024, 24(3): 154-170. doi: 10.19818/j.cnki.1671-1637.2024.03.010

Citation:

GUAN Jin-chao, DING Ling, YANG Xu, LIU Peng-fei, WANG Hai-nian. Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion[J]. Journal of Traffic and Transportation Engineering, 2024, 24(3): 154-170. doi: 10.19818/j.cnki.1671-1637.2024.03.010

Citation:

PDF( 33041 KB)

Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion

doi: 10.19818/j.cnki.1671-1637.2024.03.010

GUAN Jin-chao^{1, 2
,},
DING Ling^{3
,
,},
YANG Xu^{1, 4},
LIU Peng-fei⁵,
WANG Hai-nian¹

1.
School of Highway, Chang'an University, Xi'an 710064, Shaanxi, China
2.
Zhejiang Expressway Co., Ltd., Hangzhou 310020, Zhejiang, China
3.
School of Transportation Engineering, Chang'an University, Xi'an 710064, Shaanxi, China
4.
School of Future Transportation, Chang'an University, Xi'an 710064, Shaanxi, China
5.
Institute of Highway Engineering, RWTH Aachen University, Aachen 52074, North Rhine-Westphalia, Germany

Funds:

National Key Research and Development Program of China 2021YFB2601000

National Natural Science Foundation of China 52078049

More Information

Author Bio:
GUAN Jin-chao(1995-), male, postdoctor, guan@chd.edu.cn

DING Ling(1986-), female, assistant professor, PhD, dingling@chd.edu.cn
Received Date: 2024-01-19
Available Online: 2024-07-18
Publish Date: 2024-06-30

Abstract

Abstract

To improve the accuracy and robustness of crack and pothole detection of pavement surface in complex scenarios, the morphological irregularity of pavement surface distresses and the influence of environmental noises in practical detection scenarios were considered, and an automatic pavement surface distress segmentation model and feature fusion optimization method for multi-dimensional images were proposed. Based on high-precision pavement surface point cloud models reconstructed by multi-view stereo vision, 2D and 3D images were generated by the rasterization of homologous point clouds. The pavement surface distress image dataset in complex scenarios was established. A lightweight encoding-decoding network, namely PDU-net, integrating depthwise separable convolution and multi-layer feature combination, was developed for pixel-level crack and pothole detection. Based on the segmentation model, two multi-dimensional image fusion strategies, including pixel operation and channel recombination, were proposed to improve the extraction efficiency of deep learning networks in shallow and fine crack features. Experimental results show that the PDU-net model can effectively learn features from different types of images and distresses. The training loss of the PDU-net on different datasets can converge stably, with the training cycles of 3D images shorter than that of 2D images. Compared with existing convolutional segmentation networks, the PDU-net model achieves higher accuracy and efficiency for pavement surface distress segmentation in complex scenarios. The harmonic means of 3D crack and pothole image segmentation are 81.00% and 95.85%, respectively. The average forward inference time of the PDU-net is about 30% of the existing models. The segmentation accuracy and robustness of complex cracks can be improved by multi-dimensional fusion images. When the optimal color-depth ratio is 0.2, the harmonic mean of the crack segmentation increases to 83.31%. In conclusion, the proposed method can effectively suppress environmental noises and strengthen the surface distress features in complex scenarios.

FullText(HTML)

References(31)

References

[1]	Ministry of Transport of the People's Republic of China. Statistical bulletin on transportation industry development in 2022[R]. Beijing: Ministry of Transport of the People's Republic of China, 2022. (in Chinese)
[2]	YANG Xu, GUAN Jin-chao, DING Ling, et al. Research and applications of artificial neural network in pavement engineering: a state-of-the-art review[J]. Journal of Traffic and Transportation Engineering (English Edition), 2021, 8(6): 1000-1021. doi: 10.1016/j.jtte.2021.03.005
[3]	XU Zhi-gang, CHE Yan-li, LI Jin-long, et al. Research progress on automatic image processing technology for pavement distress[J]. Journal of Traffic and Transportation Engineering, 2019, 19(1): 172-190. (in Chinese) doi: 10.3969/j.issn.1671-1637.2019.01.017
[4]	ZHANG Ce, NATEGHINIA E, MIRANDA-MORENO L F, et al. Pavement distress detection using convolutional neural network (CNN): a case study in Montreal, Canada[J]. International Journal of Transportation Science and Technology, 2022, 11(2): 298-309. doi: 10.1016/j.ijtst.2021.04.008
[5]	LI Qing-quan, HU Qing-wu. A pavement crack image analysis approach based on automatic image dodging[J]. Journal of Highway and Transportation Research and Development, 2010, 27(4): 1-5, 27. (in Chinese) doi: 10.3969/j.issn.1002-0268.2010.04.001
[6]	SUN Zhao-yun, ZHAO Hai-wei, LI Wei, et al. 3D pavement crack identification method based on dual-phase scanning detection[J]. China Journal of Highway and Transport, 2015, 28(2): 26-32. (in Chinese) doi: 10.3969/j.issn.1001-7372.2015.02.004
[7]	JO Y, RYU S K, KIM Y R. Pothole detection based on the features of intensity and motion[J]. Transportation Research Record, 2016, 2595(1): 18-28. doi: 10.3141/2595-03
[8]	SOLLAZZO G, WANG K C P, BOSURGI G, et al. Hybrid procedure for automated detection of cracking with 3D pavement data[J]. Journal of Computing in Civil Engineering, 2016, 30(6): 04016032. doi: 10.1061/(ASCE)CP.1943-5487.0000597
[9]	YIN Guan-sheng, GAO Jian-guo, SHI Ming-hui, et al. Tunnel crack recognition method under image block[J]. Journal of Traffic and Transportation Engineering, 2022, 22(2): 148-159. (in Chinese) doi: 10.19818/j.cnki.1671-1637.2022.02.011
[10]	PARK S, BANG S, KIM H, et al. Patch-based crack detection in black box images using convolutional neural networks[J]. Journal of Computing in Civil Engineering, 2019, 33(3): 04019017. doi: 10.1061/(ASCE)CP.1943-5487.0000831
[11]	HUYAN Ju, LI Wei, TIGHE S, et al. Detection of sealed and unsealed cracks with complex backgrounds using deep convolutional neural network[J]. Automation in Construction, 2019, 107: 102946. doi: 10.1016/j.autcon.2019.102946
[12]	ZHANG Zhi-hua, DENG Yan-xue, ZHANG Xin-xiu. A method for detecting and differentiating asphalt pavement distress based on an improved SegNet algorithm[J]. Journal of Transport Information and Safety, 2022, 40(3): 127-135. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2022.03.013
[13]	YANG Fan, ZHANG Lei, YU Si-jia, et al. Feature pyramid and hierarchical boosting network for pavement crack detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(4): 1525-1535. doi: 10.1109/TITS.2019.2910595
[14]	TONG Zheng, YUAN Dong-dong, GAO Jie, et al. Pavement defect detection with fully convolutional network and an uncertainty framework[J]. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(8): 832-849. doi: 10.1111/mice.12533
[15]	HUYAN Ju, LI Wei, TIGHE S, et al. CrackU-net: a novel deep convolutional neural network for pixelwise pavement crack detection[J]. Structural Control and Health Monitoring, 2020, 27(8): e2551.
[16]	CHEN Han-shen, LIN Hui-ping, YAO Ming-hai. Improving the efficiency of encoder-decoder architecture for pixel-level crack detection[J]. IEEE Access, 2019, 7: 186657-186670. doi: 10.1109/ACCESS.2019.2961375
[17]	MATHAVAN S, KAMAL K, RAHMAN M. A review of three-dimensional imaging technologies for pavement distress detection and measurements[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(5): 2353-2362. doi: 10.1109/TITS.2015.2428655
[18]	CAO Wen-ming, LIU Qi-fan, HE Zhi-quan. Review of pavement defect detection methods[J]. IEEE Access, 2020, 8: 14531-14544. doi: 10.1109/ACCESS.2020.2966881
[19]	ZHANG De-jin, ZOU Qin, LIN Hong, et al. Automatic pavement defect detection using 3D laser profiling technology[J]. Automation in Construction, 2018, 96: 350-365. doi: 10.1016/j.autcon.2018.09.019
[20]	DING Shi-hai, ZHAN You, YANG En-hui, et al. MTD measurement of asphalt pavement based on high precision laser section elevation[J]. Journal of Southeast University (Natural Science Edition), 2020, 50(1): 137-142. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-DNDX202001018.htm
[21]	CHEN Jia-ying, HUANG Xiao-ming, ZHENG Bin-shuang, et al. Real-time identification system of asphalt pavement texture based on the close-range photogrammetry[J]. Construction and Building Materials, 2019, 226: 910-919. doi: 10.1016/j.conbuildmat.2019.07.321
[22]	ZHANG A, WANG K C P, LI Bao-xian, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network[J]. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(10): 805-819. doi: 10.1111/mice.12297
[23]	ZHANG A, WANG K C P, FEI Yue, et al. Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet[J]. Journal of Computing in Civil Engineering, 2018, 32(5): 04018041. doi: 10.1061/(ASCE)CP.1943-5487.0000775
[24]	ZHANG A, WANG K C P, FEI Yue, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces with a recurrent neural network[J]. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(3): 213-229. doi: 10.1111/mice.12409
[25]	FEI Yue, WANG K C P, ZHANG A, et al. Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(1): 273-284. doi: 10.1109/TITS.2019.2891167
[26]	GUAN Jin-chao, YANG Xu, DING Ling, et al. Automated pixel-level pavement distress detection based on stereo vision and deep learning[J]. Automation in Construction, 2021, 129: 103788. doi: 10.1016/j.autcon.2021.103788
[27]	ZENG Qing-hong, LU De-tang. Curve and surface fitting based on moving least-squares methods[J]. Journal of Graphics, 2004, 25(1): 84-89. (in Chinese) doi: 10.3969/j.issn.1003-0158.2004.01.017
[28]	HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vision applications[J]. arXiv, 2017, DOI: 10.48550/arXiv.1704.04861.
[29]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]// NAVAB N, HORNEGGER J, WELLS W, et al. 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 2015: 234-241.
[30]	LAU S L H, CHONG E K P, YANG X, et al. Automated pavement crack segmentation using U-Net-based convolutional neural network[J]. IEEE Access, 2020, 8: 114892. doi: 10.1109/ACCESS.2020.3003638
[31]	CHEN Jie, LIU Gang, CHEN Xin. Road crack image segmentation using global context U-net[C]//GOKHALE A, TAN Y. Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence. New York: Association for Computing Machinery, 2019: 181-185.

Relative Articles

Supplements(0)

Cited By

Get Citation

PDF

XML

Article Metrics

Article views (578) PDF downloads(58)

Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion

doi: 10.19818/j.cnki.1671-1637.2024.03.010

Abstract

References

Catalog

Article Metrics

Related

Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion

doi: 10.19818/j.cnki.1671-1637.2024.03.010

Abstract

References

Catalog

Article Metrics

Related

Export File

Citation

Format

Content