Pavement surface distress detection in complex scenarios driven by multi-dimensional image fusion
-
摘要: 为提升复杂场景中路表裂缝与坑槽的识别精度和鲁棒性，考虑实际检测场景中路表破损形态的不规则性和环境噪声干扰，提出了一种面向多维图像的路表破损自动分割模型与特征融合优化方法；基于多目立体视觉重构的路表高精度点云模型，通过同源点云栅格化生成二、三维图像，建立了复杂场景路表破损图像数据集；结合深度可分离卷积和多层位特征叠加，构造了轻量化编码-解码网络PDU-net，用于像素级裂缝与坑槽识别；在分割模型基础上，提出了像素运算和通道重组2种多维图像融合策略，以提升深度学习网络对浅细裂缝特征的提取效率。试验结果表明：PDU-net模型能够有效学习不同类型图像和破损特征，在不同数据集上的训练损失均能稳定收敛，其中三维图像训练周期小于二维图像；相较于现有卷积分割网络，PDU-net模型在复杂场景下的路表破损分割精度和效率更高，三维裂缝与坑槽图像分割的调和均值分别为81.00%和95.85%，平均正向推理时间约为现有模型的30%；多维融合图像可以提升复杂裂缝分割的精度和鲁棒性，在最优色彩-深度比为0.2时，裂缝分割的调和均值可提升至83.31%。综上所述，所提出的方法可在复杂场景中有效抑制环境噪声并强化病害特征。Abstract: To improve the accuracy and robustness of crack and pothole detection of pavement surface in complex scenarios, the morphological irregularity of pavement surface distresses and the influence of environmental noises in practical detection scenarios were considered, and an automatic pavement surface distress segmentation model and feature fusion optimization method for multi-dimensional images were proposed. Based on high-precision pavement surface point cloud models reconstructed by multi-view stereo vision, 2D and 3D images were generated by the rasterization of homologous point clouds. The pavement surface distress image dataset in complex scenarios was established. A lightweight encoding-decoding network, namely PDU-net, integrating depthwise separable convolution and multi-layer feature combination, was developed for pixel-level crack and pothole detection. Based on the segmentation model, two multi-dimensional image fusion strategies, including pixel operation and channel recombination, were proposed to improve the extraction efficiency of deep learning networks in shallow and fine crack features. Experimental results show that the PDU-net model can effectively learn features from different types of images and distresses. The training loss of the PDU-net on different datasets can converge stably, with the training cycles of 3D images shorter than that of 2D images. Compared with existing convolutional segmentation networks, the PDU-net model achieves higher accuracy and efficiency for pavement surface distress segmentation in complex scenarios. The harmonic means of 3D crack and pothole image segmentation are 81.00% and 95.85%, respectively. The average forward inference time of the PDU-net is about 30% of the existing models. The segmentation accuracy and robustness of complex cracks can be improved by multi-dimensional fusion images. When the optimal color-depth ratio is 0.2, the harmonic mean of the crack segmentation increases to 83.31%. In conclusion, the proposed method can effectively suppress environmental noises and strengthen the surface distress features in complex scenarios.
-
图 6 基于像素运算的多维融合图像生成
Figure 6. Multi-dimensional fusion image generation based on pixel operation
图 7 基于通道重组的多维融合图像生成
Figure 7. Multi-dimensional fusion image generation based on channel recombination
图 9 路表破损图像分割定量评价
Figure 9. Quantitative evaluation of pavement surface distress image segmentation
图 14 像素运算多维融合图像分割性能定量评价
Figure 14. Quantitative evaluation of multi-dimensional fusion image segmentation with pixel operation
图 15 不同裂缝图像分割精度分布
Figure 15. Distributions of different crack image segmentation accuracies
图 16 通道重组多维融合图像分割性能定量评价
Figure 16. Quantitative evaluation of multi-dimensional fusion image segmentation with channel recombination
图 17 多维融合裂缝图像分割可视化
Figure 17. Visualization of multi-dimensional fusion image segmentation
表 1 不同模型的计算效率
Table 1. Computational efficiencies of different models
模型名称 模型参数量/106 模型体积/MB 推理速度/s 每秒处理图像张数 PDU-net 6.3 24.3 0.038 26.3 GCU-net 10.3 39.4 0.133 7.5 U-net 10.1 38.8 0.122 8.2 
下载: 导出CSV
-
[1] 中华人民共和国交通运输部. 2021年交通运输行业发展统计公报[R]. 北京: 中华人民共和国交通运输部, 2022.Ministry of Transport of the People's Republic of China. Statistical bulletin on transportation industry development in 2022[R]. Beijing: Ministry of Transport of the People's Republic of China, 2022. (in Chinese) [2] YANG Xu, GUAN Jin-chao, DING Ling, et al. Research and applications of artificial neural network in pavement engineering: a state-of-the-art review[J]. Journal of Traffic and Transportation Engineering (English Edition), 2021, 8(6): 1000-1021. doi: 10.1016/j.jtte.2021.03.005 [3] 徐志刚, 车艳丽, 李金龙, 等. 路面破损图像自动处理技术研究进展[J]. 交通运输工程学报, 2019, 19(1): 172-190. doi: 10.3969/j.issn.1671-1637.2019.01.017XU Zhi-gang, CHE Yan-li, LI Jin-long, et al. Research progress on automatic image processing technology for pavement distress[J]. Journal of Traffic and Transportation Engineering, 2019, 19(1): 172-190. (in Chinese) doi: 10.3969/j.issn.1671-1637.2019.01.017 [4] ZHANG Ce, NATEGHINIA E, MIRANDA-MORENO L F, et al. Pavement distress detection using convolutional neural network (CNN): a case study in Montreal, Canada[J]. International Journal of Transportation Science and Technology, 2022, 11(2): 298-309. doi: 10.1016/j.ijtst.2021.04.008 [5] 李清泉, 胡庆武. 基于图像自动匀光的路面裂缝图像分析方法[J]. 公路交通科技, 2010, 27(4): 1-5, 27. doi: 10.3969/j.issn.1002-0268.2010.04.001LI Qing-quan, HU Qing-wu. A pavement crack image analysis approach based on automatic image dodging[J]. Journal of Highway and Transportation Research and Development, 2010, 27(4): 1-5, 27. (in Chinese) doi: 10.3969/j.issn.1002-0268.2010.04.001 [6] 孙朝云, 赵海伟, 李伟, 等. 基于双相扫描检测的路面三维裂缝识别方法[J]. 中国公路学报, 2015, 28(2): 26-32. doi: 10.3969/j.issn.1001-7372.2015.02.004SUN Zhao-yun, ZHAO Hai-wei, LI Wei, et al. 3D pavement crack identification method based on dual-phase scanning detection[J]. China Journal of Highway and Transport, 2015, 28(2): 26-32. (in Chinese) doi: 10.3969/j.issn.1001-7372.2015.02.004 [7] JO Y, RYU S K, KIM Y R. Pothole detection based on the features of intensity and motion[J]. Transportation Research Record, 2016, 2595(1): 18-28. doi: 10.3141/2595-03 [8] SOLLAZZO G, WANG K C P, BOSURGI G, et al. Hybrid procedure for automated detection of cracking with 3D pavement data[J]. Journal of Computing in Civil Engineering, 2016, 30(6): 04016032. doi: 10.1061/(ASCE)CP.1943-5487.0000597 [9] 尹冠生, 高建国, 史明辉, 等. 图像分块下的隧道裂缝识别方法[J]. 交通运输工程学报, 2022, 22(2): 148-159. doi: 10.19818/j.cnki.1671-1637.2022.02.011YIN Guan-sheng, GAO Jian-guo, SHI Ming-hui, et al. Tunnel crack recognition method under image block[J]. Journal of Traffic and Transportation Engineering, 2022, 22(2): 148-159. (in Chinese) doi: 10.19818/j.cnki.1671-1637.2022.02.011 [10] PARK S, BANG S, KIM H, et al. Patch-based crack detection in black box images using convolutional neural networks[J]. Journal of Computing in Civil Engineering, 2019, 33(3): 04019017. doi: 10.1061/(ASCE)CP.1943-5487.0000831 [11] HUYAN Ju, LI Wei, TIGHE S, et al. Detection of sealed and unsealed cracks with complex backgrounds using deep convolutional neural network[J]. Automation in Construction, 2019, 107: 102946. doi: 10.1016/j.autcon.2019.102946 [12] 张志华, 邓砚学, 张新秀. 基于改进SegNet的沥青路面病害提取与分类方法[J]. 交通信息与安全, 2022, 40(3): 127-135. doi: 10.3963/j.jssn.1674-4861.2022.03.013ZHANG Zhi-hua, DENG Yan-xue, ZHANG Xin-xiu. A method for detecting and differentiating asphalt pavement distress based on an improved SegNet algorithm[J]. Journal of Transport Information and Safety, 2022, 40(3): 127-135. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2022.03.013 [13] YANG Fan, ZHANG Lei, YU Si-jia, et al. Feature pyramid and hierarchical boosting network for pavement crack detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(4): 1525-1535. doi: 10.1109/TITS.2019.2910595 [14] TONG Zheng, YUAN Dong-dong, GAO Jie, et al. Pavement defect detection with fully convolutional network and an uncertainty framework[J]. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(8): 832-849. doi: 10.1111/mice.12533 [15] HUYAN Ju, LI Wei, TIGHE S, et al. CrackU-net: a novel deep convolutional neural network for pixelwise pavement crack detection[J]. Structural Control and Health Monitoring, 2020, 27(8): e2551. [16] CHEN Han-shen, LIN Hui-ping, YAO Ming-hai. Improving the efficiency of encoder-decoder architecture for pixel-level crack detection[J]. IEEE Access, 2019, 7: 186657-186670. doi: 10.1109/ACCESS.2019.2961375 [17] MATHAVAN S, KAMAL K, RAHMAN M. A review of three-dimensional imaging technologies for pavement distress detection and measurements[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(5): 2353-2362. doi: 10.1109/TITS.2015.2428655 [18] CAO Wen-ming, LIU Qi-fan, HE Zhi-quan. Review of pavement defect detection methods[J]. IEEE Access, 2020, 8: 14531-14544. doi: 10.1109/ACCESS.2020.2966881 [19] ZHANG De-jin, ZOU Qin, LIN Hong, et al. Automatic pavement defect detection using 3D laser profiling technology[J]. Automation in Construction, 2018, 96: 350-365. doi: 10.1016/j.autcon.2018.09.019 [20] 丁世海, 战友, 阳恩慧, 等. 基于高精度激光断面高程的沥青路面MTD测量[J]. 东南大学学报(自然科学版), 2020, 50(1): 137-142. https://www.cnki.com.cn/Article/CJFDTOTAL-DNDX202001018.htmDING Shi-hai, ZHAN You, YANG En-hui, et al. MTD measurement of asphalt pavement based on high precision laser section elevation[J]. Journal of Southeast University (Natural Science Edition), 2020, 50(1): 137-142. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-DNDX202001018.htm [21] CHEN Jia-ying, HUANG Xiao-ming, ZHENG Bin-shuang, et al. Real-time identification system of asphalt pavement texture based on the close-range photogrammetry[J]. Construction and Building Materials, 2019, 226: 910-919. doi: 10.1016/j.conbuildmat.2019.07.321 [22] ZHANG A, WANG K C P, LI Bao-xian, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network[J]. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(10): 805-819. doi: 10.1111/mice.12297 [23] ZHANG A, WANG K C P, FEI Yue, et al. Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet[J]. Journal of Computing in Civil Engineering, 2018, 32(5): 04018041. doi: 10.1061/(ASCE)CP.1943-5487.0000775 [24] ZHANG A, WANG K C P, FEI Yue, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces with a recurrent neural network[J]. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(3): 213-229. doi: 10.1111/mice.12409 [25] FEI Yue, WANG K C P, ZHANG A, et al. Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(1): 273-284. doi: 10.1109/TITS.2019.2891167 [26] GUAN Jin-chao, YANG Xu, DING Ling, et al. Automated pixel-level pavement distress detection based on stereo vision and deep learning[J]. Automation in Construction, 2021, 129: 103788. doi: 10.1016/j.autcon.2021.103788 [27] 曾清红, 卢德唐. 基于移动最小二乘法的曲线曲面拟合[J]. 工程图学学报, 2004, 25(1): 84-89. doi: 10.3969/j.issn.1003-0158.2004.01.017ZENG Qing-hong, LU De-tang. Curve and surface fitting based on moving least-squares methods[J]. Journal of Graphics, 2004, 25(1): 84-89. (in Chinese) doi: 10.3969/j.issn.1003-0158.2004.01.017 [28] HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vision applications[J]. arXiv, 2017, DOI: 10.48550/arXiv.1704.04861 .[29] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]// NAVAB N, HORNEGGER J, WELLS W, et al. 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 2015: 234-241. [30] LAU S L H, CHONG E K P, YANG X, et al. Automated pavement crack segmentation using U-Net-based convolutional neural network[J]. IEEE Access, 2020, 8: 114892. doi: 10.1109/ACCESS.2020.3003638 [31] CHEN Jie, LIU Gang, CHEN Xin. Road crack image segmentation using global context U-net[C]//GOKHALE A, TAN Y. Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence. New York: Association for Computing Machinery, 2019: 181-185. -
点击查看大图
图(17) / 表(1)
计量
- 文章访问数: 152
- HTML全文浏览量: 33
- PDF下载量: 49
- 被引次数: 0
