-
摘要: 针对路面裂缝检测不完整和分割出现断裂的问题,提出了一种多尺度特征增强的路面裂缝检测网络MFENet,实现端到端的路面裂缝图像检测、分类和分割处理;设计了多尺度注意力特征增强模块,建立了网络模型的上层多尺度特征通道与底层特征通道权重系数之间的映射关系,以提升有效通道的特征输出;基于路面裂缝的坐标信息和像素语义信息在物理位置上的相关性,设计了多语义特征关联模块,实现不同语义信息之间的特征融合增强,并通过特征维度转换实现对路面裂缝图像的前景特征过滤;提出了一种针对深度特征强度进行量化评估的方法,用于提升模型提取特征能力的可解释性。在自采集数据集上的研究结果表明:MFENet对路面裂缝图像检测的平均精准率和平均召回率相比Mask R-CNN分别提升了4.3%和5.4%,相比基线模型RDSNet分别提升了14.6%和14.3%;MFENet对路面裂缝图像分割的平均精准率和平均召回率相比Mask R-CNN分别提升了6.6%和8.8%,相比RDSNet分别提升了8.1%和9.7%;与Mask R-CNN等主流方法相比,MFENet对不同类型路面裂缝图像的检测、分割精度最高。在公开数据集(CFD、CRACK500)上的研究结果表明:在不同场景下的数据集上,MFENet的检测、分割精度均高于Mask R-CNN等主流方法,模型的鲁棒性更强。另外与RDSNet相比,MFENet在不同数据集上的处理速度也均有所提升。Abstract: To solve the problems of incomplete pavement crack detection and discontinuous segmentation, a detection network MFENet for pavement cracks based on multi-scale feature enhancement was proposed, and the detection, classification and segmentation of end-to-end pavement crack images were realized. A multi-scale attention-based feature enhancement module was designed, and the mapping relationships of the weight coefficients of the upper multi-scale feature channels with those of the lower feature channels in the network model were determined to highlight the feature outputs from the effective channels. Based on the correlation between the coordinate information of the pavement crack and the semantic information of the pixels in physical location, a multi-semantic feature correlation module was designed and thereby feature fusion and enhancement among different semantic information were achieved. Then, the foreground features of the pavement crack image were filtered by feature dimension transformation. A quantitative evaluation method for deep feature intensity was proposed to improve the interpretability of the model's feature extraction ability. Research results on self-collected dataset show that the average precision and average recall of the MFENet in pavement crack image detection are 4.3% and 5.4% higher than those of the Mask R-CNN, respectively, and 14.6% and 14.3% higher than those of the baseline model RDSNet, respectively. The average precision and average recall of the MFENet in pavement crack image segmentation are 6.6% and 8.8% higher than those of the Mask R-CNN, respectively, and 8.1% and 9.7% higher than those of the RDSNet, respectively. In the comparison with the Mask R-CNN and other mainstream methods, the images of different types of pavement cracks are detected and segmented with the highest accuracy by the MFENet. Research results on public datasets (CFD and CRACK500) show that the detection and segmentation accuracy of the MFENet are invariably higher than those of the Mask R-CNN and other mainstream methods on the datasets covering different scenarios, indicating the higher robustness of the proposed method. In addition, the processing speed of the MFENet is also faster than that of the RDSNet on different datasets.
-
Key words:
- pavement crack detection /
- multi-scale attention /
- feature enhancement /
- multi-semantic /
- interpretability /
- robustness
-
表 1 MFC中用于提取路面裂缝建议区域特征的网络结构
Table 1. Network structure of MFC used to extract proposal region features of pavement crack
网络层 输出特征尺寸 $\left[\begin{array}{c}\text { Conv }(256, 32, 3 \times 3, \text { stride } 1, \text { padding 1) } \\ \text { GroupNorm }(32, 256) \\ \text { Relu }\end{array}\right] \times 4$ (w, h, 256) Conv(256, 32, 3×3, stride 1, padding 1) (w, h, 32) 表 2 MFC中用于重建路面裂缝分割语义特征的网络结构
Table 2. Network structure of MFC used to reconstruct segmentation semantic features of pavement crack
网络层 输出特征尺寸 $\left[\begin{array}{c}\operatorname{Conv}(3 \times 3, 256, 32, \text { stride } 1, \text { padding } 1) \\ \text { GroupNorm }(32, 256) \\ \text { Relu }\end{array}\right] \times 1$ (w, h, 256) UpsamplingBilinear2d, scale 2, mode bilinear (2 w, 2h, 256) Conv(3×3, 256, 32, stride 1, padding 1) (2 w, 2h, 32) 表 3 自采集数据集详情
Table 3. Details of self-collected dataset
检测设备 检测车 手机 路面类型 水泥路面 沥青路面 沥青路面 分辨率/像素 1 280×960 1 280×960 1 280×960,1 440×1 072,3 264×2 248,3 120×4 160 采集数量 118 4 273 104 总数 4 495 表 4 训练集、测试集和验证集的数据详情
Table 4. Details of training dataset, test dataset, and validation dataset
数据集类型 横向裂缝 纵向裂缝 网状裂缝 总数 占比 训练集 2 468 2 253 1 443 6 164 0.794 测试集 300 300 200 800 0.103 验证集 300 300 200 800 0.103 总数 3 068 2 853 1 843 7 764 1.000 -
[1] CAO Wen-ming, LIU Qi-fan, HE Zhi-quan. Review of pavement defect detection methods[J]. IEEE Access, 2020, 8: 14531-14544. doi: 10.1109/ACCESS.2020.2966881 [2] 徐志刚, 车艳丽, 李金龙, 等. 路面破损图像自动处理技术研究进展[J]交通运输工程学报, 2019, 19(1): 172-190. doi: 10.3969/j.issn.1671-1637.2019.01.017XU Zhi-gang, CHE Yan-li, LI Jin-long, et al. Research progress on automatic image processing technology for pavement distress[J]. Journal of Traffic and Transportation Engineering, 2019, 19(1): 172-190. (in Chinese) doi: 10.3969/j.issn.1671-1637.2019.01.017 [3] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//IEEE. 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2015: 1-9. [4] HE Kai-ming, ZHANG Xiang-yu, REN Shao-qing, et al. Deep residual learning for image recognition[C]//IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2016: 770-778. [5] YUSOF N A M, OSMAN M K, HUSSAIN Z, et al. Automated asphalt pavement crack detection and classification using deep convolution neural network[C]//IEEE. 9th IEEE International Conference on Control System, Computing and Engineering. New York: IEEE, 2019: 215-220. [6] PARK S, BANG S, KIM H, et al. Patch-based crack detection in black box images using convolutional neural networks[J]. Journal of Computing in Civil Engineering, 2019, 33(3): 04019017. doi: 10.1061/(ASCE)CP.1943-5487.0000831 [7] 章天杰, 韩海航. 基于残差神经网络的沥青路面裂缝识别分类研究[J]. 公路, 2021, 66(10): 24-29. doi: 10.3969/j.issn.1002-0268.2021.10.004ZHANG Tian-jie, HAN Hai-hang. Research on identification and classification of asphalt pavement cracks using residual neural network[J]. Highway, 2021, 66(10): 24-29. (in Chinese) doi: 10.3969/j.issn.1002-0268.2021.10.004 [8] IBRAGIMOV E, LEE H J, LEE J J, et al. Automated pavement distress detection using region based convolutional neural networks[J]. International Journal of Pavement Engineering, 2022, 23(6): 1981-1992. doi: 10.1080/10298436.2020.1833204 [9] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031 [10] 孙朝云, 裴莉莉, 李伟, 等. 基于改进Faster R-CNN的路面灌封裂缝检测方法研究[J]. 华南理工大学学报(自然科学版), 2020, 48(2): 84-93. https://www.cnki.com.cn/Article/CJFDTOTAL-HNLG202002011.htmSUN Zhao-yun, PEI Li-li, LI Wei, et al. Pavement sealed crack detection method based on improved faster R-CNN[J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(2): 84-93. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-HNLG202002011.htm [11] TRAN V P, TRAN T S, LEE H J, et al. One stage detector (RetinaNet)-based crack detection for asphalt pavements considering pavement distresses and surface objects[J]. Journal of Civil Structural Health Monitoring, 2021, 11(1): 205-222. doi: 10.1007/s13349-020-00447-8 [12] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 318-327. doi: 10.1109/TPAMI.2018.2858826 [13] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//NIPS. 31st Conference on Neural Information Processing Systems. San Diego: NIPS, 2017: 1-15. [14] 刘军, 王慧民, 张兴忠, 等. 基于Transformer的端到端路面裂缝检测方法[J]. 太原理工大学学报, 2022, 53(6): 1143-1151. https://www.cnki.com.cn/Article/CJFDTOTAL-TYGY202206021.htmLIU Jun, WANG Hui-min, ZHANG Xing-zhong, et al. End-to-end pavement crack detection method based on transformer[J]. Journal of Taiyuan University of Technology, 2022, 53(6): 1143-1151. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-TYGY202206021.htm [15] 陈涵深, 姚明海, 瞿心昱. 基于U型全卷积神经网络的路面裂缝检测[J]. 光电工程, 2020, 47(12): 200036. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202012007.htmCHEN Han-shen, YAO Ming-hai, QU Xin-yu. Pavement crack detection based on the U-shaped fully convolutional neural network[J]. Opto-Electronic Engineering, 2020, 47(12): 200036. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202012007.htm [16] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[J]. Lecture Notes in Computer Science, 2015, 9351: 234-241. [17] 瞿中, 陈雯. 基于空洞卷积和多特征融合的混凝土路面裂缝检测[J]. 计算机科学, 2022, 49(3): 192-196. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA202203026.htmQU Zhong, CHEN Wen. Concrete pavement crack detection based on dilated convolution and multi-features fusion[J]. Computer Science, 2022, 49(3): 192-196. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA202203026.htm [18] 陈良全, 王彩玲, 刘华军, 等. 基于连续注意力机制和卷积金字塔的路面裂缝检测[J]. 计算机系统应用, 2021, 30(8): 249-255. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202108035.htmCHEN Liang-quan, WANG Cai-ling, LIU Hua-jun, et al. Pavement crack detection with continuous attention mechanism and convolution pyramid structure[J]. Computer Systems and Applications, 2021, 30(8): 249-255. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202108035.htm [19] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495. doi: 10.1109/TPAMI.2016.2644615 [20] 张伯树, 张志华, 张洋. 改进的HRNet应用于路面裂缝分割与检测[J]. 测绘通报, 2022(3): 83-89. https://www.cnki.com.cn/Article/CJFDTOTAL-CHTB202203016.htmZHANG Bo-shu, ZHANG Zhi-hua, ZHANG Yang. Improved HRNet applied to segmentation and detection of pavement cracks[J]. Bulletin of Surveying and Mapping, 2022(3): 83-89. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-CHTB202203016.htm [21] WU Yang-xu, YANG Wan-ting, PAN Jin-xiao, et al. Asphalt pavement crack detection based on multi-scale full convolutional network[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 1495-1508. [22] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition, New York: IEEE, 2017: 4700-4708. [23] LIU J W, YANG X, LAU S, et al. Automated pavement crack detection and segmentation based on two-step convolutional neural network[J]. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(11): 1291-1305. [24] WANG S W, GONG Y C, XING J L, et al. RDSNet: a new deep architecture for reciprocal object detection and instance segmentation[C]//AIAA. 34th AAAI Conference on Artificial Intelligence. Reston: AIAA, 2020: 12208-12215. [25] LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//IEEE. 30th IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 936-944. [26] SHI Y, CUI L M, QI Z Q, et al. Automatic road crack detection using random structured forests[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(12): 3434-3445. [27] YANG F, ZHANG L, YU S J, et al. Feature pyramid and hierarchical boosting network for pavement crack detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(4): 1525-1535. [28] SONG Liang, WANG Xuan-cang. Faster region convolutional neural network for automated pavement distress detection[J]. Road Materials and Pavement Design, 2021, 22(1): 23-41. [29] SHEN Ting-sheng, NIE Ming-xin. Pavement damage detection based on cascade R-CNN[C]//ACM. Proceedings of the 4th International Conference on Computer Science and Application Engineering. New York: ACM, 2020: 1-5. [30] HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]//IEEE. 2017 IEEE International Conference on Computer Vision. New York: IEEE, 2017: 2980-2988. [31] ZHANG Y F, CHEN B, WANG J F, et al. APLCNet: automatic pixel-level crack detection network based on instance segmentation[J]. IEEE Access, 2020, 8: 199159-199170. [32] SHRIVASTAVA A, GUPTA A, GIRSHICK R. Training region-based object detectors with online hard example mining[C]//IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2016: 761-769.