留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

多尺度特征增强的路面裂缝检测方法

翟军治 孙朝云 裴莉莉 呼延菊 李伟

翟军治, 孙朝云, 裴莉莉, 呼延菊, 李伟. 多尺度特征增强的路面裂缝检测方法[J]. 交通运输工程学报, 2023, 23(1): 291-308. doi: 10.19818/j.cnki.1671-1637.2023.01.022
引用本文: 翟军治, 孙朝云, 裴莉莉, 呼延菊, 李伟. 多尺度特征增强的路面裂缝检测方法[J]. 交通运输工程学报, 2023, 23(1): 291-308. doi: 10.19818/j.cnki.1671-1637.2023.01.022
ZHAI Jun-zhi, SUN Zhao-yun, PEI Li-li, HUYAN Ju, LI Wei. Pavement crack detection method based on multi-scale feature enhancement[J]. Journal of Traffic and Transportation Engineering, 2023, 23(1): 291-308. doi: 10.19818/j.cnki.1671-1637.2023.01.022
Citation: ZHAI Jun-zhi, SUN Zhao-yun, PEI Li-li, HUYAN Ju, LI Wei. Pavement crack detection method based on multi-scale feature enhancement[J]. Journal of Traffic and Transportation Engineering, 2023, 23(1): 291-308. doi: 10.19818/j.cnki.1671-1637.2023.01.022

多尺度特征增强的路面裂缝检测方法

doi: 10.19818/j.cnki.1671-1637.2023.01.022
基金项目: 

国家重点研发计划 2021YFB1600205

国家自然科学基金项目 52178407

国家自然科学基金项目 51978071

陕西省重点研发计划 2022JBGS3-08

中央高校基本科研业务费专项资金项目 300102242901

详细信息
    作者简介:

    翟军治(1981-), 男, 河南开封人, 长安大学工学博士研究生, 从事路面裂缝图像自动化检测技术研究

    孙朝云(1962-), 女, 安徽太和人, 长安大学教授, 工学博士

  • 中图分类号: U418.6

Pavement crack detection method based on multi-scale feature enhancement

Funds: 

National Key Research and Development Program of China 2021YFB1600205

National Natural Science Foundation of China 52178407

National Natural Science Foundation of China 51978071

Key Research and Development Program of Shaanxi Province 2022JBGS3-08

Fundamental Research Funds for the Central Universities 300102242901

More Information
  • 摘要: 针对路面裂缝检测不完整和分割出现断裂的问题,提出了一种多尺度特征增强的路面裂缝检测网络MFENet,实现端到端的路面裂缝图像检测、分类和分割处理;设计了多尺度注意力特征增强模块,建立了网络模型的上层多尺度特征通道与底层特征通道权重系数之间的映射关系,以提升有效通道的特征输出;基于路面裂缝的坐标信息和像素语义信息在物理位置上的相关性,设计了多语义特征关联模块,实现不同语义信息之间的特征融合增强,并通过特征维度转换实现对路面裂缝图像的前景特征过滤;提出了一种针对深度特征强度进行量化评估的方法,用于提升模型提取特征能力的可解释性。在自采集数据集上的研究结果表明:MFENet对路面裂缝图像检测的平均精准率和平均召回率相比Mask R-CNN分别提升了4.3%和5.4%,相比基线模型RDSNet分别提升了14.6%和14.3%;MFENet对路面裂缝图像分割的平均精准率和平均召回率相比Mask R-CNN分别提升了6.6%和8.8%,相比RDSNet分别提升了8.1%和9.7%;与Mask R-CNN等主流方法相比,MFENet对不同类型路面裂缝图像的检测、分割精度最高。在公开数据集(CFD、CRACK500)上的研究结果表明:在不同场景下的数据集上,MFENet的检测、分割精度均高于Mask R-CNN等主流方法,模型的鲁棒性更强。另外与RDSNet相比,MFENet在不同数据集上的处理速度也均有所提升。

     

  • 图  1  多尺度特征增强的路面裂缝检测网络

    Figure  1.  avement crack detection network based on multi-scale feature enhancement

    图  2  MFA增强前后的通道特征对比

    Figure  2.  Comparison of channel features before and after MFA enhancement

    图  3  MFA增强前后各通道特征图与输入图像间的PSNR对比

    Figure  3.  PSNR comparison between each channel feature map and input image before and after MFA enhancement

    图  4  MFC融合前后的通道特征对比

    Figure  4.  Comparison of channel features before and after MFC fusion

    图  5  MFC增强前后各前景通道特征图与原始图像间的PSNR对比

    Figure  5.  PSNR comparison between each foreground channel feature map and input images before and after MFC enhancement

    图  6  MFENet在训练集上的loss收敛曲线

    Figure  6.  Loss convergence curve of MFENet on training dataset

    图  7  训练过程中MFENet在验证集上的检测精度曲线

    Figure  7.  Detection accuracy curves of MFENet on validation dataset during training

    图  8  自采集数据集路面裂缝图像样本

    Figure  8.  Pavement crack image samples of self-collected dataset

    图  9  MFENet与其他方法在自采集数据集上的检测精度对比

    Figure  9.  Comparison of detection accuracies between MFENet and other methods on self-collected dataset

    图  10  MFENet与其他方法在自采集数据集上的检测速度对比

    Figure  10.  Comparison of detection speeds between MFENet and other methods on self-collected dataset

    图  11  MFENet与其他方法在自采集数据集上的分割精度对比

    Figure  11.  Comparison of segmentation accuracies between MFENet and other methods on self-collected dataset

    图  12  MFENet与其他方法在自采集数据集上的分割速度对比

    Figure  12.  Comparison of segmentation speeds between MFENet and other methods on self-collected dataset

    图  13  MFENet与其他方法在自采集数据集上对不同类型裂缝的检测精度对比

    Figure  13.  Comparison of detection accuracies of different type cracks between MFENet and other methods on self-collected dataset

    图  14  MFENet与其他方法在自采集数据集上对不同类型裂缝的检测速度对比

    Figure  14.  Comparison of detection speeds of different type cracks between MFENet and other methods on self-collected dataset

    图  15  MFENet与其他方法在自采集数据集上对不同类型裂缝的分割精度对比

    Figure  15.  Comparison of segmentation accuracies of different type cracks between MFENet and other methods on self-collected dataset

    图  16  MFENet与其他方法在自采集数据集上对不同类型裂缝的分割速度对比

    Figure  16.  Comparison of segmentation speeds of different type cracks between MFENet and other methods on self-collected dataset

    图  17  不同方法在自采集数据集上的检测、分割结果可视化对比

    Figure  17.  Visualized comparison of detection and segmentation results of different methods on self-collected dataset

    图  18  MFENet与其他方法在CFD数据集上的检测精度对比

    Figure  18.  Comparison of detection accuracies between MFENet and other methods on CFD dataset

    图  19  MFENet与其他方法在CFD数据集上的检测速度对比

    Figure  19.  Comparison of detection speeds between MFENet and other methods on CFD dataset

    图  20  MFENet与其他方法在CFD数据集上的分割精度对比

    Figure  20.  Comparison of segmentation accuracies between MFENet and other methods on CFD dataset

    图  21  MFENet与其他方法在CFD数据集上的分割速度对比

    Figure  21.  Comparison of segmentation speeds between MFENet and other methods on CFD dataset

    图  22  MFENet与其他方法在CRACK500数据集上的检测精度对比

    Figure  22.  Comparison of detection accuracies between MFENet and other methods on CRACK500 dataset

    图  23  MFENet与其他方法在CRACK500数据集上的检测速度对比

    Figure  23.  Comparison of detection speeds between MFENet and other methods on CRACK500 dataset

    图  24  MFENet与其他方法在CRACK500数据集上的分割精度对比

    Figure  24.  Comparison of segmentation accuracies between MFENet and other methods on CRACK500 dataset

    图  25  MFENet与其他方法在CRACK500数据集上的分割速度对比

    Figure  25.  Comparison of segmentation speeds between MFENet and other methods on CRACK500 dataset

    表  1  MFC中用于提取路面裂缝建议区域特征的网络结构

    Table  1.   Network structure of MFC used to extract proposal region features of pavement crack

    网络层 输出特征尺寸
    $\left[\begin{array}{c}\text { Conv }(256, 32, 3 \times 3, \text { stride } 1, \text { padding 1) } \\ \text { GroupNorm }(32, 256) \\ \text { Relu }\end{array}\right] \times 4$ (w, h, 256)
    Conv(256, 32, 3×3, stride 1, padding 1) (w, h, 32)
    下载: 导出CSV

    表  2  MFC中用于重建路面裂缝分割语义特征的网络结构

    Table  2.   Network structure of MFC used to reconstruct segmentation semantic features of pavement crack

    网络层 输出特征尺寸
    $\left[\begin{array}{c}\operatorname{Conv}(3 \times 3, 256, 32, \text { stride } 1, \text { padding } 1) \\ \text { GroupNorm }(32, 256) \\ \text { Relu }\end{array}\right] \times 1$ (w, h, 256)
    UpsamplingBilinear2d, scale 2, mode bilinear (2 w, 2h, 256)
    Conv(3×3, 256, 32, stride 1, padding 1) (2 w, 2h, 32)
    下载: 导出CSV

    表  3  自采集数据集详情

    Table  3.   Details of self-collected dataset

    检测设备 检测车 手机
    路面类型 水泥路面 沥青路面 沥青路面
    分辨率/像素 1 280×960 1 280×960 1 280×960,1 440×1 072,3 264×2 248,3 120×4 160
    采集数量 118 4 273 104
    总数 4 495
    下载: 导出CSV

    表  4  训练集、测试集和验证集的数据详情

    Table  4.   Details of training dataset, test dataset, and validation dataset

    数据集类型 横向裂缝 纵向裂缝 网状裂缝 总数 占比
    训练集 2 468 2 253 1 443 6 164 0.794
    测试集 300 300 200 800 0.103
    验证集 300 300 200 800 0.103
    总数 3 068 2 853 1 843 7 764 1.000
    下载: 导出CSV
  • [1] CAO Wen-ming, LIU Qi-fan, HE Zhi-quan. Review of pavement defect detection methods[J]. IEEE Access, 2020, 8: 14531-14544. doi: 10.1109/ACCESS.2020.2966881
    [2] 徐志刚, 车艳丽, 李金龙, 等. 路面破损图像自动处理技术研究进展[J]交通运输工程学报, 2019, 19(1): 172-190. doi: 10.3969/j.issn.1671-1637.2019.01.017

    XU Zhi-gang, CHE Yan-li, LI Jin-long, et al. Research progress on automatic image processing technology for pavement distress[J]. Journal of Traffic and Transportation Engineering, 2019, 19(1): 172-190. (in Chinese) doi: 10.3969/j.issn.1671-1637.2019.01.017
    [3] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//IEEE. 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2015: 1-9.
    [4] HE Kai-ming, ZHANG Xiang-yu, REN Shao-qing, et al. Deep residual learning for image recognition[C]//IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2016: 770-778.
    [5] YUSOF N A M, OSMAN M K, HUSSAIN Z, et al. Automated asphalt pavement crack detection and classification using deep convolution neural network[C]//IEEE. 9th IEEE International Conference on Control System, Computing and Engineering. New York: IEEE, 2019: 215-220.
    [6] PARK S, BANG S, KIM H, et al. Patch-based crack detection in black box images using convolutional neural networks[J]. Journal of Computing in Civil Engineering, 2019, 33(3): 04019017. doi: 10.1061/(ASCE)CP.1943-5487.0000831
    [7] 章天杰, 韩海航. 基于残差神经网络的沥青路面裂缝识别分类研究[J]. 公路, 2021, 66(10): 24-29. doi: 10.3969/j.issn.1002-0268.2021.10.004

    ZHANG Tian-jie, HAN Hai-hang. Research on identification and classification of asphalt pavement cracks using residual neural network[J]. Highway, 2021, 66(10): 24-29. (in Chinese) doi: 10.3969/j.issn.1002-0268.2021.10.004
    [8] IBRAGIMOV E, LEE H J, LEE J J, et al. Automated pavement distress detection using region based convolutional neural networks[J]. International Journal of Pavement Engineering, 2022, 23(6): 1981-1992. doi: 10.1080/10298436.2020.1833204
    [9] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031
    [10] 孙朝云, 裴莉莉, 李伟, 等. 基于改进Faster R-CNN的路面灌封裂缝检测方法研究[J]. 华南理工大学学报(自然科学版), 2020, 48(2): 84-93. https://www.cnki.com.cn/Article/CJFDTOTAL-HNLG202002011.htm

    SUN Zhao-yun, PEI Li-li, LI Wei, et al. Pavement sealed crack detection method based on improved faster R-CNN[J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(2): 84-93. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-HNLG202002011.htm
    [11] TRAN V P, TRAN T S, LEE H J, et al. One stage detector (RetinaNet)-based crack detection for asphalt pavements considering pavement distresses and surface objects[J]. Journal of Civil Structural Health Monitoring, 2021, 11(1): 205-222. doi: 10.1007/s13349-020-00447-8
    [12] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 318-327. doi: 10.1109/TPAMI.2018.2858826
    [13] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//NIPS. 31st Conference on Neural Information Processing Systems. San Diego: NIPS, 2017: 1-15.
    [14] 刘军, 王慧民, 张兴忠, 等. 基于Transformer的端到端路面裂缝检测方法[J]. 太原理工大学学报, 2022, 53(6): 1143-1151. https://www.cnki.com.cn/Article/CJFDTOTAL-TYGY202206021.htm

    LIU Jun, WANG Hui-min, ZHANG Xing-zhong, et al. End-to-end pavement crack detection method based on transformer[J]. Journal of Taiyuan University of Technology, 2022, 53(6): 1143-1151. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-TYGY202206021.htm
    [15] 陈涵深, 姚明海, 瞿心昱. 基于U型全卷积神经网络的路面裂缝检测[J]. 光电工程, 2020, 47(12): 200036. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202012007.htm

    CHEN Han-shen, YAO Ming-hai, QU Xin-yu. Pavement crack detection based on the U-shaped fully convolutional neural network[J]. Opto-Electronic Engineering, 2020, 47(12): 200036. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202012007.htm
    [16] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[J]. Lecture Notes in Computer Science, 2015, 9351: 234-241.
    [17] 瞿中, 陈雯. 基于空洞卷积和多特征融合的混凝土路面裂缝检测[J]. 计算机科学, 2022, 49(3): 192-196. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA202203026.htm

    QU Zhong, CHEN Wen. Concrete pavement crack detection based on dilated convolution and multi-features fusion[J]. Computer Science, 2022, 49(3): 192-196. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA202203026.htm
    [18] 陈良全, 王彩玲, 刘华军, 等. 基于连续注意力机制和卷积金字塔的路面裂缝检测[J]. 计算机系统应用, 2021, 30(8): 249-255. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202108035.htm

    CHEN Liang-quan, WANG Cai-ling, LIU Hua-jun, et al. Pavement crack detection with continuous attention mechanism and convolution pyramid structure[J]. Computer Systems and Applications, 2021, 30(8): 249-255. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202108035.htm
    [19] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495. doi: 10.1109/TPAMI.2016.2644615
    [20] 张伯树, 张志华, 张洋. 改进的HRNet应用于路面裂缝分割与检测[J]. 测绘通报, 2022(3): 83-89. https://www.cnki.com.cn/Article/CJFDTOTAL-CHTB202203016.htm

    ZHANG Bo-shu, ZHANG Zhi-hua, ZHANG Yang. Improved HRNet applied to segmentation and detection of pavement cracks[J]. Bulletin of Surveying and Mapping, 2022(3): 83-89. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-CHTB202203016.htm
    [21] WU Yang-xu, YANG Wan-ting, PAN Jin-xiao, et al. Asphalt pavement crack detection based on multi-scale full convolutional network[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 1495-1508.
    [22] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition, New York: IEEE, 2017: 4700-4708.
    [23] LIU J W, YANG X, LAU S, et al. Automated pavement crack detection and segmentation based on two-step convolutional neural network[J]. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(11): 1291-1305.
    [24] WANG S W, GONG Y C, XING J L, et al. RDSNet: a new deep architecture for reciprocal object detection and instance segmentation[C]//AIAA. 34th AAAI Conference on Artificial Intelligence. Reston: AIAA, 2020: 12208-12215.
    [25] LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//IEEE. 30th IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 936-944.
    [26] SHI Y, CUI L M, QI Z Q, et al. Automatic road crack detection using random structured forests[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(12): 3434-3445.
    [27] YANG F, ZHANG L, YU S J, et al. Feature pyramid and hierarchical boosting network for pavement crack detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(4): 1525-1535.
    [28] SONG Liang, WANG Xuan-cang. Faster region convolutional neural network for automated pavement distress detection[J]. Road Materials and Pavement Design, 2021, 22(1): 23-41.
    [29] SHEN Ting-sheng, NIE Ming-xin. Pavement damage detection based on cascade R-CNN[C]//ACM. Proceedings of the 4th International Conference on Computer Science and Application Engineering. New York: ACM, 2020: 1-5.
    [30] HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]//IEEE. 2017 IEEE International Conference on Computer Vision. New York: IEEE, 2017: 2980-2988.
    [31] ZHANG Y F, CHEN B, WANG J F, et al. APLCNet: automatic pixel-level crack detection network based on instance segmentation[J]. IEEE Access, 2020, 8: 199159-199170.
    [32] SHRIVASTAVA A, GUPTA A, GIRSHICK R. Training region-based object detectors with online hard example mining[C]//IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2016: 761-769.
  • 加载中
图(25) / 表(4)
计量
  • 文章访问数:  552
  • HTML全文浏览量:  246
  • PDF下载量:  160
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-09-02
  • 网络出版日期:  2023-03-08
  • 刊出日期:  2023-02-25

目录

    /

    返回文章
    返回