Puddle area segmentation of asphalt pavements based on FRRN attention and supervision
-
摘要: 为实现不同光照与气候条件下的沥青路面积水区域自动分割，在现有全分辨率残差网络(FRRN)的上采样层设计了一种注意力监督机制；加入积水区域注意力模块(PAAM)与深度监督模块，建立了一种含积水区域注意力监督模型(PAAM-FRRN)；利用最大池化与上采样构建编码器-解码器结构，在全局尺度下捕获了积水的视觉特征，在上采样层引入了注意力监督机制，针对积水区域进行上采样并融合不同网络层之间的特征，最小化网络各层损失函数，优化网络总体最终损失；采集不同光照与气候条件下的1 770个(弱光750个、强光740个、雨天280个)沥青路面积水图像进行五折交叉验证试验，获得了积水区域分割结果；以人工标注结果作为真值，利用Dice相似系数(DSC)、Jaccard相似系数(JSC)、精确率、召回率与豪斯多夫距离(HD95)作为量化评价指标，将提出的模型分别与FRRN和其他7种有代表性的传统模型进行了分割效果对比。研究结果表明：提出的模型在DSC、JSC、精确率与召回率所取得的均值分别为0.91、0.86、0.92和0.93，相比FRRN分别提高了3.41%、6.17%、2.22%和4.49%，且均高于传统7种对比模型；在DSC、JSC、精确率以及召回率所取得的标准差分别为0.12、0.15、0.11和0.12，与FRRN相比分别降低了20.00%、16.67%、21.43%和25.00%，且均低于传统7种对比模型；其HD95为38.56，均低于其他对比模型，提出的模型能够实现对不同光照与气候条件下的沥青路面积水区域的准确、有效分割。Abstract: To realize the automatic segmentation of puddle area of asphalt roads under various brightness levels and climate conditions, an attentional supervision mechanism was developed on the upsampling layer of existing full resolution residual network (FRRN). A puddle area attention module based on the full-resolution residual network (PAAM-FRRN) model was established by adding the puddle area attention module (PAAM) and the deep supervision module. An encoder-decoder structure was constructed by max-pooling and upsampling to capture the visual characteristics of puddle in a global scale. An attentional supervision mechanism was introduced in the upsampling layer to conduct the upsampling for the puddle area and fuse the features of different network layers to minimize the loss function of each network layer and optimize the overall final loss of the network. A total of 1 770 images (750 under low light, 740 under strong light and 280 under rainy weather) of asphalt pavement with puddle were collected for the five-fold cross-validation test, and the segmentation results of the puddle area were obtained. Manual annotation results were used as truth values, and the dice similarity coefficient (DSC), Jaccard similarity coefficient (JSC), precision, recall, and Hausdorff distance (HD95) were used as quantitative evaluation indexes. The segmentation effect of the proposed model was compared with the FRRN and seven other representative traditional models. Research results show that the mean values of DSC, JSC, precision, and recall obtained by the proposed model are 0.91, 0.86, 0.92, and 0.93, respectively, which are 3.41%, 6.17%, 2.22%, and 4.49% higher than those obtained by the FRRN, and are all higher than those obtained by the seven traditional comparison models. Standard deviation values of DSC, JSC, precision, and recall as computed by the proposed model are 0.12, 0.15, 0.11, and 0.12, respectively, which are 20.00%, 16.67%, 21.43%, and 25.00% lower than those of FRRN, and are all lower than those of the seven traditional comparison models. The HD95 of the proposed model is 38.56, which is lower than those of the other comparison models. Therefore, the proposed model can achieve accurate and effective segmentation of puddle areas of asphalt pavements under different brightness levels and climate conditions. 6 tabs, 11 figs, 40 refs.
-
图 7 不同条件下的样本与其对应的真值
Figure 7. Samples and their corresponding truth values in different conditions
图 8 模型损失值与轮次关系曲线
Figure 8. Relationship curves between loss values and epochs of models
图 9 各模型分割结果可视化对比
Figure 9. Visualized comparison for segmentation results of various models
图 10 基于PAAM-FRRN不同样本的第15通道残差特征
Figure 10. Residual features of PAAM-FRRN in 15th channel for different samples
图 11 多评价指标下各模型对验证集的分割结果
Figure 11. Segmentation results of each model on validation set under multiple evaluation indexes
表 1 每折交叉验证的数据集分布
Table 1. Dataset distribution of each cross-validation
数据集 样本分类 弱光 强光 雨天 总数 训练集 600 592 224 1 416 验证集 150 148 56 354 总数 750 740 280 1 770 
下载: 导出CSV
表 2 DSC分割效果定量对比
Table 2. Quantitative comparison of segmentation effects using DSC
验证集 数据类型 DSC 条件 图像/个 U-Net ResU-Net Attention ResU-Net Squeeze U-Net DeconvNet DANet DeepLabV3 FRRN PAAM-FRRN
(本文模型)弱光 150 平均值 0.86 0.84 0.85 0.79 0.85 0.88 0.61 0.89 0.91 标准差 0.15 0.16 0.16 0.17 0.16 0.14 0.20 0.14 0.12 强光 148 平均值 0.85 0.83 0.84 0.76 0.82 0.87 0.59 0.89 0.92 标准差 0.14 0.15 0.14 0.17 0.17 0.13 0.20 0.12 0.10 雨天 56 平均值 0.70 0.67 0.76 0.66 0.74 0.82 0.52 0.81 0.89 标准差 0.20 0.22 0.18 0.21 0.21 0.17 0.21 0.18 0.12 总计 354 平均值 0.83 0.81 0.83 0.76 0.82 0.87 0.59 0.88 0.91 标准差 0.17 0.19 0.16 0.19 0.18 0.15 0.21 0.15 0.12
下载: 导出CSV
表 3 JSC分割效果定量对比
Table 3. Quantitative comparison of segmentation effects using JSC
验证集 数据类型 JSC 条件 图像/个 U-Net ResU-Net Attention ResU-Net Squeeze U-Net DeconvNet DANet DeepLabV3 FRRN PAAM-FRRN
(本文模型)弱光 150 平均值 0.78 0.75 0.77 0.70 0.76 0.81 0.49 0.83 0.86 标准差 0.18 0.19 0.20 0.20 0.19 0.18 0.20 0.18 0.15 强光 148 平均值 0.76 0.73 0.75 0.66 0.73 0.80 0.46 0.82 0.87 标准差 0.18 0.19 0.17 0.19 0.20 0.16 0.19 0.16 0.13 总计 354 平均值 0.58 0.56 0.65 0.55 0.63 0.73 0.39 0.72 0.82 标准差 0.23 0.24 0.21 0.22 0.24 0.20 0.19 0.21 0.16 雨天 56 平均值 0.74 0.71 0.74 0.66 0.73 0.79 0.46 0.81 0.86 标准差 0.21 0.22 0.20 0.21 0.21 0.18 0.20 0.18 0.15
下载: 导出CSV
表 4 精确率分割效果定量对比
Table 4. Quantitative comparison of segmentation effects using precision
验证集 数据类型 精确率 条件 图像/个 U-Net ResU-Net Attention ResU-Net Squeeze U-Net DeconvNet DANet DeepLabV3 FRRN PAAM-FRRN
(本文模型)弱光 150 平均值 0.84 0.82 0.83 0.78 0.85 0.86 0.57 0.90 0.92 标准差 0.17 0.18 0.18 0.19 0.18 0.17 0.23 0.15 0.12 强光 148 平均值 0.87 0.84 0.82 0.76 0.85 0.87 0.59 0.90 0.92 标准差 0.14 0.16 0.16 0.19 0.17 0.14 0.22 0.11 0.11 雨天 56 平均值 0.82 0.78 0.78 0.70 0.80 0.84 0.53 0.87 0.91 标准差 0.20 0.22 0.20 0.23 0.22 0.17 0.24 0.17 0.11 总计 354 平均值 0.85 0.82 0.82 0.76 0.84 0.86 0.57 0.90 0.92 标准差 0.17 0.19 0.18 0.20 0.19 0.16 0.23 0.14 0.11
下载: 导出CSV
表 5 召回率分割效果定量对比
Table 5. Quantitative comparison of segmentation effects using recall
验证集 数据类型 召回率 条件 图像/个 U-Net ResU-Net Attention ResU-Net Squeeze U-Net DeconvNet DANet DeepLabV3 FRRN PAAM-FRRN
(本文模型)弱光 150 平均值 0.91 0.90 0.91 0.87 0.87 0.92 0.74 0.91 0.93 标准差 0.13 0.15 0.13 0.15 0.15 0.12 0.18 0.14 0.11 强光 148 平均值 0.87 0.86 0.90 0.81 0.83 0.90 0.68 0.90 0.93 标准差 0.16 0.17 0.13 0.17 0.19 0.13 0.20 0.14 0.10 雨天 56 平均值 0.67 0.65 0.78 0.68 0.75 0.85 0.62 0.79 0.90 标准差 0.23 0.25 0.19 0.22 0.23 0.18 0.20 0.21 0.15 总计 354 平均值 0.86 0.84 0.89 0.81 0.83 0.90 0.69 0.89 0.93 标准差 0.19 0.21 0.16 0.19 0.19 0.14 0.20 0.16 0.12
下载: 导出CSV
表 6 HD95分割效果定量对比
Table 6. Quantitative comparison of segmentation effects using HD95
验证集 数据类型 HD95 条件 图像/个 U-Net ResU-Net Attention ResU-Net Squeeze U-Net DeconvNet DANet DeepLabV3 FRRN PAAM-FRRN
(本文模型)弱光 150 平均值 70.72 73.12 62.37 64.25 46.07 52.82 80.58 50.33 41.57 标准差 33.77 32.54 31.25 31.62 30.66 33.11 28.75 33.38 30.98 强光 148 平均值 66.35 71.99 63.48 65.93 46.80 54.09 84.98 47.52 34.64 标准差 33.75 31.94 31.92 32.18 31.03 33.10 28.36 31.91 31.68 雨天 56 平均值 74.54 80.46 71.40 69.24 54.42 65.75 90.33 54.36 40.87 标准差 33.69 30.75 33.49 30.00 33.94 32.94 27.46 33.70 31.43 总计 354 平均值 69.50 73.81 64.26 65.74 47.70 55.40 83.96 49.79 38.56 标准差 34.33 32.30 32.19 31.93 31.67 33.64 28.70 33.25 31.83
下载: 导出CSV
-
[1] 季天剑, 黄晓明, 刘清泉. 部分滑水对路面附着系数的影响[J]. 交通运输工程学报, 2003, 3(4): 10-12. http://jtysgcxb.xml-journal.net/article/id/200304003JI Tian-jian, HUANG Xiao-ming, LIU Qing-quan. Part hydroplaning effect on pavement friction coefficient[J]. Journal of Traffic and Transportation Engineering, 2003, 3(4): 10-12. (in Chinese) http://jtysgcxb.xml-journal.net/article/id/200304003 [2] 周海超, 梁晨, 杨建, 等. 提升轮胎抗滑水性能的仿生方法[J]. 机械工程学报, 2015, 51(8): 125-130, 136. https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201508018.htmZHOU Hai-chao, LIANG Chen, YANG Jian, et al. Bionic method for improving tire anti-hydroplaning performance[J]. Journal of Mechanical Engineering, 2015, 51(8): 125-130, 136. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201508018.htm [3] 周海超, 王国林, 姜震, 等. 湿滑状态下轮胎路面摩擦特性的数值分析方法[J]. 机械工程学报, 2020, 56: 1-9. https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB202021019.htmZHOU Hai-chao, WANG Guo-lin, JIANG Zhen, et al. Numerical analysis method for friction characteristics of tire-pavement[J]. Journal of Mechanical Engineering, 2020, 56: 1-9. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB202021019.htm [4] JONSSON P, CASSELGREN J, THORNBERG B. Road surface status classification using spectral analysis of NIR camera images[J]. IEEE Sensors Journal, 2015, 15(3): 1641-1656. doi: 10.1109/JSEN.2014.2364854 [5] AMTHOR M, HARTMANN B, DENZLER J. Road condition estimation based on spatio-temporal reflection models[C]//DAGM. Proceedings of the 37th German Conference on Pattern Recognition. Aachen: DAGM, 2015: 3-15. [6] PICCARDI A, COLACE L. Optical detection of dangerous road conditions[J]. Sensors, 2019, 19(6): 1360. doi: 10.3390/s19061360 [7] AKI M, ROJANAARPA T, NAKANO K, et al. Road surface recognition using laser radar for automatic platooning[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(10): 2800-2810. doi: 10.1109/TITS.2016.2528892 [8] NAKASHIMA S, ARAMAKI S, KITAZONO Y, et al. Application of ultrasonic sensors in road surface condition distinction methods[J]. Sensors, 2016, 16(10): 1678. doi: 10.3390/s16101678 [9] KALLIRIS M, KANARACHOS S, KOTSAKIS R, et al. Machine learning algorithms for wet road surface detection using acoustic measurements[C]//IEEE. International Conference on Mechatronics. New York: IEEE, 2019: 265-270. [10] 卢俊辉, 王建强, 李克强, 等. 基于路面温度和太阳辐射强度的路面状态识别方法[J]. 农业机械学报, 2010, 41(5): 21-23, 11. https://www.cnki.com.cn/Article/CJFDTOTAL-NYJX201005004.htmLU Juan-hui, WANG Jian-qiang, LI Ke-qiang, et al. Road condition detection based on road temperature and solar radiation[J]. Transactions of the Chinese Society of Agricultural Machinery, 2010, 41(5): 21-23, 11. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-NYJX201005004.htm [11] 顾昊, 李勃, 张潇, 等. 基于偏振测量的路面积水积冰检测方法[J]. 电子测量技术, 2011, 34(7): 99-102. https://www.cnki.com.cn/Article/CJFDTOTAL-DZCL201107026.htmGU Hao, LI Bo, ZHANG Xiao, et al. Detection of road surface water and ice based on polarization measurement[J]. Electronic Measurement Technology, 2011, 34(7): 99-102. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-DZCL201107026.htm [12] GUERRERO-IBÁÑEZ J, ZEADALLY S, CONTRERAS-CASTILLO J, et al. Sensor technologies for intelligent transportation systems[J]. Sensors, 2018, 18(4): 1212. [13] YAMADA M, UEDA K, HORIBA I, et al. Detection of wet-road conditions from images captured by a vehicle-mounted camera[J]. Journal of Robotics and Mechatronics, 2005, 17(3): 269-276. [14] YANG H J, JANG H, JEONG D S. Detection algorithm for road surface condition using wavelet packet transform and SVM[C]//IEEE. Proceedings of the 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision. New York: IEEE, 2013: 323-326. [15] KIM J, BAEK J, CHOI H, et al. Wet area and puddle detection for advanced driver assistance systems (ADAS) using a stereo camera[J]. International Journal of Control Automation and Systems, 2016, 14(1): 263-271. [16] ZHAO Jian-dong, WU Hong-qiang, CHEN Liang-liang. Road surface state recognition based on SVM optimization and image segmentation processing[J]. Journal of Advanced Transportation, 2017, 2017: 1-21. [17] 刘新宇, 黄德启. 基于SVM分类器的道路湿滑图像分类方法研究[J]. 武汉理工大学学报(交通科学与工程版), 2011, 35(4): 784-787. https://www.cnki.com.cn/Article/CJFDTOTAL-JTKJ201104033.htmLIU Xin-yu, HUANG De-qi. Study on classifier of wet-road images based on SVM[J]. Journal of Wuhan University of Technology (Transportation Science and Engineering), 2011, 35(4): 784-787. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JTKJ201104033.htm [18] 陈添丁, 王娃, 朱敬意, 等. 基于图像偏振特性的野外环境积水区域检测[J]. 光电工程, 2013, 40(4): 14-23. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201304006.htmCHEN Tian-ding, WANG Wa, ZHU Jing-yi, et al. Wild environment water hazards detection based on polarization information of image[J]. Opto-Electronic Engineering, 2013, 40(4): 14-23. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201304006.htm [19] 万剑, 赵恺, 王维锋. 基于高维特征和RBP神经网络的湿滑道路图像判别方法[J]. 交通信息与安全, 2013, 31(2): 32-35. https://www.cnki.com.cn/Article/CJFDTOTAL-JTJS201302010.htmWAN Jian, ZHAO Kai, WANG Wei-feng. Classification of slippery road images based on high-dimensional features and RBP neural network[J]. Journal of Transport Information and Safety, 2013, 31(2): 32-35. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JTJS201302010.htm [20] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444. [21] 沈长青, 王旭, 王冬, 等. 基于多尺度卷积类内迁移学习的列车轴承故障诊断[J]. 交通运输工程学报, 2020, 20(5): 151-164. doi: 10.19818/j.cnki.1671-1637.2020.05.012SHEN Chang-qing, WANG Xu, WANG Dong, et al. Multi-scale convolutional intra-class transfer learning for train bearing fault diagnosis[J]. Journal of Traffic and Transportation Engineering, 2020, 20(5): 151-164. (in Chinese) doi: 10.19818/j.cnki.1671-1637.2020.05.012 [22] ZAITOUN N M, AQEL M J. Survey on image segmentation techniques[J]. Procedia Computer Science, 2015, 65: 797-806. [23] HUANG Qing-hua, LUO Yao-zhong, ZHANG Qiang-zhi. Breast ultrasound image segmentation: a survey[J]. International Journal of Computer Assisted Radiology and Surgery, 2017, 12(3): 493-507. [24] LIU Wei-bo, WANG Zi-dong, LIU Xiao-hui, et al. A survey of deep neural network architectures and their applications[J]. Neurocomputing, 2017, 234: 11-26. [25] GARCIA-GARCIA A, ORTS-ESCOLANO S, OPREA S, et al. A survey on deep learning techniques for image and video semantic segmentation[J]. Applied Soft Computing, 2018, 70: 41-65. [26] LI J, NGUYEN C. Realtime water-hazard detection and visualisation for autonomous navigation and advanced driving assistance[C]//IEEE. 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct. New York: IEEE, 2019: 287-288. [27] TAN Xue-feng, WANG Man-tao, HE Lin-chao, et al. Urban road water recognition based on deep learning[C]//Springer. Frontier Computing 2019. Berlin: Springer, 2020: 1387-1395. [28] HAN Xiao-feng, NGUYEN C, YOU S, et al. Single image water hazard detection using FCN with reflection attention units[C]//Springer. European Conference on Computer Vision 2018. Berlin: Springer, 2018: 105-121. [29] WANG L, WANG H. Water hazard detection using conditional generative adversarial network with mixture reflection attention units[J]. IEEE Access, 2019, 7: 167497-167506. [30] QIAO Jian-jun, WU Xiao, HE Jun-yan, et al. SWNet: a deep learning based approach for splashed water detection on road[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 2020: 1-14. [31] 白岗岗, 侯精明, 韩浩, 等. 基于深度学习的道路积水智能监测方法[J]. 水资源保护, 2020, https://kns.cnki.net/kcms/detail/32.1356.TV.20200806.1036.002.html.BAI Gang-gang, HOU Jing-ming, HAN Hao, et al. Intelligent monitoring method for road inundation based on deep learning[J]. Water Resources Protection, 2020, https://kns.cnki.net/kcms/detail/32.1356.TV.20200806.1036.002.html. (in Chinese) [32] 王海, 蔡柏湘, 蔡英凤, 等. 基于语义分割网络的路面积水与湿滑区域检测[J]. 汽车工程, 2021, 43(4): 485-491. https://www.cnki.com.cn/Article/CJFDTOTAL-QCGC202104007.htmWANG Hai, CAI Bai-xiang, CAI Ying-feng, et al. Detection of water-covered and wet areas on road pavement based on semantic segmentation network[J]. Automotive Engineering, 2021, 43(4): 485-491. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-QCGC202104007.htm [33] POHLEN T, HERMANS A, MATHIAS M, et al. Full-resolution residual networks for semantic segmentation in street scenes[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 3309-3318. [34] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Springer. Medical Image Computing and Computer-Assisted Intervention 2015. Berlin: Springer, 2015: 234-241. [35] LEE C Y, XIE S N, GALLAGHER, P W, et al. Deeply-supervised nets[J]. ArXiv, 2014: 562-570. [36] ALOM M Z, YAKOPCIC C, HASAN, M, et al. Recurrent residual U-Net for medical image segmentation[J]. Journal of Medical Imaging, 2019, 6(1): 014006. [37] SCHLEMPER J, OKTAY O, SCHAAP M, et al. Attention gated networks: learning to leverage salient regions in medical images[J]. Medical Image Analysis, 2019, 53: 197-207. [38] BEHESHTI N, JOHNSSON L. Squeeze U-Net: a memory and energy efficient image segmentation network[C]//IEEE. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE, 2020: 1495-1504. [39] NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[C]//IEEE. 2015 IEEE International Conference on Computer Vision. New York: IEEE, 2015: 1520-1528. [40] LI Yan-wei, CHEN Xin-ze, ZHU Zheng, et al. Attention-guided unified network for panoptic segmentation[C]//IEEE. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019: 7019-7028. -
点击查看大图
图(11) / 表(6)
计量
- 文章访问数: 653
- HTML全文浏览量: 197
- PDF下载量: 100
- 被引次数: 0
