基于融合非局部操作的YOLOv5s高密度锦鲤鱼苗检测方法

doi:10.3969/j.issn.1004-1524.20230621

浙江农业学报 ›› 2024, Vol. 36 ›› Issue (4): 952-967.DOI: 10.3969/j.issn.1004-1524.20230621

基于融合非局部操作的YOLOv5s高密度锦鲤鱼苗检测方法

汤永华¹(), 石非凡¹^,^*(), 林森², 张志鹏¹, 孟妍君¹, 刘兴通¹

1.沈阳工业大学信息科学与工程学院,辽宁沈阳 110870
2.沈阳理工大学自动化与电气工程学院,辽宁沈阳 110159

收稿日期:2023-05-12 出版日期:2024-04-25 发布日期:2024-04-29
作者简介:汤永华(1980—),男,山东寿光人,博士,讲师,主要从事深度学习、图像处理、数字系统研究。E-mail:tangyonghua@sut.edu.cn
通讯作者: *石非凡,E-mail:2328930037@qq.com
基金资助:
辽宁省机器人联合基金(20180520022);2023年度辽宁省应用基础研究计划(2023JH2/101300237)

YOLOv5s high-density koi fry detection method based on fusion non-local operation

TANG Yonghua¹(), SHI Feifan¹^,^*(), LIN Sen², ZHANG Zhipeng¹, MENG Yanjun¹, LIU Xingtong¹

1. School of Information Science and Engineering, Shenyang University of Technology, Shenyang 110870, China
2. School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang 110159, China

Received:2023-05-12 Online:2024-04-25 Published:2024-04-29
Contact: SHI Feifan

摘要/Abstract

摘要：

针对现有方法在高密度锦鲤鱼苗目标检测任务中适用性差的问题,提出一种基于非局部操作的YOLOv5s(MS-Non-local BIFPN coordinate attention YOLOv5s,NBC-YOLOv5s)目标检测算法。首先,在YOLOv5s的主干网络中,添加多尺度非局部操作算子(multi scale non-local, MS-Non-local),增强模型对高密度锦鲤鱼苗的特征提取能力;其次,在颈部网络使用双向加权特征金字塔结构(bi-directional feature pyramid network, BIFPN)提升模型特征融合效率;最后,在网络的特征融合处,引入坐标注意力机制(coordinate attention, CA),增加模型对图片关键信息的关注度。为验证本文算法的有效性,结合真实渔场环境建立锦鲤鱼苗数据集。实验结果表明,NBC-YOLOv5s的精确率、召回率、平均精度均值(mAP)分别为88.5%、89.7%、93.7%,与YOLOv5s相比,改进后网络较原模型分别提升0.6、9.0、4.4百分点。为验证MS-Non-local对YOLOv5s的性能提升效果,本文对比了卷积注意力(convolutional block attention module, CBAM)、通道注意力(squeeze and excitation, SE)、双层路由注意力(bi-level routing attention, BRA)3种机制。结果表明,MS-Non-local的mAP相较于CBAM、SE、BRA分别提升了2.6、2.1、0.9百分点。并且通过模型拆解,分析了本文方法对不同密度锦鲤鱼苗图像的检测有效性,结果显示,该算法可实现真实场景下对高密度锦鲤鱼苗的检测,能够为筛选高品质锦鲤提供有效技术支撑。

关键词: 锦鲤, 鱼苗检测, 高密度目标, YOLOv5s, 多尺度非局部操作算子

Abstract:

Aiming at the poor applicability of existing methods in the target detection task of high-density koi fry, a Ms-Non-local BIFPN coordinate attention YOLOv5s (NBC-YOLOv5s) target detection algorithm based on non-local operation is proposed. Firstly, in the backbone network of YOLOv5s, a multi-scale non-local operator (MS-Non-local) is added to enhance the feature extraction ability of the model for high-density koi fry. Secondly, the bi-directional feature pyramid network (BIFPN) is used in the neck network to improve the model feature fusion efficiency. Finally, at the feature fusion of the network, the coordinate attention (CA) mechanism is introduced to increase the model’s attention to the key information of the image. In order to verify the effectiveness of the proposed algorithm, a koi fry dataset was established based on the real fishery environment. The experimental results show that the precision, recall rate and mean average precision (mAP) of NBC-YOLOv5s are 88.5%, 89.7% and 93.7%, respectively, which are 0.6, 9.0 and 4.4 percentage points higher than the original model in the improved network compared with YOLOv5s. In order to verify the performance improvement effect of MS-Non-local on YOLOv5s, this paper compares the three mechanisms of convolutional block attention module (CBAM), squeeze and excitation (SE), and bi-level routing attention (BRA). The results showed that the mAP of MS-Non-local increased by 2.6, 2.1 and 0.9 percentage points compared with CBAM, SE and BRA, respectively. Through model disassembly, the effectiveness of the proposed method on the detection of images of koi fry of different densities is analyzed, and it is concluded that the algorithm can realize the detection of high-density koi fry in real scenarios, and can provide effective technical support for screening high-quality koi.

Key words: koi, fry detection, high-density target, YOLOv5s, Multi Scale-Non-local

中图分类号:

TP391.4

汤永华, 石非凡, 林森, 张志鹏, 孟妍君, 刘兴通. 基于融合非局部操作的YOLOv5s高密度锦鲤鱼苗检测方法[J]. 浙江农业学报, 2024, 36(4): 952-967.

TANG Yonghua, SHI Feifan, LIN Sen, ZHANG Zhipeng, MENG Yanjun, LIU Xingtong. YOLOv5s high-density koi fry detection method based on fusion non-local operation[J]. Acta Agriculturae Zhejiangensis, 2024, 36(4): 952-967.

图/表 19

图1 YOLOv5s网络结构 CONV,卷积层;BN,批量归一化层;CSP,跨阶段局部网络;SPPF,快速空间金字塔池化。下同。

Fig.1 YOLOv5s network structure CONV, Convolution; BN, Batch normalization; CSP, Cross stage partial; SPPF, Spatial pyramid pooling-fast. The same as below.

图2 NBC-YOLOv5s网络结构 SPP,空间金字塔池化;MS-NL,多尺度非局部操作算子;UP,上采样模块。

Fig.2 NBC-YOLOv5s network structure SPP, Spatial Pyramid Pooling; MS-NL, MS-Non-local; UP, Upsample.

图3 Non-local模块结构 T,批量输入图片数量;H,图像高度;W,图像宽度;C,图像通道数;X,输入特征图;Z,输出特征图。

Fig.3 Non-local module structure T, Batch size; H, Image height; W, Image width; C, Number of image channels; X, Input feature map; Z, Output feature map.

图4 Non-local Demo模块结构 θ、ϕ、g为3种卷积核;A,注意力图;M1,输出特征图;M,输入特征图。

Fig.4 Non-local Demo module structure θ, ϕ and g were three convolution kernels; A, Attention map; M1, Output feature map; M, Input feature map.

图5 MS-Non-local 模块结构 F,输入特征图;F1,输出特征图;θ、ϕ、g为3种卷积核。

Fig.5 MS-Non-local module structure F, Input feature map; F1, Output feature map; θ, ϕ and g were three convolution kernels.

图6 FPN、PAN、BIFPN结构 P,层数;in,输入;out,输出;td,中间层。

Fig.6 FPN、PAN、BIFPN structure P, Feature levels used for the final prediction; in, Input; out, Output; td, Intermediate layer.

图7 CA模块结构 H,图像高度;W,图像宽度;C,图像通道数。

Fig.7 CA module structure H, Image height; W, Image width; C, Number of image channels.

图8 优质鱼苗样图

Fig.8 Samples of good fish

图9 劣质鱼苗样图

Fig.9 Samples of bad fish

图10 不同密度情况锦鲤鱼苗样图

Fig.10 Sample drawings of koi fry at different densities

图11 数据集分析 A,锦鲤鱼苗类型分布;B,目标中心点分布;C,锦鲤鱼苗大小分布。

Fig.11 Dataset analysis A, Distribution of types of koi fry; B, Distribution of target center points; C, Size distribution of koi fry.

表1 不同注意力机制性能比较

Table 1 Performance comparison of different attention mechanisms

方法 Method	精确率 Precision/%	召回率 Recall/%	平均精度均值 mAP/%	占用内存 Memory/Mb	检测速度 Speed/ms
YOLOv5s+CBAM	86.3	85.1	89.5	28.2	35.1
YOLOv5s+SE	88.0	86.2	90.0	23.5	36.0
YOLOv5s+BRA	87.8	86.8	91.2	181.0	41.1
YOLOv5s+MS-Nonlocal	88.2	87.1	92.1	167.0	40.2

表2 不同算法性能比较

Table 2 Comparison of the performance of different algorithms

方法 Method	精确率 Precision/%	召回率 Recall/%	平均精度均值 mAP/%	占用内存 Memory/Mb	检测速度 Speed/ms
SSD	68.3	68.2	76.3	189	98
Faster Rcnn	65.1	60.0	58.1	186	191
YOLOv4	86.5	78.5	81.5	246	41
YOLOv5s	87.9	80.7	89.3	18	25
NBC-YOLOv5s	88.5	89.7	93.7	213	35

图12 不同算法检测结果

Fig.12 Detection results of different algorithms

表3 前沿算法性能比较

Table 3 Comparison of leading-edge algorithm performance

方法 Method	精确率 Precision/%	召回率 Recall/%	平均精度均值 mAP/%	占用内存 Memory/Mb	检测速度 Speed/ms
YOLOv6s	95.5	87.0	93.60	40	28
YOLOv7	82.1	90.2	93.06	298	13
NBC-YOLOv5s	88.5	89.7	93.70	213	35

图13 算法检测结果对比

Fig.13 Comparison of algorithm detection results

表4 消融实验结果

Table 4 Results of ablation experiment

方法 Method	添加CA注意力 Add CA	修改特征金字塔 Modify BIFPN	添加MS-Non-local Add MS-Non-local	平均精度均值 mAP/%	检测速度 Speed/ms
YOLOv5s	×	×	×	89.3	25
1	√	×	×	90.1	26
2	×	√	×	89.7	26
3	×	×	√	92.1	40
4	√	×	√	91.3	32
5	√	√	×	91.0	27
6	×	√	√	92.3	33
NBC-YOLOv5s	√	√	√	93.7	35

表5 改进模型在高密度锦鲤鱼苗情况下的性能对比

Table 5 Performance comparison of improved models in high-density koi fry %

方法 Method	精确率 Precision	召回率 Recall	平均精度均值 mAP
YOLOv5s	84.8	77.9	84.6
1	83.4	85.7	89.4
2	87.2	86.1	90.7
3	85.7	86.3	90.8
4	88.1	85.1	90.3
5	90.2	85.8	91.7
6	89.4	84.6	91.2
NBC-YOLOv5s	90.9	87.6	92.2

图14 不同密度的消融实验结果对比

Fig.14 Comparison of ablation experimental results of different densities

参考文献 34

[1]	陈俊杰, 李媛瑗, 阳瑞雪, 等. 四川地区一株锦鲤疱疹病毒的分离鉴定及系统进化分析[J]. 水产学报, 2018, 42(1): 141-148.
	CHEN J J, LI Y Y, YANG R X, et al. Isolation and identification of a koi herpesvirus and phylogenetic analysis from Sichuan Province[J]. Journal of Fisheries of China, 2018, 42(1): 141-148. (in Chinese with English abstract)
[2]	康慧敏, 何嘉乐, 刘晨恺, 等. 四川地区养殖锦鲤浮肿病毒检测与系统进化分析[J]. 浙江农业学报, 2019, 31(4): 539-544.
	KANG H M, HE J L, LIU C K, et al. Detection and phylogenetic analysis of carp edema virus (CEV) in farmed koi carp in Sichuan Province[J]. Acta Agriculturae Zhejiangensis, 2019, 31(4): 539-544. (in Chinese with English abstract)
[3]	杨志强, 李潇轩, 殷悦, 等. 锦鲤苗种筛选技术及成鱼鉴赏标准[J]. 水产养殖, 2022, 43(9): 49-50.
	YANG Z Q, LI X X, YIN Y, et al. Screening technology of fry and appreciation standard of adult fish in koi fish[J]. Journal of Aquaculture, 2022, 43(9): 49-50. (in Chinese)
[4]	GRAY P C, FLEISHMAN A B, KLEIN D J, et al. A convolutional neural network for detecting sea turtles in drone imagery[J]. Methods in Ecology and Evolution, 2019, 10(3): 345-355.
[5]	陈明, 张重阳, 冯国富, 等. 基于特征加权融合的鱼类摄食活动强度评估方法[J]. 农业机械学报, 2020, 51(2): 245-253.
	CHEN M, ZHANG C Y, FENG G F, et al. Intensity assessment method of fish feeding activities based on feature weighted fusion[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(2): 245-253. (in Chinese with English abstract)
[6]	张明华, 龙腾, 宋巍, 等. 一种水下鱼类动态视觉序列运动目标检测方法[J]. 图学学报, 2021, 42(1): 52-58.
	ZHANG M H, LONG T, SONG W, et al. Method for moving object detection of underwater fish using dynamic video sequence[J]. Journal of Graphics, 2021, 42(1): 52-58. (in Chinese with English abstract)
[7]	TANAKA Y, YAMABE S, FUKAE K, et al. Quantification of fish behavior using optical flow[C]//2022 IEEE 11th Global Conference on Consumer Electronics. October 18-21, 2022.: Osaka, Japan. IEEE, 2022: 125-126.
[8]	SALMAN A, MAQBOOL S, KHAN A H, et al. Real-time fish detection in complex backgrounds using probabilistic background modelling[J]. Ecological Informatics, 2019, 51: 44-51.
[9]	THARWAT A, HEMEDAN A A, HASSANIEN A E, et al. A biometric-based model for fish species classification[J]. Fisheries Research, 2018, 204: 324-336.
[10]	樊闯, 赵子皓, 张雪松, 等. 基于BP神经网络的一季稻发育期预测模型[J]. 浙江农业学报, 2023, 35(2): 434-444.
	FAN C, ZHAO Z H, ZHANG X S, et al. Prediction model of one season rice development period based on BP neural network[J]. Acta Agriculturae Zhejiangensis, 2023, 35(2): 434-444. (in Chinese)
[11]	李斌, 刘东阳, 时国龙, 等. 基于改进YOLOv4模型的群养生猪姿态检测[J]. 浙江农业学报, 2023, 35(1): 215-225.
	LI B, LIU D Y, SHI G L, et al. Pig posture detection based on improved YOLOv4 model[J]. Acta Agriculturae Zhejiangensis, 2023, 35(1): 215-225. (in Chinese with English abstract)
[12]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016. Las Vegas, NV, USA.: IEEE, 2016: 770-778.
[13]	刘雨青, 王亚茹, 黄璐瑶. 基于FML-Centernet算法的鱼类识别检测[J]. 激光与光电子学进展, 2022, 59(16): 1615002.
	LIU Y Q, WANG Y R, HUANG L Y. Fish recognition and detection based on FML-centernet algorithm[J]. Laser & Optoelectronics Progress, 2022, 59(16): 1615002. (in Chinese with English abstract)
[14]	吴立栋, 彭宗举, 李欣, 等. 基于样本迭代融合的海洋生物检测[J]. 激光与光电子学进展, 2023, 60(2): 3788/LOP212567.
	WU L D, PENG Z J, LI X, et al. Marine creature detection based on sample iterative fusion[J]. Laser & Optoelectronics Progress, 2023, 60(2): 3788/LOP212567. (in Chinese with English abstract)
[15]	杨文静, 陈明, 冯国富. 基于图像增强的水下视频鱼类识别方法[J]. 激光与光电子学进展, 2021, 58(22): 2211002.
	YANG W J, CHEN M, FENG G F. Fish recognition method for underwater video based on image enhancement[J]. Laser & Optoelectronics Progress, 2021, 58(22): 2211002. (in Chinese with English abstract)
[16]	SALMAN A, AHMAD SIDDIQUI S, SHAFAIT F, et al. Automatic fish detection in underwater videos by a deep neural network-based hybrid motion learning system[J]. ICES Journal of Marine Science, 2020, 77(4): 1295-1307.
[17]	ZHAO Z X, LIU Y, SUN X D, et al. Composited FishNet: fish detection and species recognition from low-quality underwater videos[J]. IEEE Transactions on Image Processing, 2021, 30: 4719-4734.
[18]	FAROOQ M, HAFEEZ A. Covid-resnet: a deep learning framework for screening of covid19 from radiographs[EB/OL]. (2020-05-31) [2023-05-10].https://arxiv.org/abs/2003.14395.
[19]	WU W X, QI Z A, LI F X. PointConv: deep convolutional networks on 3D point clouds[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 15-20, 2019. Long Beach, CA, USA. IEEE, 2019: 9621-9630.
[20]	ALBAHLI S, NAWAZ M, JAVED A, et al. An improved faster-RCNN model for handwritten character recognition[J]. Arabian Journal for Science and Engineering, 2021, 46(9): 8509-8523.
[21]	KASPER-EULAERS M, HAHN N, BERGER S, et al. Short communication: detecting heavy goods vehicles in rest areas in winter conditions using YOLOv5[J]. Algorithms, 2021, 14(4): 114.
[22]	WANG C Y, MARK LIAO H Y, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. June 14-19, 2020. Seattle, WA, USA. IEEE, 2020: 390-391.
[23]	WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018. Salt Lake City, UT, USA. IEEE, 2018: 7794-7803.
[24]	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. July 21-26, 2017. Honolulu, HI. IEEE, 2017: 2117-2125.
[25]	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018. Salt Lake City, UT. IEEE, 2018: 8759-8768.
[26]	TAN M X, PANG R M, LE Q V. EfficientDet: scalable and efficient object detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 13-19, 2020. Seattle, WA, USA. IEEE, 2020: 10781-10790.
[27]	SINHA D, EL-SHARKAWY M. Thin MobileNet: an enhanced MobileNet architecture[C]//2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference. October 10-12, 2019. New York City, NY, USA. IEEE, 2019: 0280-0285.
[28]	HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 20-25, 2021. Nashville, TN, USA. IEEE, 2021: 13713-13722.
[29]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018. Salt Lake City, UT. IEEE, 2018: 7132-7141.
[30]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]// Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
[31]	ZHU L, WANG X J, KE Z H, et al. BiFormer: vision transformer with Bi-level routing attention[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 17-24, 2023. Vancouver, BC, Canada. IEEE, 2023: 10323-10333.
[32]	LI C Y, LI L, JIANG H L, et al. YOLOv6: a single-stage object detection framework for industrial applications[EB/OL]. ( 2022-09-27[2023-05-10].https://arxiv.org/abs/2209.02976.
[33]	WANG C Y A. BOCHKOVSKIY A, H Y M LIAO. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, IEEE, 2023: 7464-7475.
[34]	DING X H, ZHANG X Y, MA N N, et al. RepVGG: making VGG-style ConvNets great again[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 20-25, 2021. Nashville, TN, USA. IEEE, 2021: 13733-13742.

基于融合非局部操作的YOLOv5s高密度锦鲤鱼苗检测方法

YOLOv5s high-density koi fry detection method based on fusion non-local operation

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 19

参考文献 34

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杨新宇, 冯全, 张建华, 杨森. 基于对比学习的植物叶片病害识别[J]. 浙江农业学报, 2024, 36(1): 215-224.
[2]	刘媛媛, 王定坤, 邬雷, 黄德昌, 朱路. 基于知识蒸馏和模型剪枝的轻量化模型植物病害识别[J]. 浙江农业学报, 2023, 35(9): 2250-2264.
[3]	张䶮, 周保平, 王昱, 冯洁, 叶凡恺, 何云龙. 基于迁移学习和改进残差网络的棉花叶螨为害等级识别[J]. 浙江农业学报, 2023, 35(7): 1729-1739.
[4]	李斌, 刘东阳, 时国龙, 慕京生, 徐浩然, 辜丽川, 焦俊. 基于改进YOLOv4模型的群养生猪姿态检测[J]. 浙江农业学报, 2023, 35(1): 215-225.
[5]	闫宁, 张晗, 董宏图, 康凯, 罗斌. 基于透射光和反射光图像同位分割的小麦品种识别方法研究[J]. 浙江农业学报, 2022, 34(3): 590-598.
[6]	林孝腾, 孙茜, 郑书河. 基于OpenCV的低能耗植物工厂照明系统设计与试验[J]. 浙江农业学报, 2022, 34(3): 626-635.
[7]	肖志云, 王伊凝. 基于RF-VR的紫丁香叶片叶绿素含量高光谱反演[J]. 浙江农业学报, 2021, 33(11): 2164-2173.
[8]	钟伟镇, 刘鑫磊, 杨坤龙, 李丰果. 基于Mask-RCNN的复杂背景下多目标叶片的分割和识别[J]. 浙江农业学报, 2020, 32(11): 2059-2066.
[9]	闫建伟, 苏小东, 赵源, 刘进平. 基于图像特征的小麦胚芽鞘识别[J]. 浙江农业学报, 2019, 31(2): 326-332.
[10]	张满囤, 单新媛, 于洋, 米娜, 阎刚, 郭迎春. 基于小波变换和改进KPCA的奶牛个体识别研究[J]. 浙江农业学报, 2017, 29(12): 2000-2008.
[11]	毕立恒. 基于叶片图像算法的植物种类识别方法研究[J]. 浙江农业学报, 2017, 29(12): 2142-2148.
[12]	张善文, 黄文准, 尤著宏. 基于物联网和深度卷积神经网络的冬枣病害识别方法[J]. 浙江农业学报, 2017, 29(11): 1868-1874.
[13]	李颀, 胡艺聪, 武付闯. 基于图像处理的育苗箱环境控制系统[J]. 浙江农业学报, 2017, 29(11): 1912-1919.
[14]	夏永泉, 王兵, 支俊, 黄海鹏, 孙静茹. 基于EM和K-means混合聚类方法的植物叶片病害区域自动提取[J]. 浙江农业学报, 2017, 29(8): 1384-1391.
[15]	吴庆岗, 张卫国, 常化文, 金保华, 刘朝霞. 基于梯度Hough变换的遮挡苹果目标定位[J]. 浙江农业学报, 2017, 29(6): 1009-1016.