Acta Agriculturae Zhejiangensis ›› 2023, Vol. 35 ›› Issue (1): 202-214.DOI: 10.3969/j.issn.1004-1524.2023.01.22
• Biosystems Engineering • Previous Articles Next Articles
ZHU Shisong1(), MA Wanli1, ZHAO Lishan1, ZHENG Yanmei1, ZHENG Xianbo2, LU Bibo*(
)
Received:
2022-04-06
Online:
2023-01-25
Published:
2023-02-21
CLC Number:
ZHU Shisong, MA Wanli, ZHAO Lishan, ZHENG Yanmei, ZHENG Xianbo, LU Bibo. Apple leaf image segmentation algorithm based on improved LinkNet[J]. Acta Agriculturae Zhejiangensis, 2023, 35(1): 202-214.
Add to citation manager EndNote|Ris|BibTeX
URL: http://www.zjnyxb.cn/EN/10.3969/j.issn.1004-1524.2023.01.22
Fig.1 Data annotation and processing a, Image; b, Label; c, Binaryzation. The upper left corners of b and c correspond to the contents of the yellow rectangular box in a, respectively. There was a clear transition line at the junction of the leaf and the background in figure b, which was not present in figure c.
Fig.2 Framework of LinkNet and internal structure of each block Conv, Convolution operation; De-Conv, Deconvolution operation; Max-pool, Maximum pooling; (7×7) and (3×3) meant that the size of the convolution kernel used were 7×7, 3×3, respectively; (m, n) indicated that the number of input channels was m and the number of output channels was n; (m, m/4) denoted that the number of output channels is 1/4 of that of input channels; /2 indicated a convolution or maximum pooling operation with a step size of 2, after which the size of the feature map was reduced to 1/2 of its original size, which was the down-sampling process; In comparison, *2 meant a step of 2 for the deconvolution operation, after which the feature size was magnified by 2 times, which was the up-sampling process.
Fig.5 Sub-pixel convolution H and W represented the height and width of the feature map, respectively. C indicated the number of channels, and r denoted the multiplier of up-sampling.
主要网络层 Main layers of the network | 高×宽/pixel Height× Width/pixel | 数量 Number |
---|---|---|
输入图像Input image | 1 184×800 | — |
第2层Layer 2 | 592×400 | 64 |
第6层Layer 6 | 592×400 | 64 |
第10层Layer 10 | 296×200 | 128 |
第14层Layer 14 | 148×100 | 256 |
第17层Layer 17 | 296×200 | 128 |
第20层Layer 20 | 592×400 | 64 |
第23层Layer 23 | 592×400 | 64 |
第25层Layer 25 | 1 184×800 | 2 |
输出图像Output image | 1 184×800 | — |
Table 1 Size and number of feature maps output from the main network layers
主要网络层 Main layers of the network | 高×宽/pixel Height× Width/pixel | 数量 Number |
---|---|---|
输入图像Input image | 1 184×800 | — |
第2层Layer 2 | 592×400 | 64 |
第6层Layer 6 | 592×400 | 64 |
第10层Layer 10 | 296×200 | 128 |
第14层Layer 14 | 148×100 | 256 |
第17层Layer 17 | 296×200 | 128 |
第20层Layer 20 | 592×400 | 64 |
第23层Layer 23 | 592×400 | 64 |
第25层Layer 25 | 1 184×800 | 2 |
输出图像Output image | 1 184×800 | — |
试验序号 Test No. | 主干网络 Backbone | 平均交并比 mIoU/% | 均像素精度 mPA/% | 训练总时长 Total training time/h | 推理时间 Inference time/ms | 参数量 Paras/M | 浮点数计算量 FLOPs/G |
---|---|---|---|---|---|---|---|
1 | — | 96.15 | 98.01 | 4.82 | 13.01 | 11.54 | 43.69 |
2 | ResNet18 | 96.18 | 98.74 | 3.42 | 12.00 | 11.53 | 43.64 |
3 | ResNet34 | 96.69 | 98.93 | 3.48 | 16.39 | 21.64 | 78.60 |
Table 2 Comparison of experimental results with the introduction of different pre-trained models
试验序号 Test No. | 主干网络 Backbone | 平均交并比 mIoU/% | 均像素精度 mPA/% | 训练总时长 Total training time/h | 推理时间 Inference time/ms | 参数量 Paras/M | 浮点数计算量 FLOPs/G |
---|---|---|---|---|---|---|---|
1 | — | 96.15 | 98.01 | 4.82 | 13.01 | 11.54 | 43.69 |
2 | ResNet18 | 96.18 | 98.74 | 3.42 | 12.00 | 11.53 | 43.64 |
3 | ResNet34 | 96.69 | 98.93 | 3.48 | 16.39 | 21.64 | 78.60 |
试验序号 Test No. | 不同改进结构Different improved structures | 平均交并比 mIoU/% | 均像素精度 mPA/% | 推理时间 Inference time/ms | 参数量 Paras/M | 浮点数 计算量 FLOPs/G | ||
---|---|---|---|---|---|---|---|---|
编码-解码块 Encoder-Decoder blocks | 通道约减 Channel reduction | 最终块 Final block | ||||||
1 | 4 | m/4 | — | 96.18 | 98.74 | 12.00 | 11.53 | 43.64 |
2 | 3 | m/4 | — | 96.02 | 98.04 | 10.82 | 2.89 | 35.14 |
3 | 4 | m/8 | — | 96.17 | 98.12 | 11.75 | 11.32 | 41.97 |
4 | 4 | m/16 | — | 96.16 | 98.02 | 11.64 | 11.25 | 41.47 |
5 | 4 | m/32 | — | 95.45 | 97.54 | 11.56 | 11.24 | 41.36 |
6 | 4 | m/4 | Sub-pixel conv | 95.42 | 97.41 | 9.64 | 11.52 | 37.89 |
7 | 3 | m/16 | Sub-pixel conv | 95.01 | 97.27 | 7.82 | 2.81 | 27.88 |
Table 3 Ablation studies design and comparison of the results
试验序号 Test No. | 不同改进结构Different improved structures | 平均交并比 mIoU/% | 均像素精度 mPA/% | 推理时间 Inference time/ms | 参数量 Paras/M | 浮点数 计算量 FLOPs/G | ||
---|---|---|---|---|---|---|---|---|
编码-解码块 Encoder-Decoder blocks | 通道约减 Channel reduction | 最终块 Final block | ||||||
1 | 4 | m/4 | — | 96.18 | 98.74 | 12.00 | 11.53 | 43.64 |
2 | 3 | m/4 | — | 96.02 | 98.04 | 10.82 | 2.89 | 35.14 |
3 | 4 | m/8 | — | 96.17 | 98.12 | 11.75 | 11.32 | 41.97 |
4 | 4 | m/16 | — | 96.16 | 98.02 | 11.64 | 11.25 | 41.47 |
5 | 4 | m/32 | — | 95.45 | 97.54 | 11.56 | 11.24 | 41.36 |
6 | 4 | m/4 | Sub-pixel conv | 95.42 | 97.41 | 9.64 | 11.52 | 37.89 |
7 | 3 | m/16 | Sub-pixel conv | 95.01 | 97.27 | 7.82 | 2.81 | 27.88 |
模型 Model | 平均交并比 mIoU/% | 均像素精度 mPA/% | 推理时间 Inferernce time/ms | 参数量 Paras/M | 浮点数计算量 FLOPs/G |
---|---|---|---|---|---|
FCN | 95.80 | 97.81 | 36.42 | 20.10 | 322.98 |
U-Net | 96.27 | 98.75 | 89.82 | 34.53 | 946.04 |
DeepLabV3+ | 98.51 | 99.36 | 53.58 | 54.61 | 299.30 |
ENet | 95.24 | 98.08 | 21.16 | 13.64 | 84.60 |
BiSeNet | 92.50 | 97.24 | 35.67 | 23.07 | 146.79 |
Improved LinkNet | 95.01 | 97.27 | 7.82 | 2.81 | 27.88 |
Table 4 Segmentation results of apple leaf by using different network models
模型 Model | 平均交并比 mIoU/% | 均像素精度 mPA/% | 推理时间 Inferernce time/ms | 参数量 Paras/M | 浮点数计算量 FLOPs/G |
---|---|---|---|---|---|
FCN | 95.80 | 97.81 | 36.42 | 20.10 | 322.98 |
U-Net | 96.27 | 98.75 | 89.82 | 34.53 | 946.04 |
DeepLabV3+ | 98.51 | 99.36 | 53.58 | 54.61 | 299.30 |
ENet | 95.24 | 98.08 | 21.16 | 13.64 | 84.60 |
BiSeNet | 92.50 | 97.24 | 35.67 | 23.07 | 146.79 |
Improved LinkNet | 95.01 | 97.27 | 7.82 | 2.81 | 27.88 |
[1] | 于东玉, 冯天祥, 李奕昕, 等. 基于植物图像的活体叶片面积测量方法研究与实现[J]. 智能计算机与应用, 2019, 9(4): 173-176. |
YU D Y, FENG T X, LI Y X, et al. Research and implementation of living leaf area measurement based on plant image[J]. Intelligent Computer and Applications, 2019, 9(4): 173-176. (in Chinese with English abstract) | |
[2] | 李杨先, 张慧春, 杨旸. 一种基于图像处理技术的植物形态表型参数获取方法[J]. 林业工程学报, 2020, 5(6): 128-136. |
LI Y X, ZHANG H C, YANG Y. A method for obtaining plant morphological phenotypic parameters using image processing technology[J]. Journal of Forestry Engineering, 2020, 5(6): 128-136. (in Chinese with English abstract)
DOI URL |
|
[3] | 李方一, 黄璜, 官春云. 作物叶面积测量的研究进展[J]. 湖南农业大学学报(自然科学版), 2021, 47(3): 274-282. |
LI F Y, HUANG H, GUAN C Y. Review on measurement of crop leaf area[J]. Journal of Hunan Agricultural University (Natural Sciences), 2021, 47(3): 274-282. (in Chinese with English abstract) | |
[4] | 吴千华, 孟祥丽, 刘一鸣. 植物叶面积测定方法探讨[J]. 热带林业, 2018, 46(2): 37-39. |
WU Q H, MENG X L, LIU Y M. Discussion about the methods of measuring leaf area of plants[J]. Tropical Forestry, 2018, 46(2): 37-39. (in Chinese with English abstract) | |
[5] | 孙玉婷, 杨红云, 王映龙, 等. 基于支持向量机的水稻叶面积测定[J]. 江苏农业学报, 2018, 34(5): 1027-1035. |
SUN Y T, YANG H Y, WANG Y L, et al. Determination of rice leaf area based on support vector machine[J]. Jiangsu Journal of Agricultural Sciences, 2018, 34(5): 1027-1035. (in Chinese with English abstract) | |
[6] | CRISTOFORI V, GYVES E M, BIGNAMI C, et al. Non-destructive, analogue model for leaf area estimation in persimmon (Diospyros kaki L.F.) based on leaf length and width measurement[J]. European Journal of Horticultural Science, 2008, 73(5):216-221. |
[7] | 刘小锐, 黄成东, 祝红伟. 叶用莴苣叶面积测定方法的研究[J]. 中国蔬菜, 2020(12): 78-81. |
LIU X R, HUANG C D, ZHU H W. Studies on measuring methods of leaf area in lettuce[J]. China Vegetables, 2020(12): 78-81. (in Chinese with English abstract) | |
[8] | 林权虹, 陈胧, 郑禹轩, 等. 植物比叶面积量化:ImageJ软件法与打孔称重法的比较[J]. 首都师范大学学报(自然科学版), 2022, 43(1): 48-55. |
LIN Q H, CHEN L, ZHENG Y X, et al. Plant specific leaf area: comparison of ImageJ software method and disc method[J]. Journal of Capital Normal University (Natural Science Edition), 2022, 43(1): 48-55. (in Chinese with English abstract) | |
[9] | 王永皎, 张引, 张三元. 基于图像处理的植物叶面积测量方法[J]. 计算机工程, 2006, 32(8): 210-212. |
WANG Y J, ZHANG Y, ZHANG S Y. Approach to measure plant leaf area based on image process[J]. Computer Engineering, 2006, 32(8): 210-212. (in Chinese with English abstract) | |
[10] | 热娜古丽·热西提, 刘生智, 刘同金, 等. 植物叶面积测量方法综述[J]. 安徽农学通报, 2020, 26(5): 22-23. |
RENAGULI·R, LIU S Z, LIU T J, et al. Survey of plant leaf area measurement methods[J]. Anhui Agricultural Science Bulletin, 2020, 26(5): 22-23. (in Chinese with English abstract) | |
[11] |
GAO M, VAN DER HEIJDEN G W A M, VOS J, et al. Estimation of leaf area for large scale phenotyping and modeling of rose genotypes[J]. Scientia Horticulturae, 2012, 138: 227-234.
DOI URL |
[12] | 曹栎, 赵小光, 王竹云, 等. 甘蓝型油菜叶面积测定方法的研究[J]. 江西农业学报, 2018, 30(8): 33-36. |
CAO Y, ZHAO X G, WANG Z Y, et al. Study on method for determination of leaf area of Brassica napus[J]. Acta Agriculturae Jiangxi, 2018, 30(8): 33-36. (in Chinese with English abstract) | |
[13] | 张万红. 基于图像分割的苹果叶片几何参数计算[J]. 中国农业大学学报, 2018, 23(8): 101-108. |
ZHANG W H. Calculating geometrical parameters of apple leaf based on the image segmentation[J]. Journal of China Agricultural University, 2018, 23(8): 101-108. (in Chinese with English abstract) | |
[14] |
刘哲, 张善文, 黄文准, 等. 基于双远心镜头的高精度叶面积测量方法研究[J]. 中国农机化学报, 2020, 41(4): 138-145.
DOI |
LIU Z, ZHANG S W, HUANG W Z, et al. Study on the measurement method of high precision leaf area based on telecentric lens[J]. Journal of Chinese Agricultural Mechanization, 2020, 41(4): 138-145. (in Chinese with English abstract) | |
[15] |
施滢, 夏春华. 基于棋盘格的可变参照物叶面积测量方法研究[J]. 中国农机化学报, 2021, 42(5): 191-196.
DOI |
SHI Y, XIA C H. Research on leaf area measurement of variable reference based on checkerboards[J]. Journal of Chinese Agricultural Mechanization, 2021, 42(5): 191-196. (in Chinese with English abstract) | |
[16] |
EASLON H M, BLOOM A J. Easy Leaf Area: automated digital image analysis for rapid and accurate measurement of leaf area[J]. Applications in Plant Sciences, 2014, 2(7): 1400033.
DOI URL |
[17] | LIN K, CHEN J, SI H, et al. Measurement of leaf area index using reference object and computer vision[J]. Oxidation Communications, 2016, 39(2A): 1832-1844. |
[18] |
SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
DOI PMID |
[19] | RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, 2015. |
[20] | CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[J]. Computer Science, 2014(4):357-361. |
[21] |
CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
DOI URL |
[22] | CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. 2017. https://arxiv.org/abs/1706.05587 |
[23] | CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]// Computer Vision-ECCV 2018, 2018. |
[24] | 赵兵, 冯全. 基于全卷积网络的葡萄病害叶片分割[J]. 南京农业大学学报, 2018, 41(4): 752-759. |
ZHAO B, FENG Q. Segmentation of grape diseases leaf based on full convolution network[J]. Journal of Nanjing Agricultural University, 2018, 41(4): 752-759. (in Chinese with English abstract) | |
[25] |
TASSIS L M, DE SOUZA J E T, KROHLING R A. A deep learning approach combining instance and semantic segmentation to identify diseases and pests of coffee leaves from in-field images[J]. Computers and Electronics in Agriculture, 2021, 186: 106191.
DOI URL |
[26] |
HUSSEIN B R, MALIK O A, ONG W H, et al. Automated extraction of phenotypic leaf traits of individual intact herbarium leaves from herbarium specimen images using deep learning based semantic segmentation[J]. Sensors (Basel, Switzerland), 2021, 21(13): 4549.
DOI URL |
[27] |
REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
DOI PMID |
[28] | 李余康, 翟长远, 王秀, 等. 基于DeepLab v3+的葡萄叶片分割算法[J]. 农机化研究, 2022, 44(2): 149-155. |
LI Y K, ZHAI C Y, WANG X, et al. Grape leaf segmentation based on DeepLab V3+[J]. Journal of Agricultural Mechanization Research, 2022, 44(2): 149-155. (in Chinese with English abstract) | |
[29] | PASZKE A, CHAURASIA A, KIM S, et al. ENet: a deep neural network architecture for real-time semantic segmentation[EB/OL]. 2016. https://arxiv.org/pdf/1606.02147.pdf. |
[30] | YU C Q, WANG J B, PENG C, et al. BiSeNet: bilateral segmentation network for real-time semantic segmentation[C]// Computer Vision - ECCV 2018, 2018. |
[31] | CHAURASIA A, CULURCIELLO E. LinkNet: exploiting encoder representations for efficient semantic segmentation[C]// 2017 IEEE Visual Communications and Image Processing. St. Petersburg, FL, USA. IEEE, 2017. |
[32] | 郑艳梅, 黄光耀, 芦碧波, 等. 结合深度学习和引导滤波的苹果叶片图像分割[J]. 中国农业大学学报, 2020, 25(9): 112-120. |
ZHENG Y M, HUANG G Y, LU B B, et al. An apple leaf image segmentation algorithm using deep learning and guided filtering[J]. Journal of China Agricultural University, 2020, 25(9): 112-120. (in Chinese with English abstract) | |
[33] | HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA. IEEE, 2016 : 770-778. |
[34] | 夏雪, 李壮, 吴定峰, 等. 苹果品种标准叶片图像和光谱数据集[J]. 中国科学数据, 2016, 1(1): 43-48. |
XIA X, LI Z, WU D F, et al. An image and spectral dataset for standard leaves of apple varieties[J]. China Scientific Data, 2016, 1(1): 43-48. (in Chinese with English abstract) | |
[35] |
PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
DOI URL |
[36] | LU Z, PU H M, WANG F C, et al. The expressive power of neural networks: a view from the width[EB/OL]. 2017: arXiv: 1709.02540[cs.LG]. https://arxiv.org/abs/1709.02540 |
[37] | SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA. IEEE, 2016: 1874-1883. |
[38] |
WU S, LI G Q, DENG L, et al. L1-norm batch normalization for efficient training of deep neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(7): 2043-2051.
DOI URL |
[39] | HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaption of feature detectors[J]. Computer Science, 2012, 3(4):212-223. |
[40] | LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]// 2017 IEEE International Conference on Computer Vision (ICCV), 2017: 2999-3007. |
[1] | CHEN Daohuai, WANG Hangjun. Detection of forest pests based on improved YOLOv4 [J]. Acta Agriculturae Zhejiangensis, 2022, 34(6): 1306-1315. |
[2] | LI Chao, LI Feng, HUANG Weijia. Fruit variety recognition based on parallel convolutional neural network [J]. Acta Agriculturae Zhejiangensis, 2022, 34(11): 2533-2541. |
[3] | ZHANG Qingqing, LIU Lianzhong, NING Jingming, WU Guodong, JIANG Zhaohui, LI Mengjie, LI Dongliang. Tea buds recognition under complex scenes based on optimized YOLOV3 model [J]. Acta Agriculturae Zhejiangensis, 2021, 33(9): 1740-1747. |
[4] | BAO Lie, WANG Mantao, LIU Jiangchuan, WEN Bo, MING Yue. Estimation method of wheat yield based on convolution neural network [J]. Acta Agriculturae Zhejiangensis, 2020, 32(12): 2244-2252. |
[5] | WANG Yanxiang, ZHANG Yan, YANG Chengya, MENG Qinglong, SHANG Jing. Advances in new nondestructive detection and identification techniques of crop diseases based on deep learning [J]. , 2019, 31(4): 669-676. |
[6] | YANG Guoliang, XU Nan, KANG Lele, GONG Man, HONG Zhiyang. Identification of navel orange lesions leaves based on parametric exponential non-linear residual neural network [J]. , 2018, 30(6): 1073-1081. |
[7] | YANG Fuqin, FENG Haikuan, LI Zhenhai, YANG Guijun, DAI Huayang. Estimation of apple leaf chlorophyll content based on hyperspectral data [J]. , 2017, 29(10): 1742-1748. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||