A light-weight model for plant disease identification based on model pruning and knowledge distillation

doi:10.3969/j.issn.1004-1524.20221193

Acta Agriculturae Zhejiangensis ›› 2023, Vol. 35 ›› Issue (9): 2250-2264.DOI: 10.3969/j.issn.1004-1524.20221193

• Biosystems Engineering • Previous Articles Next Articles

A light-weight model for plant disease identification based on model pruning and knowledge distillation

LIU Yuanyuan(), WANG Dingkun, WU Lei, HUANG Dechang, ZHU Lu()

School of Information Engineering, East China Jiaotong University, Nanchang 330013, China

Received:2022-08-14 Online:2023-09-25 Published:2023-10-09

Abstract

Abstract:

The emergence of deep learning has provided a new method for plant disease identification, but the current deep learning models have many parameters, which are difficult to use on edge devices such as smartphones or embedded sensor nodes with limited storage and computing resources. In the present study, plant leaves are taken as the research objects, and the methods based on knowledge distillation and model pruning are used to construct a light-weight model for plant disease identification. By improving the ResNet model, one or more teaching assistant network training models are introduced into the knowledge distillation. After sparse training, a light-weight student network model is obtained by model pruning; and the student network is retrained by using the teaching assistant network and learning rate rewinding, which can reduce the size of the model and effectively ensure the performance of the model. The experimental results show that, on a dataset including 38 categories of 14 plants, after pruning the model by 90%, the accuracy of the model is 97.78%, with an increase of 1.49 percentage points over the original model. On the dataset including 5 categories of apple leaves, after pruning the model by 70%, the accuracy of the model is 91.94%, which is 4.85 percentage points higher than the original model. The proposed light-weight model can be transplanted on Android platform and run effectively, which provides a new solution for the embedded terminal to accurately identify plant diseases.

Key words: disease identification, model pruning, knowledge distillation, learning rate rewinding, residual network

CLC Number:

LIU Yuanyuan, WANG Dingkun, WU Lei, HUANG Dechang, ZHU Lu. A light-weight model for plant disease identification based on model pruning and knowledge distillation[J]. Acta Agriculturae Zhejiangensis, 2023, 35(9): 2250-2264.

Figures/Tables 17

Table 1 Basic informatino of New Plant Diseases Dataset

植物 Plant	类别(编号) Category (No.)	训练集图像数量 Images quantity in training set	测试集图像数量 Image quantity in test set
苹果Apple	疮痂病Scab (0)	2 016	504
	黑腐病Black rot (1)	1 987	497
	锈病Cedar rust (2)	1 760	440
	健康Healthy (3)	2 008	502
蓝莓Blueberry	健康Healthy (4)	1 816	454
樱桃Cherry	白粉病Powdery mildew (5)	1 683	421
	健康Healthy (6)	1 826	456
玉米Maize	灰叶斑病Gray leaf spot (7)	1 642	410
	锈病Common rust (8)	1 907	477
	北方叶枯病Northern leaf blight (9)	1 908	477
	健康Healthy (10)	1 859	465
葡萄Grape	黑腐病Black rot (11)	1 888	472
	黑麻疹Black measles (12)	1 920	480
	叶枯病Leaf blight (13)	1 722	430
	健康Healthy (14)	1 692	423
橘子Orange	黄龙病Huanglongbing (15)	2 010	503
桃Peach	桃菌斑Bacterial spot (16)	1 838	459
	健康Healthy (17)	1 728	432
胡椒Pepper	铃菌斑Bacterial spot (18)	1 913	478
	健康Healthy (19)	1 988	492
马铃薯Potato	早疫病Early blight (20)	1 939	485
	晚疫病Late blight (21)	1 939	485
	健康Healthy (22)	1 824	456
树莓Raspberry	健康Healthy (23)	1 781	445
黄豆Soybean	健康Healthy (24)	2 022	505
南瓜Squash	白粉病Powdery mildew (25)	1 736	424
草莓Strawberry	叶焦病Leaf scorch (26)	1 774	444
	健康Healthy (27)	1 824	456
番茄Tomato	细菌斑Bacterial spot (28)	1 702	425
	早疫病Early blight (29)	1 920	480
	晚疫病Late blight (30)	1 851	463
	叶霉病Leaf mold (31)	1 882	470
	叶斑病Septoria leaf spot (32)	1 745	436
	二斑叶螨病Two-spotted spider mite (33)	1 741	435
	轮斑病Target spot (34)	1 827	457
	黄曲叶病Yellow leaf curl virus (35)	1 961	490
	花叶病Mosaic virus (36)	1 790	448
	健康Healthy (37)	1 926	481

Fig.1 Partial samples of New Plant Diseases Dataset

Fig.2 Samples of apple leaf diseases pictures

Fig.3 Algorithm flow chart

Table 2 Diagram of model structure

层名称 Layer name	ResNet-(9n+2)模型层输出大小 Layer output size of ResNet-(9n+2) model	ResNet-(9n+2) 模型结构 Structure of ResNet- (9n+2) model	ResNet-(12n+2)模型层输出大小 Layer output size of ResNet-(12n+2) model	ResNet-(12n+2) 模型结构 Structure of ResNet- (12n+2) model
Conv1	32×32	3×3,16	112×112	7×7,64
Layer1	32×32	$1 × 1,16 3 × 3,16 1 × 1,64$ ×n	112×112	$1 × 1,64 3 × 3,64 1 × 1,256$ ×n
Layer2	16×16	$1 × 1,32 3 × 3,32 1 × 1,128$ ×n	56×56	$1 × 1,128 3 × 3,128 1 × 1,512$ ×n
Layer3	8×8	$1 × 1,32 3 × 3,32 1 × 1,128$ ×n	28×28	$1 × 1,256 3 × 3,256 1 × 1,1 ? 024$ ×n
Layer4	—	—	14×14	$1 × 1,512 3 × 3,512 1 × 1,2 ? 048$ ×n

Table 2 Diagram of model structure

层名称 Layer name	ResNet-(9n+2)模型层输出大小 Layer output size of ResNet-(9n+2) model	ResNet-(9n+2) 模型结构 Structure of ResNet- (9n+2) model	ResNet-(12n+2)模型层输出大小 Layer output size of ResNet-(12n+2) model	ResNet-(12n+2) 模型结构 Structure of ResNet- (12n+2) model
Conv1	32×32	3×3,16	112×112	7×7,64
Layer1	32×32	$1 × 1,16 3 × 3,16 1 × 1,64$ ×n	112×112	$1 × 1,64 3 × 3,64 1 × 1,256$ ×n
Layer2	16×16	$1 × 1,32 3 × 3,32 1 × 1,128$ ×n	56×56	$1 × 1,128 3 × 3,128 1 × 1,512$ ×n
Layer3	8×8	$1 × 1,32 3 × 3,32 1 × 1,128$ ×n	28×28	$1 × 1,256 3 × 3,256 1 × 1,1 ? 024$ ×n
Layer4	—	—	14×14	$1 × 1,512 3 × 3,512 1 × 1,2 ? 048$ ×n

Fig.4 Diagram of knowledge distillation C, Convolution layer; P, Pooling layer; FC, Fully connected layer. The same as below.

Fig.5 Diagram of structure of teaching assistant network

Fig.6 Diagram of channel pruning

Fig.7 Training flow chart

Table 3 Experimental results on New Plant Diseases Dataset

模型 Model	准确率 Accuracy/%	参数量 Params	计算量 FLOPs/G
ResNet-50(基础Base)	96.29	25 892 262	16.08
ResNet-50(剪枝90%+学习率倒带Pruning 90%+learning rate rewinding)	95.65	3 386 114	5.39
ResNet-50(剪枝90%+教师学习率倒带Pruning 90%+learning rate rewinding of teacher)	96.52	3 386 114	5.39
ResNet-50(剪枝90%+助教学习率倒带Pruning 90%+learning rate rewinding of teacher assistant)	97.78	3 399 477	4.9
ResNet-26(基础Base)	95.37	14 024 102	9.05
ResNet-26(剪枝70%+学习率倒带Pruning 70%+learning rate rewinding)	91.23	3 256 645	3.93
ResNet-26(剪枝70%+教师学习率倒带Pruning 70%+learning rate rewinding of teacher)	95.66	3 256 645	3.93
ResNet-26(剪枝70%+助教学习率倒带Pruning 90%+learning rate rewinding of teacher assistant)	97.29	3 570 393	4.52
ResNet-14(基础Base)	91.54	8 090 022	5.54
ResNet-14(剪枝50%+学习率倒带Pruning 50%+learning rate rewinding)	90.85	3 123 523	3.16
ResNet-14(剪枝50%+教师学习率倒带Pruning 50%+learning rate rewinding of teacher)	91.32	3 123 523	3.16
ResNet-14(剪枝50%+助教学习率倒带Pruning 50%+learning rate rewinding of teacher assistant)	96.35	3 533 589	3.65
MobileNetV2	90.90	2 272 550	0.32

Fig.8 Photos of the identification effect on the multiple plant disease dataset on the Android platform

Table 4 Experimental results on apple leaf disease dataset

模型 Model	准确率 Accuracy/%	参数 Params	计算量 FLOPs/G
ResNet-26(基础Base)	87.09	13 956 485	9.05
ResNet-26(剪枝70%+学习率倒带Pruning 70%+learning rate rewinding)	89.10	4 653 755	3.80
ResNet-26(剪枝70%+教师学习率倒带Pruning 70%+learning rate rewinding of teacher)	89.32	4 653 755	3.80
ResNet-26(剪枝70%+助教学习率倒带Pruning 90%+learning rate rewinding of teacher assistant)	91.94	4 837 876	3.55
MobileNetV2	33.57	2 230 277	0.32

Fig.9 Photos of the identification effect on the apple leaf data on the Android platform

Fig.10 Dynamic of accuracy under different sparsities

Table 5 Effect of pruning rate on models %

模型 Model	剪枝前准确率 Accuracy before pruning	剪枝后准确率 Accuracy after pruning
ResNet-56	89.51	—
(稀疏化Sparse)
ResNet-56	89.51	89.54
(剪枝30% Pruning 30%)
ResNet-56	89.43	91.41
(剪枝40% Pruning 40%)
ResNet-56	14.09	91.31
(剪枝50% Pruning 50%)
ResNet-56	10.00	90.19
(剪枝60% Pruning 60%)
ResNet-50(稀疏化Sparse)	83.57	—
ResNet-50	83.57	85.99
(剪枝50% Pruning 50%)
ResNet-50	83.58	85.81
(剪枝60% Pruning 60%)
ResNet-50	83.62	85.59
(剪枝70% Pruning 70%)
ResNet-50	55.78	85.37
(剪枝80% Pruning 80%)

Fig.11 Identification accuracy of ravious diseases by the proposed model 0, Apple scab; 1, Apple black rot; 2, Apple cedar rust; 3, Healty apple; 4, Healthy blueberry; 5, Cherry powdery mildew; 6, Healthy cherry; 7, Maize gray leaf spot; 8, Maize common rust; 9, Maize northern leaf blight; 10, Healthy maize; 11, Grape black rot; 12, Grape black measles; 13, Grape leaf blight; 14, Healthy grape; 15, Orange Huanglongbing; 16, Peach bacterial spot; 17, Healthy peach; 18, Pepper bacterial spot; 19, Healthy pepper; 20, Potato early blight; 21, Potato late blight; 22, Healthy potato; 23, Healthy raspberry; 24, Healthy soybean; 25, Squash powdery mildew; 26, Strawberry leaf scorch; 27, Healthy strawberry; 28, Tomato bacterial spot; 29, Tomato early bilght; 30, Tomato late blight; 31, Tomato leaf mold; 32, Tomato septoria leaf spot; 33, Tomato two-spotted spider mite; 34, Tomato target spot; 35, Tomato yellow leaf curl virus; 36, Tomato mosaic virus; 37, Healthy tomato. The same as below.

Fig.12 Quantity distribution of misidentifications of 4 diseases with the highest error rate

References 29

[1]	KAMILARIS A, PRENAFETA-BOLDU’ F X. Deep learning in agriculture: a survey[J]. Computers and Electronics in Agriculture, 2018, 147: 70-90.
[2]	戴建国, 张国顺, 郭鹏, 等. 基于无人机遥感可见光影像的北疆主要农作物分类方法[J]. 农业工程学报, 2018, 34(18): 122-129.
	DAI J G, ZHANG G S, GUO P, et al. Classification method of main crops in northern Xinjiang based on UAV visible waveband images[J]. Transactions of the Chinese Society of Agricultural Engineering, 2018, 34(18): 122-129. (in Chinese with English abstract)
[3]	韩文霆, 郭聪聪, 张立元, 等. 基于无人机遥感的灌区土地利用与覆被分类方法[J]. 农业机械学报, 2016, 47(11): 270-277.
	HAN W T, GUO C C, ZHANG L Y, et al. Classification method of land cover and irrigated farm land use based on UAV remote sensing in irrigation[J]. Transactions of the Chinese Society for Agricultural Machinery, 2016, 47(11): 270-277. (in Chinese with English abstract)
[4]	NI C, WANG D Y, VINSON R, et al. Automatic inspection machine for maize kernels based on deep convolutional neural networks[J]. Biosystems Engineering, 2019, 178: 131-144.
[5]	陈进, 顾琰, 练毅, 等. 基于机器视觉的水稻杂质及破碎籽粒在线识别方法[J]. 农业工程学报, 2018, 34(13): 187-194.
	CHEN J, GU Y, LIAN Y, et al. Online recognition method of impurities and broken paddy grains based on machine vision[J]. Transactions of the Chinese Society of Agricultural Engineering, 2018, 34(13): 187-194. (in Chinese with English abstract)
[6]	王丹丹, 何东健. 基于R-FCN深度卷积神经网络的机器人疏果前苹果目标的识别[J]. 农业工程学报, 2019, 35(3): 156-163.
	WANG D D, HE D J. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network[J]. Transactions of the Chinese Society of Agricultural Engineering, 2019, 35(3): 156-163. (in Chinese with English abstract)
[7]	OLANIYI E O, OYEDOTUN O K, ADNAN K. Intelligent grading system for banana fruit using neural network arbitration[J]. Journal of Food Process Engineering, 2017, 40(1): e12335.
[8]	MA J C, DU K M, ZHENG F X, et al. A recognition method for cucumber diseases using leaf symptom images based on deep convolutional neural network[J]. Computers and Electronics in Agriculture, 2018, 154: 18-24.
[9]	谢忠红, 徐焕良, 黄秋桂, 等. 基于高光谱图像和深度学习的菠菜新鲜度检测[J]. 农业工程学报, 2019, 35(13): 277-284.
	XIE Z H, XU H L, HUANG Q G, et al. Spinach freshness detection based on hyperspectral image and deep learning method[J]. Transactions of the Chinese Society of Agricultural Engineering, 2019, 35(13): 277-284. (in Chinese with English abstract)
[10]	张建华, 孔繁涛, 吴建寨, 等. 基于改进VGG卷积神经网络的棉花病害识别模型[J]. 中国农业大学学报, 2018, 23(11): 161-171.
	ZHANG J H, KONG F T, WU J Z, et al. Cotton disease identification model based on improved VGG convolution neural network[J]. Journal of China Agricultural University, 2018, 23(11): 161-171. (in Chinese with English abstract)
[11]	杨森, 冯全, 张建华, 等. 基于深度学习与复合字典的马铃薯病害识别方法[J]. 农业机械学报, 2020, 51(7): 22-29.
	YANG S, FENG Q, ZHANG J H, et al. Identification method for potato disease based on deep learning and composite dictionary[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(7): 22-29. (in Chinese with English abstract)
[12]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016. Las Vegas, NV, USA: IEEE, 2016: 770-778.
[13]	刘翱宇, 吴云志, 朱小宁, 等. 基于深度残差网络的玉米病害识别[J]. 江苏农业学报, 2021, 37(1): 67-74.
	LIU A Y, WU Y Z, ZHU X N, et al. Corn disease recognition based on deep residual network[J]. Jiangsu Journal of Agricultural Sciences, 2021, 37(1): 67-74. (in Chinese with English abstract)
[14]	孔建磊, 金学波, 陶治, 等. 基于多流高斯概率融合网络的病虫害细粒度识别[J]. 农业工程学报, 2020, 36(13): 148-157.
	KONG J L, JIN X B, TAO Z, et al. Fine-grained recognition of diseases and pests based on multi-stream Gaussian probability fusion network[J]. Transactions of the Chinese Society of Agricultural Engineering, 2020, 36(13): 148-157. (in Chinese with English abstract)
[15]	YE J B, LU X, LIN Z, et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers[EB/OL].[2022-08-14]. https://arxiv.org/abs/1802.00124.
[16]	LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient ConvNets[EB/OL]. (2016-08-31) [2022-08-14]. https://arxiv.org/abs/1608.08710.
[17]	CHANGPINYO S, SANDLER M, ZHMOGINOV A. The power of sparsity in convolutional neural networks[EB/OL]. (2017-02-21) [2022-08-14]. https://arxiv.org/abs/1702.06257.
[18]	WEN W, WU C P, WANG Y D, et al. Learning structured sparsity in deep neural networks[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. December 5-10, 2016, Barcelona: ACM, 2016: 2082-2090.
[19]	LIU Z, LI J G, SHEN Z Q, et al. Learning efficient convolutional networks through network slimming[C]// 2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice: IEEE, 2017: 2755-2763.
[20]	BUCILUǍ C, CARUANA R, NICULESCU-MIZIL A. Model compression[C]// Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. August 20-23, 2006, Philadelphia, PA, USA: ACM, 2006: 535-541.
[21]	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. (2015-03-09) [2022-08-14]. https://arxiv.org/abs/1503.02531.
[22]	BA L J, CARUANA R. Do deep nets really need to be deep? [EB/OL]. (2013-12-21)[2022-08-14]. https://arxiv.org/abs/1312.6184.
[23]	ROMERO A, BALLAS N, KAHOU S E, et al. FitNets: hints for thin deep nets[EB/OL]. (2014-12-19) [2022-08-14]. https://arxiv.org/abs/1412.6550.
[24]	CZARNECKI W M, OSINDERO S, JADERBERG M, et al. So-bolev training for neural networks[C]// VON LUXBURGU, GUYOUI, BENGIOS, et al. Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2017: 4281-4290.
[25]	HE T, SHEN C H, TIAN Z, et al. Knowledge adaptation for efficient semantic segmentation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA: IEEE, 2020: 578-587.
[26]	MIRZADEH S I, FARAJTABAR M, LI A, et al. Improved knowledge distillation via teacher assistant[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 5191-5198.
[27]	RENDA A, FRANKLE J, CARBIN M. Comparing rewinding and fine-tuning in neural network pruning[EB/OL]. (2020-03-05) [2022-08-14]. https://arxiv.org/abs/2003.02389.
[28]	IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[EB/OL]. (2015-02-11) [2022-08-14]. https://arxiv.org/abs/1502.03167.
[29]	FRANKLE J, DZIUGAITE G K, ROY D M, et al. Linear mode connectivity and the lottery ticket hypothesis[C]// Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020: 3259-3269.

A light-weight model for plant disease identification based on model pruning and knowledge distillation

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 29

Related Articles 3

Recommended Articles

Metrics

Comments

[1]	ZHU Dongqin, FENG Quan, ZHANG Jianhua. Plant disease identification based on pruning [J]. Acta Agriculturae Zhejiangensis, 2023, 35(6): 1462-1472.
[2]	YANG Guoliang, XU Nan, KANG Lele, GONG Man, HONG Zhiyang. Identification of navel orange lesions leaves based on parametric exponential non-linear residual neural network [J]. , 2018, 30(6): 1073-1081.
[3]	ZHENG Ji-rong;WANG Hui-li. Resistance identification and genetic analysis of Tomato yellow leaf curl virus(TYLCV) [J]. , 2010, 22(4): 0-447.