Named entity recognition in agriculture based on BERT embedding and adversarial training

doi:10.14088/j.cnki.issn0439-8114.2022.18.035

Abstract

Abstract: Named entity recognition task for the agricultural domain is a key step in information extraction and question and answer system in the agricultural domain. The goal of this step is to find out the required named entities from a huge amount of unstructured agricultural texts, which usually have challenges such as diverse entity names and contextual semantics missing. To accomplish the task of recognizing agricultural named entities in complex contexts, this paper first constructs an annotated corpus in the agricultural domain, which contains 16 048 samples of six types of entities; then uses the BERT pre-trained language model as the word embedding layer, which can well solve the problem of different semantics and referents of the same word in different contexts compared with the traditional word embedding model; then uses the BiGRU network model for context encoding; finally, the output sequence is annotated using CRF. At the same time, this paper introduces a certain amount of noise to the input data, which is used for adversarial training to improve the generalization and robustness of the model. After the experiments, the accuracy, recall, and F-value of the proposed model are 92.75%, 91.53%, and 92.49%, respectively. Compared with the baseline model, this method has better performance and can effectively identify named entities in the agricultural field.

Key words: agriculture, natural language processing, named entity recognition, information extraction, BERT, BiGRU, adversarial training

CLC Number:

TP391.1

FEI Fan, YANG Lin-nan. Named entity recognition in agriculture based on BERT embedding and adversarial training[J]. HUBEI AGRICULTURAL SCIENCES, 2022, 61(18): 196-202.

References

[1] CHINCHOR N, ROBINSON P.MUC-7 named entity task definition[C].Proceedings of the 7th conference on message understanding,1997.1-21.
[2] DRURY B, ROCHE M.A survey of the applications of text mining for agriculture[J]. Computers and electronics in agriculture,2019, 163: 104864.
[3] LIU W, YU B, ZHANG C, et al.Chinese named entity recognition based on rules and conditional random field[A]. Proceedings of the 2018 2nd international conference on computer science and artificial intelligence[C].United States,2018.268-272.
[4] WANG X, JIANG X, LIU M, et al.Bacterial named entity recognition based on dictionary and conditional random field[A]. IEEE international conference on bioinformatics and biomedicine (BIBM)[C]. 2017.439-444.
[5] WALLACH H M.Conditional random fields: An introduction[R]. University of Pennsylvania CIS Technical Report, 2004.22.
[6] LI X, WEI X H, JIA L, et al.Recognition of crops, diseases and pesticides named entities in Chinese based in conditional random fields[J]. Trans Chin Soc Agric Mach, 2017, 48: 178-185.
[7] MALARKODI C S,LEX E, DEVI S L.Named entity recognition for the agricultural domain[J]. Res Comput Sci,2016,117:121-132.
[8] ZHOU G D, SU J.Named entity recognition using an HMM-based chunk tagger[A].Proceedings of the 40th annual meeting of the association for computational linguistics[C].2002.473-480.
[9] LECUN Y,BENGIO Y,HINTON G.Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[10] HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J]. Neural computation, 1997, 9(8): 1735-1780.
[11] HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[J/OL].arXiv preprint,https://arxiv.org/abs/1508.01991,2015-08-09.
[12] ESPEJO-GARCIA B,LOPEZ-PELLICER F J,LACASTA J, et al. End-to-end sequence labeling via deep learning for automatic extraction of agricultural regulations[J]. Computers and electronics in agriculture, 2019, 162: 106-111.
[13] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C].Advances in neural information processing systems. 2017. 5998-6008.
[14] DEVLIN J,CHANG M W,LEE K,et al.Bert: Pre-training of deep bidirectional transformers for language understanding[DB/OL]. https://aclanthology.org/N19-1423, 2019.
[15] GOODFELLOW I J,SHLENS J,SZEGEDY C. Explaining and harnessing adversarial examples[J/OL]. arXiv preprint,https://doi.org/10.48550/arXiv.1412.6572, 2014.
[16] MIYATO T,DAI A M,GOODFELLOW I. Adversarial training methods for semi-supervised text classification[J/OL]. arXiv preprint,https://doi.org/10.48550/arXiv.1605.07725,2016.
[17] CHEN X, CARDIE C. Multinomial adversarial networks for multi-domain text classification[J/OL]. arXiv preprint,https://doi.org/10.48550/arXiv.1802.05694,2018.
[18] ZHOU J T,ZHANG H,JIN D, et al.Dual adversarial neural transfer for low-resource named entity recognition[A].Proceedings of the 57th annual meeting of the association for computational linguistics[C]. 2019.3461-3471.
[19] 赵鹏飞,赵春江,吴华瑞,等. 基于注意力机制的农业文本命名实体识别[J].农业机械学报,2021,52(1):185-192.
[20] 李林,周晗,郭旭超,等. 基于多源信息融合的病虫害命名实体识别研究[J/OL].农业机械学报,http://kns.cnki.net/kcms/detail/11.1964.S.20210915.0933.006.html,2021-11-20.
[21] CHUNG J, GULCEHRE C, CHO K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J/OL]. arXiv preprint,https://doi.org/10.48550/arXiv.1412.3555, 2014.
[22] MIYATO T,DAI A M,GOODFELLOW L.Adversarial training methods for semi-supervised text classification[J/OL]. arXiv preprint,https://doi.org/10.48550/arXiv.1605.07725, 2017.
[23] DONG C, ZHANG J, ZONG C, et al.Character-based LSTM-CRF with radical-level features for Chinese named entity recognition[A].Natural language processing and Chinese computing[C]. Kunming,2016,2016.239-250.