Study on cleaning and quality control technology of agricultural micro-meteorological observation data

doi:10.14088/j.cnki.issn0439-8114.2020.14.006

Abstract

Abstract: In order to improve the quality of agrometeorological observation data, the cleaning standard and quality control method of agrometeorological data are established based on the unique meteorological observation equipment and environmental attributes of farmland. For the case of abnormal data attributes and repeated records, the method of cleaning the blank value of Bohn data cleaning model and the method of cleaning the noise data are selected. Through the historical data statistics of observation elements in the space of agricultural micro meteorological observation station, the clean data index is obtained and applied to the dynamic threshold generation method of data quality, and the quality control model of agricultural micro meteorological data is established. The data evaluation indexes after cleaning and quality control showed that the accuracy and repeatability of the data are significantly improved after data cleaning and quality control model. The data cleaning quality control method is helpful to obtain the monitoring information of agrometeorological disaster accurately and provide effective decision support for agricultural disaster prevention and reduction.

Key words: agromicro meteorology, data quality control, Bohn data cleaning model

CLC Number:

P49
TP274

ZHOU Qiang. Study on cleaning and quality control technology of agricultural micro-meteorological observation data[J]. HUBEI AGRICULTURAL SCIENCES, 2020, 59(14): 37-40.

References

[1] 于力超,金勇进,王俊. 缺失数据插补方法探讨——基于最近邻插补法和关联规则法[J]. 统计与信息论坛,2015,30(1):35-40.
[2] 戴明锋,金勇进,查奇芬,等. 二分类Logistic回归插补法及其应用[J]. 数学的实践与认识,2013, 43(21):162-167.
[3] 刘燕. 基于Logistic回归的近邻择优插补法[D]. 天津:天津财经大学,2013.
[4] 俞荣华,田增平,周傲英. 一种检测多语言文本相似重复记录的综合方法[J]. 计算机科学,2002, 29(1):118-121.
[5] 赵一凡,卞良,丛昕. 数据清洗方法研究综述[J]. 软件导刊,2017,16(12):222-224.
[6] OTHMAN L B,YAHIA S B.GBARMVC: Generic basis of association rules based approach for missing values completion[J]. International journal of computing & information sciences,2011,9(1):16-22.
[7] 肖心园,江冰,任其文,等. 基于插值法和皮尔逊相关的光伏数据清洗[J]. 信息技术,2019(5):19-22,28.
[8] 潘腾辉,林金城,郑细烨,等. 面向数据库清洗的数据质量控制设计[J]. 信息技术,2017(10):133-136.
[9] 李昌华,卜亮亮,刘欣. 基于聚类和神经网络对建筑节能气候数据清洗的算法[J]. 计算机应用,2018,38(S1):83-86,111.
[10] 窦以文,屈玉贵,陶士伟,等. 北京自动气象站实时数据质量控制应用[J]. 气象,2008,34(8):77-81.
[11] SHAFER M A,FIEBRICH C A,ARNDT D S,et al.Quality assurance procedures in the oklahoma mesonetwork[J]. Journal of atmospheric & oceanic technology,2000,17(4):474-494.
[12] 陈奕隆. 美国自动地面观测系统[J]. 气象科技,1994(3):48-54.
[13] 廖捷,周自江. 全球常规气象观测资料质量控制研究进展与展望[J]. 气象科技进展,2018,8(1):56-62.
[14] 任芝花,张志富,孙超,等. 全国自动气象站实时观测资料三级质量控制系统研制[J]. 气象,2015, 41(10):1268-1277.
[15] 韩海涛,李仲龙. 地面实时气象数据质量控制方法研究进展[J]. 干旱气象,2012,30(2):261-265.
[16] JEFFERY S R,ALONSO G,FRANKLIN M J,et al.Declarative support for sensor data cleaning[A]. Proceedings of 4th international conference on pervasive computing[C]. Springer, New York,2006.83-100.
[17] GILL S,LEE B.A framework for distributed cleaning of data streams[J]. Procedia computer science,2015,52(1):1186-1191.
[18] 李良富,王汉杰,刘金玉,等. 基于黑板模型的地面气象数据质量控制[J]. 气象科技,2006,34(2):199-204.
[19] 范文波. 地面气象观测数据综合质量控制方法研究与实现[D]. 南京:南京信息工程大学,2016.