Advanced Streaming Data Cleansing

Yingying ZHENG, S. Joe QIN, Lisa A. BRENSKELLE

Research output: Book Chapters | Papers in Conference ProceedingsConference paper (refereed)Referred Conference Paperpeer-review


A frequent problem experienced throughout industry is that of missing or poor quality data in data historians. This can have various causes, such as field instrument failures, loss of communication, or even issues with the setup of the historian itself. The end result is that data required to perform analyses needed to improve facility operations may be unavailable. This generally incurs delays, as the data analyst must manually "clean up" the data before using it, or could even result in erroneous conclusions if the data is used as is without any corrections. In this paper, a novel multivariate statistical method is proposed to detect incorrect data values and reconstruct corrected values to be stored in the historian. This method works on streaming data, and thus makes its corrections continuously in near real-time. The method has been successfully tested in a laboratory setting using real operating data from a Chevron facility. Chevron plans to test the data error detection and reconstruction method in the field in the near future. Use of this method will ensure that good quality data for needed analyses is available in the data historian, and will save analyst time as well. Copyright 2013, Society of Petroleum Engineers.
Original languageEnglish
Title of host publicationSociety of Petroleum Engineers - SPE Digital Energy Conference and Exhibition 2013
PublisherSociety of Petroleum Engineers (SPE)
Number of pages9
ISBN (Print)9781627480253
Publication statusPublished - Mar 2013
Externally publishedYes
EventSPE Digital Energy Conference and Exhibition 2013 - The Woodlands, United States
Duration: 5 Mar 20137 Mar 2013


ConferenceSPE Digital Energy Conference and Exhibition 2013
Country/TerritoryUnited States
CityThe Woodlands


Dive into the research topics of 'Advanced Streaming Data Cleansing'. Together they form a unique fingerprint.

Cite this