Advanced Streaming Data Cleansing

Alisha DESHPANDE, Yining DONG, Gang LI, Yingying ZHENG, Si-Zhao (Joe) QIN, Lisa A. BRENSKELLE

Research output: Book Chapters | Papers in Conference ProceedingsConference paper (refereed)Referred Conference Paperpeer-review

4 Citations (Scopus)


A frequent problem experienced throughout industry is that of missing or poor quality data in data historians. While this can have many causes, the end result is that data required to perform analyses needed to improve facility operations may be unavailable. This generally occasions delays and wastes valuable time, as the data analyst must manually "clean up" the data before using it, or could even result in erroneous conclusions if the data is used as is without any corrections.

This work has uncovered a dynamic principal component analysis model-based method to detect the presence of erroneous data, identify which sensor is at fault, and reconstruct corrected values for that sensor, to be stored in the historian. However, the dynamic principal component analysis model-based method is not appropriate for all sensors, so a second method for detecting errors in data from a single sensor and calculating corrected values has also been developed. Both methods work on streaming data, and thus make corrections continuously in near real-time. The dynamic principal component analysis model-based method has been successfully tested in the field by injecting errors such as a missing, bias, spike, drift, frozen, etc.. into real streaming operating data from a Chevron facility. The single sensor data cleansing methodology has not yet undergone field test, but has been tested offline using operating data into which errors such as drift, spike, frozen, missing... have been introduced. Use of these methods can ensure that good quality data for needed analyses is available in the data historian, thereby saving analyst time and assuring that erroneous conclusions are not reached by using faulty data.
Original languageEnglish
Title of host publicationSociety of Petroleum Engineers - SPE Digital Energy Conference and Exhibition 2015
PublisherSociety of Petroleum Engineers (SPE)
Number of pages9
ISBN (Print)9781510800595
Publication statusPublished - Mar 2015
Externally publishedYes
EventSPE Digital Energy Conference and Exhibition 2015 - The Woodlands, United States
Duration: 3 Mar 20155 Mar 2015


ConferenceSPE Digital Energy Conference and Exhibition 2015
Country/TerritoryUnited States
CityThe Woodlands


Dive into the research topics of 'Advanced Streaming Data Cleansing'. Together they form a unique fingerprint.

Cite this