Density estimation-based method to determine sample size for random sample partition of big data

Yulin HE, Jiaqi CHEN, Jiaxing SHEN, Philippe FOURNIER-VIGER, Joshua Zhexue HUANG*

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

Abstract

Random sample partition (RSP) is a newly developed big data representation and management model to deal with big data approximate computation problems. Academic research and practical applications have confirmed that RSP is an efficient solution for big data processing and analysis. However, a challenge for implementing RSP is determining an appropriate sample size for RSP data blocks. While a large sample size increases the burden of big data computation, a small size will lead to insufficient distribution information for RSP data blocks. To address this problem, this paper presents a novel density estimation-based method (DEM) to determine the optimal sample size for RSP data blocks. First, a theoretical sample size is calculated based on the multivariate Dvoretzky-Kiefer-Wolfowitz (DKW) inequality by using the fixed-point iteration (FPI) method. Second, a practical sample size is determined by minimizing the validation error of a kernel density estimator (KDE) constructed on RSP data blocks for an increasing sample size. Finally, a series of persuasive experiments are conducted to validate the feasibility, rationality, and effectiveness of DEM. Experimental results show that (1) the iteration function of the FPI method is convergent for calculating the theoretical sample size from the multivariate DKW inequality; (2) the KDE constructed on RSP data blocks with sample size determined by DEM can yield a good approximation of the probability density function (p.d.f.); and (3) DEM provides more accurate sample sizes than the existing sample size determination methods from the perspective of p.d.f. estimation. This demonstrates that DEM is a viable approach to deal with the sample size determination problem for big data RSP implementation.
Original languageEnglish
Article number185322
Number of pages14
JournalFrontiers of Computer Science
Volume18
Issue number5
Early online date16 Dec 2023
DOIs
Publication statusE-pub ahead of print - 16 Dec 2023

Bibliographical note

Publisher Copyright:
© 2024, Higher Education Press.

Keywords

  • random sample partition
  • big data
  • sample size
  • Dvoretzky-Kiefer-Wolfowitz inequality
  • kernel density estimator
  • probability density function

Fingerprint

Dive into the research topics of 'Density estimation-based method to determine sample size for random sample partition of big data'. Together they form a unique fingerprint.

Cite this