Random sample partition (RSP) is a newly developed big data representation and management model to deal with big data approximate computation problems. Academic research and practical applications have confirmed that RSP is an efficient solution for big data processing and analysis. However, a challenge for implementing RSP is determining an appropriate sample size for RSP data blocks. While a large sample size increases the burden of big data computation, a small size will lead to insufficient distribution information for RSP data blocks. To address this problem, this paper presents a novel density estimation-based method (DEM) to determine the optimal sample size for RSP data blocks. First, a theoretical sample size is calculated based on the multivariate Dvoretzky-Kiefer-Wolfowitz (DKW) inequality by using the fixed-point iteration (FPI) method. Second, a practical sample size is determined by minimizing the validation error of a kernel density estimator (KDE) constructed on RSP data blocks for an increasing sample size. Finally, a series of persuasive experiments are conducted to validate the feasibility, rationality, and effectiveness of DEM. Experimental results show that (1) the iteration function of the FPI method is convergent for calculating the theoretical sample size from the multivariate DKW inequality; (2) the KDE constructed on RSP data blocks with sample size determined by DEM can yield a good approximation of the probability density function (p.d.f.); and (3) DEM provides more accurate sample sizes than the existing sample size determination methods from the perspective of p.d.f. estimation. This demonstrates that DEM is a viable approach to deal with the sample size determination problem for big data RSP implementation.
|Number of pages||14|
|Journal||Frontiers of Computer Science|
|Publication status||Accepted/In press - May 2023|
Bibliographical noteThe authors would like to sincerely thank the editors and three anonymous reviewers whose valuable suggestions considerably helped improve the paper after two rounds of review. This paper was supported by the National Natural Science Foundation of China (Grant No. 61972261), the Natural Science Foundation of Guangdong Province (No. 2023A1515011667), the Key Basic Research Foundation of Shenzhen (No. JCYJ20220818100205012), and the Basic Research Foundation of Shenzhen (No. JCYJ20210324093609026).
- random sample partition
- big data
- sample size
- Dvoretzky-Kiefer-Wolfowitz inequality
- kernel density estimator
- probability density function