Abstract
We offer theoretical and empirical insights into the impact of exogenous randomness on the effectiveness of random forests with tree-building rules independent of training data. We formally introduce the concept of exogenous randomness which can come from feature subsampling or tie-breaking in tree-building processes. We develop non-asymptotic expansions for the mean squared error (MSE) for both individual trees and forests and establish sufficient and necessary conditions for their consistency. In the special example of the linear regression model with independent features, our MSE expansions are more explicit, providing more understanding of the random forests' mechanisms. It also allows us to derive an upper bound on the MSE with explicit consistency rates for trees and forests. Guided by our theoretical findings, we conduct simulations to further explore how exogenous randomness enhances random forests performance. Our findings unveil that feature subsampling reduces both the bias and variance of random forests compared to individual trees, serving as an adaptive mechanism to balance bias and variance. Furthermore, our results reveal an intriguing phenomenon: the presence of noise features can act as a “blessing" in enhancing the performance of random forests thanks to feature subsampling.
| Original language | English |
|---|---|
| Article number | 98 |
| Pages (from-to) | 1-95 |
| Number of pages | 95 |
| Journal | Journal of Machine Learning Research |
| Volume | 27 |
| Issue number | 98 |
| Publication status | Published - May 2026 |
Bibliographical note
Publisher Copyright:© 2026 Tianxing Mei, Yingying Fan, and Jinchi Lv.
Funding
This work was supported in part by NSF grants EF-2125142, DMS-2310981, and DMS-2324490.
Keywords
- random forests
- partitioning rule
- feature subsampling
- ensemble
- exogenous randomness
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver