Random Forest

This page focuses on Random Forest classification. Regression forests exist as a related extension. They are not derived here.

A Random Forest is a supervised ensemble of tree-structured classifiers generated with randomness. Breiman defines a random forest as a collection $\{h(x,{\Theta }_k),k=1,\dots \}$ of tree classifiers, where the ${\Theta }_k$ are independent identically distributed random vectors that govern the random elements of each tree, and each tree casts a unit vote for the most popular class at $x$. Randomization of the training procedure and aggregation of the trees are part of that definition. The method is not merely a collection of ordinary decision trees.

Training uses labeled observations. The fitted forest assigns a new observation among the learned class labels under an explicit aggregation rule. That assignment is discriminant classification. It is not SIMCA-style class modelling with optional rejection.

A classification tree recursively partitions predictor space by split rules. Each split sends an observation down one of two branches according to a threshold on one predictor. Terminal nodes produce class predictions according to the fitted tree criterion, commonly the majority class or the class proportions in the leaf.

One tree is a single hierarchical partition fitted to its training data. A Random Forest grows many such trees under randomized training conditions and aggregates their outputs. The ensemble is the classifier. This page does not derive CART split search, pruning, or stopping rules in full. Those details belong to a decision-tree treatment. Software growth constraints such as maximum depth are implementation options, not a second methodological definition of Random Forest.

Under the classical Breiman construction, tree $b$ is grown on a bootstrap training set $D_b$ drawn from the original training set $D$. Sampling is with replacement. The bootstrap replica is the in-bag set for that tree. Observations omitted from that replica are out-of-bag for that tree.

That is the methodological core. It is not a claim that every software forest must draw exactly $n$ observations with replacement. Current scikit-learn RandomForestClassifier defaults to bootstrap=True. If bootstrap=False, the whole training set is used to build each tree while random predictor selection can still operate. That option is not Breiman's original bootstrap forest. max_samples can further change the number or fraction of rows drawn when bootstrap sampling is on. MATLAB TreeBagger defaults to sampling with replacement and InBagFraction equal to 1, so each replica has $n$ draws from $n$ observations unless those options are changed.

At each eligible split, Random Forest considers a randomly selected subset of predictors rather than necessarily searching all $p$ variables. Let $m_{\mathrm{try}}\le p$ denote the number of candidate predictors considered at a split. The split is then chosen among those candidates by the selected impurity or loss criterion.

Breiman's Forest-RI procedure selects a small group of inputs at random at each node, grows the tree with CART methodology to maximum size, and does not prune. Restricting the split search is a source of randomization among trees. It is intended to reduce similarity among the individual classifiers. The fitted trees are not generally independent predictors.

Current scikit-learn classification default max_features="sqrt" uses $\sqrt{n_{\mathrm{features}}}$. Current MATLAB classification TreeBagger default NumPredictorsToSample is the square root of the number of predictors, rounded toward positive infinity if that value is not an integer. Those are software defaults and common heuristics. They are not a universal Random Forest rule, and they are not claimed to be optimal for every chemometric problem.

Write the fitted trees as $h_1(x),\dots ,h_B(x)$. In Breiman's formulation each tree predicts a class and casts one vote for that class. The predicted label is the most popular class among those votes.

Current scikit-learn does not use that hard unit-vote rule for predict. The 1.6 user guide states that, in contrast to the original publication, the implementation combines classifiers by averaging their probabilistic predictions. The predicted class is the one with the highest mean probability estimate across the trees. A single tree's class probability is the fraction of training samples of that class in the leaf. Forest predict_proba returns those averaged leaf fractions. They are not calibrated posterior probabilities.

Current MATLAB TreeBagger classification prediction is documented in the Algorithms section as the class that maximizes the weighted average of the trees' class posterior estimates, where a tree posterior is the class fraction in the leaf. Score columns are those averaged leaf fractions. Classification predict returns class labels as a cell array of character vectors.

Multiclass outcomes are handled directly by the tree and forest aggregation. Common implementations are not one-versus-one or one-versus-rest decompositions unless an implementation explicitly uses such a wrapper.

Random Forest is an algorithmic ensemble. It does not have a single closed-form parametric classifier analogous to LDA. The equations below record the training ensemble, the Breiman vote rule, the software mean-probability rule, and Gini impurity when that split criterion is used.

The procedure below is the classical bootstrap forest with random predictor selection at splits. Tree-growth details such as pruning thresholds, minimum impurity decrease, and tie rules are omitted unless they belong to a named software implementation. Prediction aggregation must be stated: Breiman unit votes, or mean leaf class fractions as in current sklearn and MATLAB TreeBagger.

For a given tree, observations not included in that tree's bootstrap sample are out-of-bag for that tree. Each training observation can receive predictions from the trees for which it was omitted. Aggregating those tree-specific predictions yields an out-of-bag prediction for the observation. The out-of-bag error rate is the error of that aggregated OOB classifier on the training observations. Breiman notes that about one third of the instances are left out of each bootstrap training set. That is an approximate expected fraction, not an exact finite-sample rule that 36.8% of rows are always out-of-bag.

Out-of-bag evaluation is not the same procedure as cross-validation. Cross-validation uses explicitly defined folds. OOB uses bootstrap omissions. The two can give similar numerical summaries in some problems. They are not equivalent by construction. OOB does not automatically replace an independent external assessment when that is the scientific claim being made.

Ordinary bootstrap sampling operates at the observation level unless a grouped design is used. If several spectra come from the same physical sample, batch, or subject, OOB status at the spectrum level does not automatically ensure independence at the physical-sample level. Grouped chemometric validation still belongs to the split design. See Cross-Validation.

If OOB error is used while choosing the number of trees or other settings, that use is part of model development. A final performance claim still requires an assessment matched to the intended generalization problem. scikit-learn exposes oob_score_ and oob_decision_function_ only when bootstrap=True and oob_score=True. MATLAB stores OOB indices when OOBPrediction is on and provides oobError and oobPredict.

Random Forest supports more than one variable-importance definition. They are not interchangeable. None of them is causal importance, chemical importance, or proof that a wavelength corresponds to a discriminating bond.

scikit-learn feature_importances_ is impurity-based importance: the normalized total reduction of the splitting criterion attributed to each variable across trees. When the criterion is Gini, that quantity is commonly called Gini importance. If another criterion is used, mean decrease in impurity is the safer general name. The current sklearn documentation warns that this training-set statistic can be misleading for high-cardinality features. It is not a held-out importance measure.

MATLAB OOBPredictorImportance is an out-of-bag permutation importance: the increase in prediction error when a predictor is permuted across OOB observations. That is not the same metric as sklearn feature_importances_. Permutation importance in sklearn is available separately through sklearn.inspection.permutation_importance. If used, evaluate it on held-out or validation data. Computing it on the training set does not provide independent interpretive evidence. Permutation importance is not unbiased and is not causal.

Adjacent spectral wavelengths are often strongly correlated. Menze and coworkers observed that some spectral regions are selected as a whole, so neighboring channels can receive similar importance, and that this importance spectrum can overestimate major peaks that span many channels. They also note that factorial cardinality bias of Gini importance applies to variables with unequal numbers of distinct categories. Continuous spectral channels that each take $N$ distinct values in a given data set are not that factorial case. The sklearn high-cardinality warning and the spectral-correlation warning are therefore different issues. High impurity importance is still a model statistic. Chemical assignment requires independent spectroscopic or domain evidence.

The number of trees $B$ is n_estimators in scikit-learn. Current sklearn 1.6 default is 100. That default changed from 10 in version 0.22. It is an implementation default, not a scientifically sufficient forest size. Increasing $B$ changes the Monte Carlo average of the ensemble. Breiman proves that the generalization error converges almost surely to a limit as the number of trees increases, and writes that this explains why random forests do not overfit as more trees are added but produce a limiting generalization error. That statement is about adding trees in the forest combination. It is not the claim that Random Forest cannot overfit from other hyperparameters, from data leakage, or from an invalid validation design. It is also not the claim that more trees always improve accuracy on a finite test set.

max_features controls $m_{\mathrm{try}}$. max_depth and min_samples_leaf constrain tree growth in modern software. Original Forest-RI grew unpruned maximum-size trees. Current sklearn defaults max_depth=None and min_samples_leaf=1. MATLAB classification MinLeafSize defaults to 1. Those settings can still produce large trees. Shallow or deep forests are not universally better. class_weight is an optional sklearn imbalance control. This page does not prescribe "balanced" or "balanced_subsample".

Random Forest contains stochastic elements. Reproducible educational runs require a fixed random_state and a stated software environment. Two different seeds need not produce identical forests. Hyperparameter selection belongs inside training folds or another valid model-development design. The final test partition remains unused until evaluation with Classification Metrics.

Tree splits are not Euclidean nearest-neighbor rules. Random Forest therefore does not require feature scaling for the same geometric reason as k-NN or an RBF SVM. That is not the statement that Random Forest is invariant to all preprocessing. Spectroscopic choices such as SNV, MSC, derivatives, or baseline correction change the data representation. They remain model-development decisions. This page does not place StandardScaler automatically before RandomForestClassifier.

Recursive splits can represent nonlinear decision rules and interactions. That is a property of trees. It is not the claim that Random Forest discovers all interactions automatically. Trees do not invert a covariance matrix in the LDA or OLS sense. Correlated predictors can still share predictive information, affect importance scores, and produce interchangeable split choices. Random Forest does not solve multicollinearity.

Menze, Kelm, Masuch, Himmelreich, Bachert, Petrich, and Hamprecht compared Random Forest and Gini importance with chemometric classifiers on spectral data. In their study, Gini importance was useful for ranking spectral features, while a regularized classifier on a selected subset could outperform the forest applied to the full spectrum. Orthogonal axis-aligned splits were discussed as a possible mismatch to strongly correlated spectral channels. Those results are study-specific. They do not establish that Random Forest always needs feature selection, or that it never does.

Scott and coworkers evaluated chemometric classifiers, including Random Forest, under external validation on high-dimensional NMR and mass-spectral data with 914 to 1898 features in that benchmark. They reported that cross-validation can be optimistic relative to external validation on data of different provenance, and that merits of Random Forest emerged in that external comparison. That is a benchmark-specific finding. It is not the claim that Random Forest is the best chemometric classifier, that it is immune to $p\gg n$, or that it automatically selects all useful variables.

Relative to a single decision tree, Random Forest aggregates randomized trees rather than using one fitted partition. Relative to k-NN, it does not classify by a global Euclidean neighborhood. Relative to SVM, it does not solve a margin quadratic program. Relative to LDA, it does not use a shared-covariance Gaussian discriminant. Relative to PLS-DA, it is not a latent-variable classifier. Relative to SIMCA, it assigns among learned labels and does not provide open-set class rejection by default.

The first Python listing implements Gini impurity, Breiman vote counts $V_c(x)$, majority-vote prediction, and the mean-probability aggregation used by current sklearn. It does not grow trees. Production classification uses sklearn.ensemble.RandomForestClassifier. Numeric n_estimators values in the sklearn listing are illustrative. predict_proba returns averaged leaf class fractions. feature_importances_ is impurity-based importance, not chemical importance. Grid search is fitted on the training partition only.

MATLAB examples use official TreeBagger with Method set to classification. There is no invented randomForest function. NumPredictorsToSample is the MATLAB analog of $m_{\mathrm{try}}$. Default TreeBagger is a bagged-tree ensemble that can use the random forest predictor-sampling algorithm when that argument is not "all". It is not claimed to be numerically identical to sklearn RandomForestClassifier. Forests can differ by bootstrap implementation, split search, random number generation, tie handling, stopping criteria, and defaults.

import numpy as np  def gini_impurity(class_counts):    counts = np.asarray(class_counts, dtype=float).reshape(-1)    if np.any(counts < 0):        raise ValueError("class_counts must be non-negative.")    n = float(counts.sum())    if n <= 0:        raise ValueError("class_counts must sum to a positive total.")    p = counts / n    return float(np.sum(p * (1.0 - p)))  def class_vote_counts(tree_class_predictions, classes):    preds = np.asarray(tree_class_predictions)    classes = np.asarray(classes)    if preds.ndim != 2:        raise ValueError("tree_class_predictions must be trees by observations.")    n_query = preds.shape[1]    votes = np.zeros((n_query, classes.shape[0]), dtype=int)    for j, label in enumerate(classes):        votes[:, j] = np.sum(preds == label, axis=0)    return votes  def majority_vote_predict(tree_class_predictions, classes):    votes = class_vote_counts(tree_class_predictions, classes)    classes = np.asarray(classes)    return classes[np.argmax(votes, axis=1)]  def mean_tree_class_probabilities(tree_class_probabilities):    probs = np.asarray(tree_class_probabilities, dtype=float)    if probs.ndim != 3:        raise ValueError(            "tree_class_probabilities must be trees by observations by classes."        )    return probs.mean(axis=0)  def predict_from_mean_probabilities(mean_probabilities, classes):    mean_probabilities = np.asarray(mean_probabilities, dtype=float)    classes = np.asarray(classes)    if mean_probabilities.ndim != 2:        raise ValueError("mean_probabilities must be observations by classes.")    if mean_probabilities.shape[1] != classes.shape[0]:        raise ValueError("mean_probabilities must have one column per class.")    return classes[np.argmax(mean_probabilities, axis=1)] 

1. 1.

   Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32.

   doi:10.1023/A:1010933404324
2. 2.

   Menze, B. H., Kelm, B. M., Masuch, R., Himmelreich, U., Bachert, P., Petrich, W., & Hamprecht, F. A. (2009). A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data. BMC Bioinformatics, 10, 213.

   doi:10.1186/1471-2105-10-213
3. 3.

   Scott, I. M., Lin, W., Liakata, M., Wood, J. E., Vermeer, C. P., Allaway, D., Ward, J. L., Draper, J., Beale, M. H., Corol, D. I., Baker, J. M., & King, R. D. (2013). Merits of random forests emerge in evaluation of chemometric classifiers by external validation. Analytica Chimica Acta, 801, 22-33.

   doi:10.1016/j.aca.2013.09.027
4. 4.

   scikit-learn Developers (n.d.). RandomForestClassifier, ensemble user guide section on forests of randomized trees, and decision-tree classification criteria. scikit-learn 1.6 documentation.

5. 5.

   The MathWorks, Inc. (n.d.). TreeBagger, predict, oobError, oobPredict, OOBIndices, and OOBPredictorImportance. MATLAB documentation.