Hi I am fairly new to Random Forest estimation, but I could not find a questions similiar to mine. I was surprised that the predictions are different using the same predictors. I would have expected the same. I understand, that the model would be different with each estimation, but getting different predictions for the same predictors?

library(randomForest)

set.seed(100)

df<-mtcars

rt.est<-randomForest(mpg ~ ., 

                     data = df,

                     ntree=1000)

predict(rt.est)

df.double<-rbind(df,df[32,])

rt.est<-randomForest(mpg ~ ., 

                     data = df.double,

                    ntree=1000)

predict(rt.est)    

The results for the last Observation on the Volvo142E are similiar but not the same. Why?

If you supply the argument newdata the discrepancy disappears: predict(rt.est, newdata=df) gives

Volvo 142E 

22.15557 



Volvo 142E1 

22.15557

When you do not supply newdata, it's probably reporting the out-of-bag results, but I haven't found an explicit clarification of this in the documentation. Samples that are "in-bag" were included in a tree during training as a result of the bootstrap re-sampling procedure; out-of-bag samples were omitted.

We can verify that this is out-of-bag data by calling rt.est$predicted which reports the out-of-bag predictions. The results match predict(rt.est).

Volvo 142E 

22.83609 



Volvo 142E1 

22.85975 

One way to think of predict.randomForest is that it's a shortcut to the out-of-bag predictions unless you supply newdata.

OP originally asked about 2 different ensembles of random forests. This portion of the answer addresses why 2 random forest ensembles might make different predictions on the same data.

The trees are different.

First, randomForest is a random procedure: both the samples chosen for each tree are different (bootstrap resampling), and the features chosen at each split are chosen at random (randomized feature subspaces). Without fixing the random seed, we would expect two randomForest runs to produce different results with high probability for the same reason that flipping a fair coin 1000 times will plausibly result in a different sequence of heads and tails. (You have fixed the seed, however the two instances of randomForest will still be different because the random state is altered after the first randomForest is produced.)

Second, The data used to train the models is different. It appears that you've added an additional row. Different data makes for a different model, which makes for a different prediction. When randomForest conducts its bootstrapping, the probability that df[32,] is in-bag for that tree is larger than for each non-duplicated sample. This change to the data will also change the trees, because choices about where to make splits will be influenced by the increased prominence of this sample.

Different trees make different predictions.

Having the same feature values is only half the battle. The other half is how the trees are constructed.

As an example, suppose I have 3 trees constructed with the random forest procedure, each with 1 split.

1. This tree has a bootstrap resample and randomly samples cyl, disp and hp. It picks splitting on cyl at 5 as the best split.


2. This tree has a bootstrap resample and randomly samples cyl, disp and hp. It picks splitting on cyl at 7 as the best split.


3. This tree has a bootstrap resample and randomly samples wt, disp and hp. It picks splitting on hp at 123 as the best split.

Clearly there will be different predictions whenever the splits change the decision of a sample. A sample with cyl 6 might go "right" for tree 1, but "left" for tree 2. Feature hp doesn't have a one-to-one relationship to cyl, so a split on hp won't generally match splits on cyl.