Model Selection and Multi-Model Inference

by Rosemary Hartman, 20 February 2013


At D-RUG this week Rosemary Hartman presented a really useful case study in model selection, based on her work on frog habitat. Here is her code run through ‘knitr’. Original code and data are posted here.

(yes, I am just doing this for the flying monkey)

Editor’s note: we’re giving away flying monkey dolls from our sponsor, Revolution Analytics, to all our D-RUG presenters.

So, let’s say you want to find out where things are and why they are there. But there are a lot of reasons someone might be somewhere

Let’s say these things are fishermen, And we aren’t sure what the reasons are yet.

Start by making some good guesses. Maybe they all go to the big lakes. Maybe they all go to the lakes that CDFW stocks with fish. Maybe they go to the lakes with lots of frogs (not likely, but hey, why not try?) Once you have all your guesses, you need to figure out which one is right. There are two basic methods to use (or two that I have been exploring)

Method #1: The kitchen sink

load package “glmulti”

library(glmulti)

load the data

lakes.df2 <- read.csv("lakes.df2.csv")

create a model that has all the predictor variables you would like to test

global.model <-  glm(fish~ # number of fishermen
                       stocked+ # CDFW fish stocking y/n
                       cows+ # cattle grazing in area y/n
                       herp.rich+ # amphibian species richness
                       fish.rich+ # fish species richness
                       visitors+ # total number of visitors
                       logsize, # log of the area
                     data=lakes.df2) # data frame we got this from

Now we will use the “glmulti” function to find the best model; this goes through every possible model and finds the best one.

fish.model <- glmulti(global.model, # use the model with built as a starting point
                      level = 1,  #  just look at main effects
                      crit="aicc") # use AICc because it works better than AIC for small sample sizes
## Initialization...
## TASK: Exhaustive screening of candidate set.
## Fitting...
## 
## After 50 models:
## Best model: fish~1+cows+herp.rich+visitors+logsize
## Crit= 201.820969266156
## Mean crit= 355.848076682524
## Completed.

summary(fish.model)
## $name
## [1] "glmulti.analysis"
## 
## $method
## [1] "h"
## 
## $fitting
## [1] "glm"
## 
## $crit
## [1] "aicc"
## 
## $level
## [1] 1
## 
## $marginality
## [1] FALSE
## 
## $confsetsize
## [1] 100
## 
## $bestic
## [1] 201.8
## 
## $icvalues
##  [1] 201.8 203.4 204.7 205.0 206.4 206.5 206.5 208.3 225.4 227.7 230.6
## [12] 231.2 232.7 233.2 233.5 235.1 236.0 236.4 238.2 238.3 238.4 238.8
## [23] 240.7 240.9 258.2 259.3 260.2 261.0 261.1 261.7 262.0 262.6 265.1
## [34] 267.6 268.1 270.5 276.0 277.4 278.3 279.8 310.2 311.5 313.6 314.0
## [45] 315.4 316.2 319.0 319.7 500.6 502.0 502.5 503.0 503.8 504.8 511.3
## [56] 513.2 599.7 601.4 610.5 614.9 615.3 624.1 629.2 638.6
## 
## $bestmodel
## [1] "fish ~ 1 + cows + herp.rich + visitors + logsize"
## 
## $modelweights
##  [1] 4.485e-01 2.024e-01 1.075e-01 9.285e-02 4.464e-02 4.343e-02 4.310e-02
##  [8] 1.767e-02 3.399e-06 1.065e-06 2.477e-07 1.895e-07 8.849e-08 6.754e-08
## [15] 6.011e-08 2.630e-08 1.704e-08 1.404e-08 5.716e-09 5.287e-09 4.990e-09
## [22] 4.134e-09 1.618e-09 1.436e-09 2.540e-13 1.465e-13 9.334e-14 6.306e-14
## [29] 5.969e-14 4.361e-14 3.790e-14 2.885e-14 8.119e-15 2.304e-15 1.837e-15
## [36] 5.540e-16 3.542e-17 1.739e-17 1.096e-17 5.151e-18 1.300e-24 7.003e-25
## [43] 2.397e-25 1.957e-25 9.964e-26 6.413e-26 1.594e-26 1.123e-26 5.852e-66
## [50] 2.976e-66 2.260e-66 1.801e-66 1.220e-66 7.203e-67 2.847e-68 1.069e-68
## [57] 1.836e-87 7.713e-88 8.222e-90 9.195e-91 7.394e-91 9.070e-93 7.028e-94
## [64] 6.396e-96
## 
## $includeobjects
## [1] TRUE

That showed us the best model, now lets look at some of the others

weightable(fish.model)
##                                                                     model
## 1                        fish ~ 1 + cows + herp.rich + visitors + logsize
## 2            fish ~ 1 + cows + herp.rich + fish.rich + visitors + logsize
## 3              fish ~ 1 + stocked + cows + herp.rich + visitors + logsize
## 4                                    fish ~ 1 + cows + visitors + logsize
## 5                        fish ~ 1 + cows + fish.rich + visitors + logsize
## 6                          fish ~ 1 + stocked + cows + visitors + logsize
## 7  fish ~ 1 + stocked + cows + herp.rich + fish.rich + visitors + logsize
## 8              fish ~ 1 + stocked + cows + fish.rich + visitors + logsize
## 9                         fish ~ 1 + stocked + cows + herp.rich + logsize
## 10            fish ~ 1 + stocked + cows + herp.rich + fish.rich + logsize
## 11                                  fish ~ 1 + cows + herp.rich + logsize
## 12                                    fish ~ 1 + stocked + cows + logsize
## 13                                              fish ~ 1 + cows + logsize
## 14                      fish ~ 1 + cows + herp.rich + fish.rich + logsize
## 15                        fish ~ 1 + stocked + cows + fish.rich + logsize
## 16                                  fish ~ 1 + cows + fish.rich + logsize
## 17                              fish ~ 1 + herp.rich + visitors + logsize
## 18                                          fish ~ 1 + visitors + logsize
## 19                  fish ~ 1 + herp.rich + fish.rich + visitors + logsize
## 20                              fish ~ 1 + fish.rich + visitors + logsize
## 21                    fish ~ 1 + stocked + herp.rich + visitors + logsize
## 22                                fish ~ 1 + stocked + visitors + logsize
## 23        fish ~ 1 + stocked + herp.rich + fish.rich + visitors + logsize
## 24                    fish ~ 1 + stocked + fish.rich + visitors + logsize
## 25                     fish ~ 1 + cows + herp.rich + fish.rich + visitors
## 26                                 fish ~ 1 + cows + herp.rich + visitors
## 27           fish ~ 1 + stocked + cows + herp.rich + fish.rich + visitors
## 28                                 fish ~ 1 + cows + fish.rich + visitors
## 29                       fish ~ 1 + stocked + cows + herp.rich + visitors
## 30                                             fish ~ 1 + cows + visitors
## 31                       fish ~ 1 + stocked + cows + fish.rich + visitors
## 32                                   fish ~ 1 + stocked + cows + visitors
## 33                               fish ~ 1 + stocked + herp.rich + logsize
## 34                   fish ~ 1 + stocked + herp.rich + fish.rich + logsize
## 35                                           fish ~ 1 + stocked + logsize
## 36                               fish ~ 1 + stocked + fish.rich + logsize
## 37                                                     fish ~ 1 + logsize
## 38                                         fish ~ 1 + herp.rich + logsize
## 39                                         fish ~ 1 + fish.rich + logsize
## 40                             fish ~ 1 + herp.rich + fish.rich + logsize
## 41                      fish ~ 1 + stocked + cows + herp.rich + fish.rich
## 42                                  fish ~ 1 + stocked + cows + fish.rich
## 43                                            fish ~ 1 + cows + fish.rich
## 44                                fish ~ 1 + cows + herp.rich + fish.rich
## 45                                  fish ~ 1 + stocked + cows + herp.rich
## 46                                              fish ~ 1 + stocked + cows
## 47                                                        fish ~ 1 + cows
## 48                                            fish ~ 1 + cows + herp.rich
## 49                            fish ~ 1 + herp.rich + fish.rich + visitors
## 50                  fish ~ 1 + stocked + herp.rich + fish.rich + visitors
## 51                                        fish ~ 1 + herp.rich + visitors
## 52                                        fish ~ 1 + fish.rich + visitors
## 53                              fish ~ 1 + stocked + herp.rich + visitors
## 54                              fish ~ 1 + stocked + fish.rich + visitors
## 55                                                    fish ~ 1 + visitors
## 56                                          fish ~ 1 + stocked + visitors
## 57                             fish ~ 1 + stocked + herp.rich + fish.rich
## 58                                         fish ~ 1 + stocked + fish.rich
## 59                                         fish ~ 1 + stocked + herp.rich
## 60                                                   fish ~ 1 + fish.rich
## 61                                       fish ~ 1 + herp.rich + fish.rich
## 62                                                     fish ~ 1 + stocked
## 63                                                   fish ~ 1 + herp.rich
## 64                                                               fish ~ 1
##     aicc   weights
## 1  201.8 4.485e-01
## 2  203.4 2.024e-01
## 3  204.7 1.075e-01
## 4  205.0 9.285e-02
## 5  206.4 4.464e-02
## 6  206.5 4.343e-02
## 7  206.5 4.310e-02
## 8  208.3 1.767e-02
## 9  225.4 3.399e-06
## 10 227.7 1.065e-06
## 11 230.6 2.477e-07
## 12 231.2 1.895e-07
## 13 232.7 8.849e-08
## 14 233.2 6.754e-08
## 15 233.5 6.011e-08
## 16 235.1 2.630e-08
## 17 236.0 1.704e-08
## 18 236.4 1.404e-08
## 19 238.2 5.716e-09
## 20 238.3 5.287e-09
## 21 238.4 4.990e-09
## 22 238.8 4.134e-09
## 23 240.7 1.618e-09
## 24 240.9 1.436e-09
## 25 258.2 2.540e-13
## 26 259.3 1.465e-13
## 27 260.2 9.334e-14
## 28 261.0 6.306e-14
## 29 261.1 5.969e-14
## 30 261.7 4.361e-14
## 31 262.0 3.790e-14
## 32 262.6 2.885e-14
## 33 265.1 8.119e-15
## 34 267.6 2.304e-15
## 35 268.1 1.837e-15
## 36 270.5 5.540e-16
## 37 276.0 3.542e-17
## 38 277.4 1.739e-17
## 39 278.3 1.096e-17
## 40 279.8 5.151e-18
## 41 310.2 1.300e-24
## 42 311.5 7.003e-25
## 43 313.6 2.397e-25
## 44 314.0 1.957e-25
## 45 315.4 9.964e-26
## 46 316.2 6.413e-26
## 47 319.0 1.594e-26
## 48 319.7 1.123e-26
## 49 500.6 5.852e-66
## 50 502.0 2.976e-66
## 51 502.5 2.260e-66
## 52 503.0 1.801e-66
## 53 503.8 1.220e-66
## 54 504.8 7.203e-67
## 55 511.3 2.847e-68
## 56 513.2 1.069e-68
## 57 599.7 1.836e-87
## 58 601.4 7.713e-88
## 59 610.5 8.222e-90
## 60 614.9 9.195e-91
## 61 615.3 7.394e-91
## 62 624.1 9.070e-93
## 63 629.2 7.028e-94
## 64 638.6 6.396e-96

So this is the best model

f <- glm(fish ~ 1 + cows + herp.rich + visitors + logsize, data = lakes.df2)
summary(f)
## 
## Call:
## glm(formula = fish ~ 1 + cows + herp.rich + visitors + logsize, 
##     data = lakes.df2)
## 
## Deviance Residuals: 
##    Min      1Q  Median      3Q     Max  
## -4.744  -1.350  -0.243   0.986  10.483  
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -0.5262     0.8914   -0.59    0.559    
## cowsy        -1.5214     1.2088   -1.26    0.217    
## herp.rich     0.7614     0.3219    2.37    0.024 *  
## visitors      0.1799     0.0277    6.49  1.8e-07 ***
## logsize       1.0498     1.1590    0.91    0.371    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 
## 
## (Dispersion parameter for gaussian family taken to be 7.225)
## 
##     Null deviance: 791.60  on 39  degrees of freedom
## Residual deviance: 252.87  on 35  degrees of freedom
##   (79 observations deleted due to missingness)
## AIC: 199.3
## 
## Number of Fisher Scoring iterations: 2

I hate trying to interpret models based on tables of coeficients. Let’s look at some graphs

library(visreg)
visreg(f)

But according to theory, models with AIC within two points of each other are basically equal. So what about the other models? Should we totally throw them out? Looking at the table of aicc weights, there is a pretty big jump between model 8 and model 9. So lets try averaging the top 8 models if we want to use this to make predictions, evaluate variable importance, etc.

library(MuMIn)

run the top 8 models

f2 <- glm(fish ~ 1 + cows + herp.rich + fish.rich + visitors + logsize, data = lakes.df2)
f3 <- glm(fish ~ 1 + stocked + cows + herp.rich + visitors + logsize, data = lakes.df2)
f4 <- glm(fish ~ 1 + cows + visitors + logsize, data = lakes.df2)
f5 <- glm(fish ~ 1 + cows + fish.rich + visitors + logsize, data = lakes.df2)
f6 <- glm(fish ~ 1 + stocked + cows + visitors + logsize, data = lakes.df2)
f7 <- glm(fish ~ 1 + stocked + cows + herp.rich + fish.rich + visitors + logsize, 
    data = lakes.df2)
f8 <- glm(fish ~ 1 + stocked + cows + fish.rich + visitors + logsize, data = lakes.df2)

average the models together

f.ave <- model.avg(f, f2, f3, f4, f5, f6, f7, f8)
summary(f.ave)
## 
## Call:
## model.avg.default(object = f, f2, f3, f4, f5, f6, f7, f8)
## 
## Component models:
##        df logLik  AICc Delta Weight
## 1346    6 -93.64 201.8  0.00   0.45
## 12346   7 -92.96 203.4  1.59   0.20
## 13456   7 -93.59 204.7  2.86   0.11
## 146     5 -96.60 205.0  3.15   0.09
## 1246    6 -95.95 206.4  4.61   0.04
## 1456    6 -95.97 206.5  4.67   0.04
## 123456  8 -92.93 206.5  4.68   0.04
## 12456   7 -95.39 208.3  6.47   0.02
## 
## Term codes:
##      cows fish.rich herp.rich   logsize   stocked  visitors 
##         1         2         3         4         5         6 
## 
## Model-averaged coefficients: 
##             Estimate Std. Error Adjusted SE z value Pr(>|z|)    
## (Intercept)  -0.5422     1.2363      1.2673    0.43    0.669    
## cowsy        -1.3751     1.2596      1.3019    1.06    0.291    
## herp.rich     0.7523     0.3274      0.3393    2.22    0.027 *  
## visitors      0.1841     0.0305      0.0316    5.83   <2e-16 ***
## logsize       0.7763     1.2689      1.3119    0.59    0.554    
## fish.rich     0.7784     0.7245      0.7512    1.04    0.300    
## stockedy     -0.7316     1.6009      1.6539    0.44    0.658    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 
## 
## Full model-averaged coefficients (with shrinkage): 
##  (Intercept)  cowsy herp.rich visitors logsize fish.rich stockedy
##       -0.542 -1.375     0.603    0.184   0.776     0.240   -0.155
## 
## Relative variable importance:
##      cows   logsize  visitors herp.rich fish.rich   stocked 
##      1.00      1.00      1.00      0.80      0.31      0.21

Now we know the relative variable importance, all of the avereged coefficents, etc, and can use the “predict” funciton to predict how many fishermen will be at the next lake. Graphing these are more difficult, so we’ll skip this for now. Ask me later if you really want to know.

Model selection method #2: Use your brain We often can discard (or choose) some models a priori based on our knowlege of the system. It’s usually better to do it this way if you have several hundered possible combination of variables, or want to put in some interaction terms. I used this method for my frog data. load package bbmle

library(bbmle)

load the data

fishlakes <- read.csv("fishlakes.csv")

Decide on which set of models you want to use. This is hard. A statistician who knows a lot more than I do told me so. I spent a long time playing around with different transformations, predictor variables, and combinations before I came up with a set of hypotheses (models) that I was happy with.

m1 <- glm(treatment ~ 1 + logveg + bank.slope, family = binomial("logit"), data = fishlakes)
m2 <- glm(treatment ~ 1 + logveg + silt.total + bank.slope, family = binomial("logit"), 
    data = fishlakes)
m3 <- glm(treatment ~ 1 + raca.basin + logveg + bank.slope, family = binomial("logit"), 
    data = fishlakes)
m4 <- glm(treatment ~ 1 + BUBO.breeding + logveg + bank.slope, family = binomial("logit"), 
    data = fishlakes)
m5 <- glm(treatment ~ 1 + BUBO.breeding + logveg + bank.slope + raca.basin, 
    family = binomial("logit"), data = fishlakes)
m6 <- glm(treatment ~ 1 + raca.basin * BUBO.breeding, family = binomial("logit"), 
    data = fishlakes)
m7 <- glm(treatment ~ 1 + herbacious + raca.basin + lakes.basin, family = binomial("logit"), 
    data = fishlakes)
m8 <- glm(treatment ~ 1 + BUBO.breeding, family = binomial("logit"), data = fishlakes)
m9 <- glm(treatment ~ BUBO.breeding + herbacious + bank.slope + logveg, family = binomial("logit"), 
    data = fishlakes)
m10 <- glm(treatment ~ BUBO.breeding * raca.basin + herbacious + lakes.basin, 
    family = binomial("logit"), data = fishlakes)
m11 <- glm(treatment ~ 1 + bank.slope + logveg + BUBO.breeding:raca.basin, family = binomial("logit"), 
    data = fishlakes)

Now let’s rank them via AICc

AICctab(m1, m2, m3, m4, m5, m6, m7, m8, m9, m10, m11, base = T, weights = T, 
    nobs = length(fishlakes))
##     AICc df dAICc weight 
## m11 35.9 5   0.0  0.79445
## m4  40.5 4   4.6  0.08072
## m9  40.9 5   5.1  0.06336
## m1  42.9 3   7.0  0.02424
## m5  43.0 5   7.1  0.02275
## m2  45.3 4   9.4  0.00734
## m3  45.3 4   9.4  0.00708
## m8  56.6 2  20.7  < 0.001
## m10 57.1 6  21.2  < 0.001
## m6  57.3 4  21.4  < 0.001
## m7  62.1 4  26.2  < 0.001

Model m11 comes out ahead by quite a bit, but we’ll average the top two models, just to show you how its done.

m.ave <- model.avg(m4, m11)
summary(m.ave)
## 
## Call:
## model.avg.default(object = m4, m11)
## 
## Component models:
##     df logLik  AICc Delta Weight
## 134  5 -12.09 35.84  0.00   0.91
## 123  4 -15.68 40.43  4.59   0.09
## 
## Term codes:
##               bank.slope            BUBO.breeding                   logveg 
##                        1                        2                        3 
## BUBO.breeding:raca.basin 
##                        4 
## 
## Model-averaged coefficients: 
##                           Estimate Std. Error Adjusted SE z value Pr(>|z|)
## (Intercept)                 -1.356      3.575       3.694    0.37    0.714
## bank.slope                 -19.935      8.233       8.503    2.34    0.019
## logveg                       1.783      1.366       1.409    1.26    0.206
## BUBO.breedingn:raca.basin   -0.965      0.973       1.006    0.96    0.337
## BUBO.breedingy:raca.basin    1.313      0.699       0.722    1.82    0.069
## BUBO.breedingy               2.077      1.055       1.090    1.91    0.057
##                            
## (Intercept)                
## bank.slope                *
## logveg                     
## BUBO.breedingn:raca.basin  
## BUBO.breedingy:raca.basin .
## BUBO.breedingy            .
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 
## 
## Full model-averaged coefficients (with shrinkage): 
##  (Intercept) bank.slope  logveg BUBO.breedingn:raca.basin
##       -1.356    -19.935   1.783                    -0.877
##  BUBO.breedingy:raca.basin BUBO.breedingy
##                      1.193          0.190
## 
## Relative variable importance:
##               bank.slope                   logveg BUBO.breeding:raca.basin 
##                     1.00                     1.00                     0.91 
##            BUBO.breeding 
##                     0.09
# OK, there is a predictive model, but how good is it?

Let’s try cross-validation first. If this was a single model, we could try using the cv.glm function from the “boot” library like this:

library("boot")
cost <- function(r, pi = 0) mean(abs(r - pi) > 0.5)  ## cost function necessary for binomial data
m11.cv <- cv.glm(data = fishlakes, m11, cost, K = 42)  # use leave-one-out cross validation (can use K-fold cross validation for larger data sets)
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred

Now lets see what our error rate was:

m11.cv$delta
## [1] 0.2381 0.2438

That’s not too bad.

IF we want to check the error rate of an averaged model, you need to get more creative. I’ve written some code to do this for averaged models that only have two component models. It shouldn’t be too hard to adapt this for more models.

function for leave one out cross validation of averaged models

Cross <- function(model1, model2, data, cost) {
    library(MuMIn)
    nobs <- nrow(data)
    model.ave <- model.avg(model1, model2)
    values <- matrix(NA, nrow = nobs, ncol = 5)
    values[, 1] <- data$treatment
    values[, 2] <- predict(model.ave, type = "response")
    CV = 0
    for (i in 1:nobs) {
        data2 <- data[-i, ]  # leave out one observasion
        model12 <- glm(model1$formula, family = model1$family, data = data2)
        model22 <- glm(model2$formula, family = model2$family, data = data2)
        model.ave2 <- model.avg(model12, model22)
        values[i, 3] <- predict(model.ave2, newdata = data[i, ], type = "response")
        values[i, 4] <- round(values[i, 3])
        if (values[i, 4] == values[i, 1]) 
            (values[i, 5] = 1) else values[i, 5] = 0
        
    }
    cv = mean(abs(values[, 3] - values[, 1]) > 0.5)
    return(cv)
}

Use the function on the component models

Cross(m11, m4, fishlakes, cost)
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred

## [1] 0.2381

That looks reasonable

Another method to test model accurace is Area Under the Reciever Operater Curve (AUC) This is baisically a plot of true presences versus false presences in a presence-absense model.

Load the library “pROC”

library(pROC)

Make your reciever-operater curve

m.roc <- roc(fishlakes$treatment, predict(m.ave, backtransform = TRUE))
plot(m.roc)
## 
## Call:
## roc.default(response = fishlakes$treatment, predictor = predict(m.ave,     backtransform = TRUE))
## 
## Data: predict(m.ave, backtransform = TRUE) in 24 controls (fishlakes$treatment 0) < 18 cases (fishlakes$treatment 1).
## Area under the curve: 0.949

Looks like a pretty good fit. Not too bad for the small size of the data set.

And that’s all I got. Hopefully you will find it helpful. Let me know if there is anything else I have forgotton or done wrong.

~ Rosemary

rosehartman@gmail.com