Lesson 3 Logistic Regression Diagnostics

Lesson 3 Logistic Regression Diagnostics. Chapter 3 Logistic Regression Diagnostics. NOTE: This page is under construction!! In the previous two chapters, we focused on issues regarding logistic regression analysis, such as how to create interaction variables and how to interpret the results of our logistic model. In order for our analysis to be valid, our model has to satisfy the assumptions of logistic regression. When the assumptions of logistic regression analysis are not met, we may have problems, such as biased coefficient estimates or very large standard errors for the logistic regression coefficients, and these problems may lead to invalid statistical inferences. Therefore, before we can use our model to make any statistical inference, we need to check that our model fits sufficiently well and check for influential observations that have impact on the estimates of the coefficients. In this chapter, we are going to focus on how to assess model fit, how to diagnose potential problems in our model and how to identify observations that have significant impact on model fit or parameter estimates. Let’s begin with a review of the assumptions of logistic regression. The true conditional probabilities are a logistic function of the independent variables. No important variables are omitted. No extraneous variables are included. The independent variables are measured without error. The observations are independent. The independent variables are not linear combinations of each other. In this chapter, we are going to continue to use the apilog dataset. 3.1 Specification Error. When we build a logistic regression model, we assume that the logit of the outcome variable is a linear combination of the independent variables. This involves two aspects, as we are dealing with the two sides of our logistic regression equation. First, consider the link function of the outcome variable on the left hand side of the equation. We assume that the logit function (in logistic regression) is the correct function to use. Secondly, on the right hand side of the equation, we assume that we have included all the relevant variables, that we have not included any variables that should not be in the model, and the logit function is a linear combination of the predictors. It could happen that the logit function as the link function is not the correct choice or the relationship between the logit of outcome variable and the independent variables is not linear. In either case, we have a specification error. The misspecification of the link function is usually not too severe compared with using other alternative link function choices such as probit (based on the normal distribution). In practice, we are more concerned with whether our model has all the relevant predictors and if the linear combination of them is sufficient. The Stata command linktest can be used to detect a specification error, and it is issued after the logit or logistic command. The idea behind linktest is that if the model is properly specified, one should not be able to find any additional predictors that are statistically significant except by chance. After the regression command (in our case, logit or logistic ), linktest uses the linear predicted value ( _hat ) and linear predicted value squared ( _hatsq ) as the predictors to rebuild the model. The variable _hat should be a statistically significant predictor, since it is the predicted value from the model. This will be the case unless the model is completely misspecified. On the other hand, if our model is properly specified, variable _hatsq shouldn’t have much predictive power except by chance. Therefore, if _hatsq is significant, then the linktest is significant. This usually means that either we have omitted relevant variable(s) or our link function is not correctly specified. Now let’s look at an example. In our api dataset, we have a variable called cred_ml , which is defined for 707 observations (schools) whose percentage of credential teachers are in the middle and lower range. For this subpopulation of schools, we believe that the variables yr_rnd , meals and cred_ml are powerful predictors for predicting if a school’s api score is high. So we ran the following logit command followed by the linktest command. We first see in the output from the logit command that the three predictors are all statistically significant predictors, and in the linktest that followed, the variable _hatsq is significant (with p-value = 0.006). This confirms, on one hand, that we have chosen meaningful predictors. On the other hand, it tells us that we have a specification error (since the linktest is significant). The first thing to do to remedy the situation is to see if we have included all of the relevant variables. More often than not, we thought we had included all of the variables, but we have overlooked the possible interactions among some of the predictor variables. This may be the case with our model. So we try to add an interaction term to our model. We create an interaction variable ym =yr_rnd*meals and add it to our model and try the linktest again. First of all, the interaction term is significant with p-value =.015. Secondly, the linktest is no longer significant. This is an indication that we should include the interaction term in the model, and by including it, we get a better model in terms of model specification. Let’s now compare the two models we just built. From the output of our first logit command, we have the following regression equation: logit ( hiqual ) = 2.411226 – 1.185658* yr_rnd -.0932877* meals + .7415145* cred_ml. This model does not have the interaction of the variables yr_rnd and meals . Therefore, the effect of the variable meals is the same regardless whether a school is a year-around school or not. On the other hand, in the second model, logit(hiqual) = 2.668048 – 2.816989* yr_rnd -.1014958* meals + .7795476* cred_ml + .0459029* ym, the effect of the variable meals is different depending on if a school is a year-around school or not. More precisely, if a school is not a year-around school, the effect of the variable meals is -.1014958 on logit of the outcome variable hiqual and the effect is -.1014958 + .0459029 = -.0555929 for a year-around school. This makes sense since a year-around school usually has a higher percentage of students on free or reduced-priced meals than a non-year-around school. Therefore, within year-around schools, the variable meals is no longer as powerful as it is for a general school. This tells us that if we do not specify our model correctly, the effect of variable meals could be estimated with bias. We need to keep in mind that linkest is simply a tool that assists in checking our model. It has its limits. It is better if we have a theory in mind to guide our model building, that we check our model against our theory, and that we validate our model based on our theory. Let’s look at another example where the linktest is not working so well. We will build a model to predict hiqual using yr_rnd and awards as predictors. Notice that the pseudo R-square is .076, which is on the low side. Nevertheless, we run the linktest , and it turns out to be very non-significant (p=.909). It turns out that _hatsq and _hat are highly correlated with correlation of -.9617, yielding a non-significant _hatsq since it does not provide much new information beyond _hat itself. We know that the variable meals is very much related with the outcome variable and that we should have it in our model. So we consequently run another model with meals as an additional predictor. This time the linktest turns out to be significant. Which one is the better model? If we look at the pseudo R-square, for instance, it goes way up from .076 to .5966. We will definitely go with the second model. This tells us that the linktest is a limited tool to detect specification errors just as any other tools. It is useful to help us to detect, but we need to use our best judgment, as always. We have seen earlier that lacking an interaction term could cause a model specification problem. Similarly, we could also have a model specification problem if some of the predictor variables are not properly transformed. For example, the change of a dependent variable on a predictor may not be linear, but only the linear term is used as a predictor in the model. To address this, a Stata program called boxtid can be used. It is a user-written program that you can download over the internet by typing " search boxtid ". boxtid stands for Box-Tidwell model, which transforms a predictor using power transformations and finds the best power for model fit based on maximal likelihood estimate. More precisely, a predictor x is transformed into B 1 + B 2 x p and the best p is found using maximal likelihood estimate. Besides estimating the power transformation, boxtid also estimates exponential transformations, which can be viewed as power functions on the exponential scale. Let’s look at another model where we predict hiqaul from yr_rnd and meals . We’ll start with a model with only two predictors. The linktest is significant, indicating problem with model specification. We then use boxtid , and it displays the best transformation of the predictor variables, if needed. The test of nonlinearity for the variable meals is statistically significant with p-value =.005 . The null hypothesis is that the predictor variable meals is of a linear term, or, equivalently, p1 = 1. But it shows that p1 is around .55 to be optimal. This suggests a square-root transformation of the variable meals . So let’s try this approach and replace the variable meals with the square-root of itself. This might be consistent with a theory that the effect of the variable meals will attenuate at the end. This shows that sometimes the logit of the outcome variable may not be a linear combination of the predictors variables, but a linear combination of transformed predictor variables, possibly with interaction terms. We have only scratched the surface on how to deal with the issue of specification errors. In practice, a combination of a good grasp of the theory behind the model and a bundle of statistical tools to detect specification error and other potential problems is necessary to guide us through model building. References on where to find more information and/or examples? We have seen from our previous lessons that Stata’s output of logistic regression contains the log likelihood chi-square and pseudo R-square for the model. These measures, together with others that we are also going to discuss in this section, give us a general gauge on how the model fits the data. Let’s start with a model that we have shown previously. The log likelihood chi-square is an omnibus test to see if the model as a whole is statistically significant. It is 2 times the difference between the log likelihood of the current model and the log likelihood of the intercept-only model. Since Stata always starts its iteration process with the intercept-only model, the log likelihood at Iteration 0 shown above corresponds to the log likelihood of the empty model. The four degrees of freedom comes from the four predictor variables that the current model has. A pseudo R-square is in slightly different flavor, but captures more or less the same thing in that it is the proportion of change in terms of likelihood. It is a "pseudo" R-square because it is unlike the R-square found in OLS regression, where R-square measures the proportion of variance explained by the model. The pseudo R-square is not measured in terms of variance, since in logistic regression the variance is fixed as the variance of the standard logistic distribution. However, it is still a proportion in terms of the log likelihood. Because of the problem that it (what??) will never be 1, there have been many variations of this particular pseudo R-square. We should also note that different pseudo R-squares can give very different assessments of a model’s fit, and that there is no one version of pseduo R-square that is preferred by most data analysts over other versions. Another commonly used test of model fit is the Hosmer and Lemeshow’s goodness-of-fit test. The idea behind the Hosmer and Lemeshow’s goodness-of-fit test is that the predicted frequency and observed frequency should match closely, and that the more closely they match, the better the fit. The Hosmer-Lemeshow goodness-of-fit statistic is computed as the Pearson chi-square from the contingency table of observed frequencies and expected frequencies. Similar to a test of association of a two-way table, a good fit as measured by Hosmer and Lemeshow’s test will yield a large p-value. When there are continuous predictors in the model, there will be many cells defined by the predictor variables, making a very large contingency table, which would yield significant result more than often. So a common practice is to combine the patterns formed by the predictor variables into 10 groups and form a contingency table of 2 by 10. With a p-value of .33, we can say that Hosmer and Lemeshow’s goodness-of-fit test indicates that our model fits the data well. There are many other measures of model fit, such AIC (Akaike Information Criterion ) and BIC ( Bayesian Information Criterion ) . A command called fitstat will display most of them after a model. Many times, fitstat is used to compare models. Let’s say we want to compare the current model which includes the interaction term of yr_rnd and meals with a model without the interaction term. We can use the fitsat options using and saving to compare models. Note that fitstat should only be used to compare nested models. The first fitstat displays and saves the fit statistics for the larger model, and the second one uses the saved information to compare with the current model. The result supports the model with no interaction over the model with the interaction, but only weakly. On the other hand, we have already shown that the interaction term is significant. But if we look more closely, we can see its coefficient fairly small in the logit scale and is very close to 1 in the odds ratio scale. So the substantive meaning of the interaction being statistically significant may not be as prominent as it looks. 3.3 Multicollinearity. Multicollinearity (or collinearity for short) occurs when two or more independent variables in the model are approximately determined by a linear combination of other independent variables in the model. For example, we would have a problem with multicollinearity if we had both height measured in inches and height measured in feet in the same model. The degree of multicollinearity can vary and can have different effects on the model. When perfect collinearity occurs, that is, when one independent variable is a perfect linear combination of the others, it is impossible to obtain a unique estimate of regression coefficients with all the independent variables in the model. What Stata does in this case is to drop a variable that is a perfect linear combination of the others, leaving only the variables that are not exactly linear combinations of others in the model to assure unique estimate of regression coefficients. For example, we can artificially create a new variable called perli as the sum of yr_rnd and meals . Notice that the only purpose of this example and the creation of the variable perli is to show what Stata does when perfect collinearity occurs. Notice that Stata issues a note, informing us that the variable yr_rnd has been dropped from the model due to collinearity. We cannot assume that the variable that Stata drops from the model is the "correct" variable to omit from the model; rather, we need to rely on theory to determine which variable should be omitted. Moderate multicollinearity is fairly common since any correlation among the independent variables is an indication of collinearity. When severe multicollinearity occurs, the standard errors for the coefficients tend to be very large (inflated), and sometimes the estimated logistic regression coefficients can be highly unreliable. Let’s consider the following example. In this model, the dependent variable will be hiqual , and the predictor variables will include avg_ed , yr_rnd , meals , full , and the interaction between yr_rnd and full, yxfull . After the logit procedure, we will also run a goodness-of-fit test. Notice that the goodness-of-fit test indicates that, overall, our model fits pretty well. Nevertheless, notice the odd ratio and standard error for the variable yr_rnd are incredibly high. Apparently something went wrong. A direct cause for the incredibly large odd ratio and very large standard error is the multicollinearity among the independent variables. We can use a program called collin to detect the multicollinearity. You can download the program from the ATS website of Stata programs for teaching and research. (search tag) All the measures in the above output are measures of the strength of the interrelationships among the variables. Two commonly used measures are tolerance (an indicator of how much collinearity that a regression analysis can tolerate) and VIF ( v ariance i nflation f actor-a n indicator of how much of the inflation of the standard error could be caused by collinearity ). The tolerance for a particular variable is 1 minus the R 2 that results from the regression of the other variables on that variable. The corresponding VIF is simply 1/tolerance. If all of the variables are orthogonal to each other, in other words, completely uncorrelated with each other, both the tolerance and VIF are 1. If a variable is very closely related to another variable(s), the tolerance goes to 0, and the variance inflation gets very large. For example, in the output above, we see that the tolerance and VIF for the variable yxfull is 0.0291 and 34.34, respectively. We can reproduce these results by doing the corresponding regression. Notice that the R 2 is .9709. Therefore, the tolerance is 1-.9709 = .0291. The VIF is 1/.0291 = 34.36 (the difference between 34.34 and 34.36 being rounding error). As a rule of thumb, a tolerance of 0.1 or less (equivalently VIF of 10 or greater) is a cause for concern. Now we have seen what tolerance and VIF measure and we have been convinced that there is a serious collinearity problem, what do we do about it? Notice that in the above regression, the variables full and yr_rnd are the only significant predictors and the coefficient for yr_rnd is very large. This is because often times when we create an interaction term, we also create some collinearity problem. This can be seen in the output of the correlation below. One way of fixing the collinearity problem is to center the variable full as shown below. We use the sum command to obtain the mean of the variable full , and then generate a new variable called fullc , which is full minus its mean. Next, we generate the interaction of yr_rnd and fullc , called yxfc . Finally, we run the logit command with fullc and yxfc as predictors instead of full and yxfull . Remember that if you use a centered variable as a predictor, you should create any necessary interaction terms using the centered version of that variable (rather than the uncentered version). We display the correlation matrix before and after the centering and notice how much change the centering has produced. (Where are these correlation matrices??) The centering of the variable full in this case has fixed the problem of collinearity, and our model fits well overall. The variable yr_rnd is no longer a significant predictor, but the interaction term between yr_rnd and full is. By being able to keep all the predictors in our model, it will be easy for us to interpret the effect of each of the predictors. This centering method is a special case of a transformation of the variables. Transformation of the variables is the best remedy for multicollinearity when it works, since we don’t lose any variables from our model. But the choice of transformation is often difficult to make, other than the straightforward ones such as centering. It would be a good choice if the transformation makes sense in terms of modeling since we can interpret the results. (What would be a good choice? Is this sentence redundant?) Other commonly suggested remedies include deleting some of the variables and increasing sample size to get more information. The first one is not always a good option, as it might lead to a misspecified model, and the second option is not always possible. We refer our readers to Berry and Feldman (1985, pp. 46-50) for more detailed discussion of remedies for collinearity. title of book or article? 3.4 Influential Observations. So far, we have seen how to detect potential problems in model building. We will focus now on detecting potential observations that have a significant impact on the model. There are several reasons that we need to detect influential observations. First, these might be data entry errors. Secondly, influential observations may be of interest by themselves for us to study. Also, influential data points may badly skew the regression estimation. (I’m not clear about what this really means??) In OLS regression, we have several types of residuals and influence measures that help us understand how each observation behaves in the model, such as if the observation is too far away from the rest of the observations, or if the observation has too much leverage on the regression line. Similar techniques have been developed for logistic regression. Pearson residuals and its standardized version is one type of residual. Pearson residuals are defined to be the standardized difference between the observed frequency and the predicted frequency. They measure the relative deviations between the observed and fitted values. Deviance residual is another type of residual. It measures the disagreement between the maxima of the observed and the fitted log likelihood functions. Since logistic regression uses the maximal likelihood principle, the goal in logistic regression is to minimize the sum of the deviance residuals. Therefore, this residual is parallel to the raw residual in OLS regression, where the goal is to minimize the sum of squared residuals. Another statistic, sometimes called the hat diagonal since technically it is the diagonal of the hat matrix, measures the leverage of an observation. It is also sometimes called the Pregibon leverage. These three statistics, Pearson residual, deviance residual and Pregibon leverage are considered to be the three basic building blocks for logistic regression diagnostics. We always want to inspect these first. They can be obtained from Stata after the logit or logistic command. A good way of looking at them is to graph them against either the predicted probabilities or simply case numbers. Let us see them in an example. We continue to use the model we built in our last section, as shown below. We’ll get both the standardized Pearson residuals and deviance residuals and plot them against the predicted probabilities. There seems to be more than just the plots of the Pearson residuals and deviance residuals below. Also, it might be helpful to have a comment in the code describing the plot, for example, * plot of Pearson residuals versus predicted probabilities. As you can see, we have produced two types of plots using these statistics: the plots of the statistics against the predicted values, and the plots of these statistics against the index id (it is therefore also called an index plot.) These two types of plots basically convey the same information. The data points seem to be more spread out on index plots, making it easier to see the index for the extreme observations. What do we see from these plots? We see some observations that are far away from most of the other observations. These are the points that need particular attention. For example, the observation with school number 1403 has a very high Pearson and deviance residual. The observed outcome hiqual is 1 but the predicted probability is very, very low (meaning that the model predicts the outcome to be 0) . This leads to large residuals. But notice that observation 1403 is not that bad in terms of leverage. That is to say, that by not including this particular observation, our logistic regression estimate won’t be too much different from the model that includes this observation. Let’s list the most outstanding observations based on the graphs. What can we find in each of the observation? What makes them stand out from the others? Observation with snum = 1402 has a large leverage value. Its percentage of fully credential teachers is 36. When we look at the distribution of full with the detail option, we realized that 36 percent is really low, since the cutoff point for the lower 5% is 61. On the other hand, its api score is fairly high with api00 = 761. This is somewhat counter to our intuition that with the low percent of fully credential teachers, that the school should be a poor performance school. Now let’s compare the logistic regression with this observation and without it to see how much impact it has on our regression coefficient estimates. We see that this single observation changes the variable yxfc from being significant to not significant, and the variable yr_rnd from not significant to almost significant. (Can we say "almost significant? Give the p-values instead? yr_rnd would be stat sig if our alpha level was .06?) This one single observation has a huge leverage on the regression model. How about the other two observations? You may want to compare the logistic regression analysis with the observation included and without the observation just as we have done here. One thing we notice is that avg_ed is 5 for observation with snum = 1819, the highest possible. This means that every students’ family has some graduate school education. This sounds too good to be true. This may well be a data entry error. This may well be the reason why this observation stands out so much from the others. This leads us to inspect our data set more carefully. We can list all the observations with perfect avg_ed . There are three schools with a perfect avg_ed score. It is very unlikely that the average education for any of the schools would reach a perfect score of 5. The observation with snum = 3098 and the observation with snum = 1819 seem more unlikely than the observation with snum = 1081, though, since their api scores are very low. In any case, it seems that we should double check the data entry here. What do we want to do with these observations? It really depends. Sometimes, we may be able to go back to correct the data entry error. Sometimes we may have to exclude them. Regression diagnostics can help us to find these problems, but they don’t tell us exactly what to do about them. So far, we have seen the basic three diagnostic statistics: the Pearson residual, the deviance residual and the leverage (the hat value). They are the basic building blocks in logistic regression diagnostics. There are other diagnostic statistics that are used for different purposes. One important aspect of diagnostics is to identify observations with substantial impact on either the chi-square fit statistic or the deviance statistic. For example, we may want to know how much change in either the chi-square fit statistic or in the deviance statistic a single observation would cause. This leads to the dx2 and dd statistics. dx2 stands for the difference of chi-squares and dd stands for the difference of deviances. In Stata, we can simply use the predict command after the logit or logistic command to create these variables, as shown below. We can then visually inspect them. It is worth noticing that, first of all, these statistics are only one-step approximation of the difference, not quite the exact difference, since it would be computationally too extensive to obtain exact difference for every observation. (I’m not clear about what a "one-step" approximation is?) Secondly, Stata does all the diagnostic statistics for logistic regression using covariate patterns. Each observation will have exactly the same diagnostic statistics as all of the other observations in the same covariate pattern. Perhaps give the variables names that are different than the options, just to avoid confusion. The observation with snum =1403 is obviously substantial in terms of both chi-square fit and the deviance fit statistic. For example, in the first plot, we see that dx2 is about 216 for this observation and below 100 for the rest of the observations. This means that when this observation is excluded from our analysis, the Pearson chi-square fit statistic will decrease by roughly 216. In the second plot, the observation with snum = 1403 will increase the deviance about 11. We can run two analysis and compare their Pearson chi-squares to see if this is the case. It is not precisely 216. (Umm, in most cases, 171 isn’t considered to be anywhere near 216. Is this really a good example?) This is because of one-step approximation. We can also look at the difference between deviances in a same way. Since the deviance is simply 2 times the log likelihood, we can compute the difference of deviances as 2 times the difference in log likelihoods. When could it happen that an observation has great impact on fit statistics, but not too much impact on parameter estimates? This is actually the case for the observation with snum =1403, because its leverage is not very large. Notice that the observation with snum =1403 has a fairly large residual. This means that the values for the independent variables of the observation are not in an extreme region, but the observed outcome for this point is very different from the predicted value. From the list of the observation below, we see that the percent of students receiving free or reduced-priced meals is about 100 percent, the avg_ed score is 2.19, and it is a year-around school. All things considered, we wouldn’t expect that this school is a high performance school. But its api score is 808, which is very high. With information on school number and district number, we can find out to which school this observation corresponds. It turns out that this school is Kelso Elementary School in Inglewood that has been doing remarkably well. One can easily find many interesting articles about the school. Therefore, regression diagnostics help us to recognize those schools that are of interest to study by themselves. The last type of diagnostic statistics is related to coefficient sensitivity. It concerns how much impact each observation has on each parameter estimate. Similar to OLS regression, we also have dfbeta’s for logistic regression. A program called ldfbeta is available for download (search tag) . Like other diagnostic statistics for logistic regression, ldfbeta also uses one-step approximation. Unlike other logistic regression diagnostics in Stata, ldfbeta is at the individual observation level, instead of at the covariate pattern level. After either the logit or logistic command, we can simply issue the ldfbeta command. It can be used without any arguments, and in that case, dfbeta is calculated for each predictor. It will take some time since it is somewhat computationally intensive. Or we can specify a variable, as shown below. For example, suppose that we want to know how each individual observation affects the parameter estimate for the variable meals . There is another statistic called Pregibon’s dbeta which is provides summary information of influence on parameter estimates of each individual observation (more precisely each covariate pattern). dbeta is very similar to Cook’s D in ordinary linear regression. This is more commonly used since it is much less computationally intensive. We can obtain dbeta using the predict command after the logit or logistic command. We have seen quite a few logistic regression diagnostic statistics. Now how large does each one have to be, to be considered influential? First of all, we always have to make our judgment based on our theory and our analysis. Secondly, there are some rule-of-thumb cutoffs when the sample size is large. These are shown below. When the sample size is large, the asymptotic distribution of some of the measures would follow some standard distribution. That is why we have these cutoff values, and why they only apply when the sample size is large enough. Usually, we would look at the relative magnitude of a statistic an observation has compared to others. That is, we look for data points that are farther away from most of the data points.