WEBVTT Kind: captions Language: en 00:00:00.930 --> 00:00:04.340 In a typical research paper that  uses multiple regression analysis, 00:00:04.573 --> 00:00:07.590 we do many different regression models. 00:00:07.590 --> 00:00:09.210 And the reason for that is that 00:00:09.210 --> 00:00:10.830 we want to do model comparisons. 00:00:10.830 --> 00:00:12.390 Now we will take a look at, 00:00:12.390 --> 00:00:14.610 why we compare models and how we do that? 00:00:15.427 --> 00:00:16.680 In Hekman's paper, 00:00:16.680 --> 00:00:18.480 which is our example for this video, 00:00:18.480 --> 00:00:21.690 we will be focusing on their first study. 00:00:21.690 --> 00:00:26.207 So they say that they used hierarchical  moderated regression analysis. 00:00:26.207 --> 00:00:27.570 So what does that mean? 00:00:27.570 --> 00:00:29.760 The hierarchical here is the key term, 00:00:29.760 --> 00:00:33.120 it simply means that you're  estimating multiple models. 00:00:33.120 --> 00:00:36.300 Start with a simple one then add more variables, 00:00:36.300 --> 00:00:38.850 compare, add more variables and compare. 00:00:39.067 --> 00:00:42.067 The moderated part here means that 00:00:42.067 --> 00:00:43.890 they have interaction terms in their model. 00:00:43.890 --> 00:00:45.840 They could just as well have said that 00:00:45.840 --> 00:00:47.100 they used regression analysis, 00:00:47.100 --> 00:00:50.040 because we use regression analysis nearly always 00:00:50.040 --> 00:00:51.480 in a hierarchical way. 00:00:51.480 --> 00:00:54.990 And it's obvious based on  the regression results that 00:00:54.990 --> 00:00:56.670 they contain interaction terms. 00:00:56.670 --> 00:01:00.780 So this is a bit unnecessary,  complicated way of saying, 00:01:00.780 --> 00:01:03.660 we did regression, we estimated multiple models. 00:01:04.160 --> 00:01:06.180 Now let's look at the actual models, 00:01:06.180 --> 00:01:07.020 and the modeling results, 00:01:07.020 --> 00:01:09.420 and the logic for multiple model comparisons. 00:01:10.520 --> 00:01:14.030 They say that they entered in the first model, 00:01:14.030 --> 00:01:16.230 they have the control variables only here. 00:01:16.797 --> 00:01:21.750 And in the second model, they included  some of the interesting variables. 00:01:21.750 --> 00:01:24.240 So we'll be focusing on the first two models, 00:01:24.240 --> 00:01:25.530 Model 1 and Model 2. 00:01:25.747 --> 00:01:27.817 Model 1 has control variables only, 00:01:27.817 --> 00:01:31.440 Model 2 is controls and  some interesting variables. 00:01:31.773 --> 00:01:33.950 The logic in model comparison, 00:01:33.950 --> 00:01:35.610 when we do that kind of comparison, 00:01:35.610 --> 00:01:36.687 is to ask the question, 00:01:36.687 --> 00:01:40.130 do the interesting variables  and the controls together 00:01:40.130 --> 00:01:43.350 explain the dependent variable  more than the controls only. 00:01:43.350 --> 00:01:47.010 If the control variables and the  interesting variables together 00:01:47.010 --> 00:01:48.990 don't explain the data more  than the controls only, 00:01:48.990 --> 00:01:54.060 then we conclude that the interesting  variables are not very useful in 00:01:54.060 --> 00:01:55.740 explaining the dependent variable, 00:01:55.740 --> 00:01:58.410 and we can conclude that they  don't really have an effect. 00:01:59.060 --> 00:02:02.730 How we do a model comparison is that  we compare the R-squared statistic. 00:02:02.730 --> 00:02:06.740 So here they have the adjusted  R-squared and the actual R-squared. 00:02:07.240 --> 00:02:08.610 The model comparison, 00:02:08.610 --> 00:02:10.847 if we just want to assess the magnitude, 00:02:10.847 --> 00:02:13.770 how much better the Model 2 is? 00:02:13.770 --> 00:02:18.210 In small samples, the more appropriate  statistic is the adjusted R-squared. 00:02:18.210 --> 00:02:23.577 However, the adjusted R-squared statistic  doesn't really have a well-known test. 00:02:23.577 --> 00:02:26.880 So instead of looking at the adjusted R-squared, 00:02:26.880 --> 00:02:29.220 we test the R-squared difference. 00:02:29.570 --> 00:02:31.813 They present the R-squared difference here. 00:02:31.813 --> 00:02:35.730 So this is the difference between the first  model R-squared and the second model R-squared, 00:02:35.730 --> 00:02:38.610 and they have some stars here. 00:02:39.027 --> 00:02:41.007 So the important question is, 00:02:41.007 --> 00:02:44.610 does the second model explain the  data better than the first model? 00:02:44.610 --> 00:02:47.760 The adjusted R-squared difference is 4, 00:02:47.760 --> 00:02:53.580 the actual R-squared difference is 7 or 0.07 %, 00:02:53.580 --> 00:02:58.800 so the interesting variables explain the data  a bit more than the control variable only. 00:02:59.150 --> 00:03:03.800 Now we will be focusing on  these test statistics here. 00:03:03.800 --> 00:03:05.220 So where do these stars come from? 00:03:05.753 --> 00:03:08.850 These stars come from an F test that 00:03:08.850 --> 00:03:11.040 tests the null hypothesis that 00:03:11.040 --> 00:03:17.190 all the regression coefficients for  every variable added to this model are zero. 00:03:17.773 --> 00:03:20.310 We look at the logic of the test now. 00:03:21.127 --> 00:03:25.606 So the idea of the F test  between the first two models is 00:03:25.606 --> 00:03:28.410 that it is a nested model comparison test. 00:03:28.710 --> 00:03:31.290 So one model is nested in another, 00:03:31.290 --> 00:03:34.130 that means that one model is  a special case of another. 00:03:34.547 --> 00:03:39.780 So in this case Model 2 is the  unrestricted model or unconstraint model, 00:03:39.780 --> 00:03:44.040 Model 1 is the restricted  model or constraint model. 00:03:45.000 --> 00:03:50.070 So, why can we say that Model 1 is a special  case of a more general model, Model 2? 00:03:50.070 --> 00:03:52.360 The reason is that Model 1, 00:03:52.593 --> 00:03:55.353 which leaves out these variables, 00:03:55.353 --> 00:03:58.710 is the same model as Model 2, 00:03:58.710 --> 00:04:02.610 except that the effects of these  variables are constrained to be zero. 00:04:02.827 --> 00:04:05.047 So by leaving out variables, 00:04:05.047 --> 00:04:09.570 we constrain the regression coefficient  of that variable to be zero. 00:04:09.570 --> 00:04:10.800 And that's the reason, 00:04:10.800 --> 00:04:15.750 why we say that this model is a  constrained version of that model here. 00:04:15.750 --> 00:04:18.960 The effects of the last three  variables are freely estimated, 00:04:18.960 --> 00:04:21.360 here they are constrained to be 0. 00:04:22.277 --> 00:04:24.930 So how do we test these differences, 00:04:24.930 --> 00:04:30.840 whether the difference in R-squared is more  than what we can expect by a chance of only? 00:04:31.123 --> 00:04:34.123 Remember that every time we  add something to the model, 00:04:34.123 --> 00:04:35.803 the R-squared can only go up. 00:04:35.803 --> 00:04:37.660 It can stay the same or go up, 00:04:37.810 --> 00:04:39.400 typically it goes up. 00:04:39.400 --> 00:04:43.020 So is that increase in R-squared  statistically significant? 00:04:43.287 --> 00:04:44.520 To answer that question, 00:04:44.520 --> 00:04:45.870 we do the t test. 00:04:45.870 --> 00:04:48.320 And let's do a t test by hand now. 00:04:48.320 --> 00:04:52.627 We need to first have the degrees  of freedom for the first two models, 00:04:52.627 --> 00:04:54.810 to do the F test. 00:04:55.210 --> 00:04:57.760 The degrees of freedom for the regression model is 00:04:57.760 --> 00:05:02.377 n, the sample size, minus k, the  number of estimated parameters, 00:05:02.377 --> 00:05:06.453 or regression coefficients, or  number of variables in the model, 00:05:06.453 --> 00:05:08.360 minus 1 for the intercept. 00:05:08.360 --> 00:05:13.410 So we have a sample that provides  us with 113 units of information, 00:05:13.410 --> 00:05:15.870 we estimate for the first model, 00:05:15.870 --> 00:05:17.910 effects of 15 variables, 00:05:17.910 --> 00:05:19.320 we estimate the intercept, 00:05:19.320 --> 00:05:25.560 so we have 97 degrees of freedom  remaining for the restricted model. 00:05:26.210 --> 00:05:30.730 In the unrestricted model, we  estimate three more things, 00:05:30.730 --> 00:05:36.060 so it's 113 - 18 - 1 = 94 degrees of freedom, 00:05:36.060 --> 00:05:36.887 for that model. 00:05:37.037 --> 00:05:39.570 So these degrees of freedom  calculations are pretty simple, 00:05:39.570 --> 00:05:41.730 it's just basic subtraction. 00:05:42.730 --> 00:05:45.360 Now we need to have an F statistic as well. 00:05:45.360 --> 00:05:51.370 And the F statistic can be are  defined based on the R-squared values. 00:05:51.370 --> 00:05:56.760 So it's the R-squared difference divided  by the degrees of freedom difference, 00:05:56.760 --> 00:05:59.280 divided by that thing there. 00:05:59.930 --> 00:06:01.730 So that's the F statistic, 00:06:01.730 --> 00:06:04.923 your econometrics textbook will explain, 00:06:05.000 --> 00:06:06.770 where that comes from. 00:06:06.770 --> 00:06:10.000 But importantly we are here interested in, 00:06:10.000 --> 00:06:15.060 how much the R-squared increases  per the degrees of freedom consumed, 00:06:15.060 --> 00:06:16.680 when we estimate the model. 00:06:16.680 --> 00:06:22.200 Quite often we compare increased  explanation against increased complexity, 00:06:22.200 --> 00:06:24.150 that's a fairly general comparison, 00:06:24.150 --> 00:06:26.490 which we use in multiple different tests. 00:06:26.907 --> 00:06:28.737 So we do that, 00:06:28.737 --> 00:06:29.947 we plug in the numbers, 00:06:29.947 --> 00:06:33.473 we get the result of 3.22. 00:06:33.473 --> 00:06:36.323 We compare that against the proper F distribution, 00:06:36.573 --> 00:06:39.810 we get a p-value of 0.026, 00:06:39.810 --> 00:06:41.820 which has one significance star. 00:06:42.303 --> 00:06:45.153 So they presented two stars, 00:06:45.153 --> 00:06:48.063 The reason, I have no idea, 00:06:48.063 --> 00:06:52.260 but I've done this example in multiple classes, 00:06:52.260 --> 00:06:56.040 over multiple years and I don't  know why this is different. 00:06:56.040 --> 00:06:58.350 It could be that there's a typo in the paper, 00:06:58.350 --> 00:07:03.080 or it could be, that's probably the case, 00:07:03.080 --> 00:07:05.390 because this kind of difference, 00:07:05.773 --> 00:07:10.170 getting that because of rounding error  in the R-squared is quite unlikely. 00:07:10.487 --> 00:07:13.067 So that's the idea of F test, 00:07:13.067 --> 00:07:15.313 you take a constraint model, 00:07:15.363 --> 00:07:17.040 and you take an unconstraint model, 00:07:17.040 --> 00:07:21.240 you calculate the difference per  the degrees of freedom difference, 00:07:21.240 --> 00:07:23.880 you scale it with this thing, 00:07:23.880 --> 00:07:28.830 and then you will get a test statistic that  you compare against the F distribution. 00:07:29.297 --> 00:07:32.327 In more complicated models 00:07:32.327 --> 00:07:36.510 for which we don't know, how  they behave in small samples, 00:07:36.510 --> 00:07:39.480 we use the chi-square distribution  instead of the F distribution. 00:07:39.480 --> 00:07:41.670 But the principle is the same. 00:07:42.453 --> 00:07:46.083 In practice, your software will  do the calculation for you, 00:07:46.083 --> 00:07:49.260 but it is useful to understand that  these calculations are not complicated, 00:07:49.260 --> 00:07:53.760 and have a little bit of understanding  of the logic behind the calculations.