WEBVTT WEBVTT Kind: captions Language: en 00:00:00.399 --> 00:00:04.550 Reliability and validity are two important qualities of research. 00:00:04.550 --> 00:00:06.350 They also apply to measurement. 00:00:06.350 --> 00:00:13.049 The idea of reliability is that your study is reliable if repeating the same study again 00:00:13.049 --> 00:00:14.959 gives you same result. 00:00:14.959 --> 00:00:21.050 In quantitative research the analysis is done by a computer and ideally is so well documented 00:00:21.050 --> 00:00:26.470 that if your do the study again you do the exact same calculations. 00:00:26.470 --> 00:00:33.940 Because computer doesn't introduce any random error to your result then the only reason 00:00:33.940 --> 00:00:39.760 why your study could be unreliable is because your measures could be unreliably. 00:00:39.760 --> 00:00:47.690 For that reason reliability in quantitative research is typically an attribute that is 00:00:47.690 --> 00:00:51.230 associated with the measurement procedures and nothing else. 00:00:51.230 --> 00:00:58.080 Then validity is a more complicated concept and it can be thought of as four different 00:00:58.080 --> 00:00:59.440 levels of validity. 00:00:59.440 --> 00:01:04.580 First of all you have measurement validity which refers to whether your variables actually 00:01:04.580 --> 00:01:07.690 measure the things that they are supposed to measure. 00:01:07.690 --> 00:01:14.830 Then you have statistical conclusion validity which means that you have identified the correct 00:01:14.830 --> 00:01:20.320 associations or correct differences in the population so our statistics are correct. 00:01:20.320 --> 00:01:26.980 Then we have internal validity which means that the theory is correct or the association 00:01:26.980 --> 00:01:30.310 that we claim are causal are actually causal. 00:01:30.310 --> 00:01:35.390 So that's whether the causal inference part has been done correctly. 00:01:35.390 --> 00:01:41.300 Then external validity is simply a whether the study generalizes to other populations. 00:01:41.300 --> 00:01:46.979 That's a mostly about generalibility and we cannot really assist that statistically. 00:01:46.979 --> 00:01:48.650 That's theoretical argument. 00:01:48.650 --> 00:01:54.330 The two important qualities of measurement are reliability and measurement validity. 00:01:54.330 --> 00:02:00.710 To understand what those two concepts actually mean it's useful to take look at these target 00:02:00.710 --> 00:02:01.710 diagrams. 00:02:01.710 --> 00:02:05.520 This is a target that somebody is shooting. 00:02:05.520 --> 00:02:12.040 Here we have only a small amount of dispersion in the hits but the sight are off. 00:02:12.040 --> 00:02:17.599 So this shooter is very precise but he's not hitting a target. 00:02:17.599 --> 00:02:22.930 This is reliable but not valid measurement. 00:02:22.930 --> 00:02:29.879 Then this another shooter which is a not very precise so the hits are all over. 00:02:29.879 --> 00:02:31.650 But the sights are correct. 00:02:31.650 --> 00:02:33.969 On average he's hitting the target. 00:02:33.969 --> 00:02:39.790 So this is not reliable but valid. 00:02:39.790 --> 00:02:48.390 There is some disagreement in the literature whether you can have validity without reliability. 00:02:48.390 --> 00:02:55.109 Let's postbone for that for awhile but it's at this point important to understand what 00:02:55.109 --> 00:02:56.239 these concepts are. 00:02:56.239 --> 00:03:02.650 So the validity is whether the sights are correct and then reliability is whether the 00:03:02.650 --> 00:03:05.760 shooter is getting the same point all the time. 00:03:05.760 --> 00:03:10.079 So are you hitting the same spot - are you hitting the bulls eye. 00:03:10.079 --> 00:03:13.139 Which one of these is more serious problem? 00:03:13.139 --> 00:03:21.249 You can think of it as would it be safer to stand here in front of this target or would 00:03:21.249 --> 00:03:24.529 it be safer to stand here in front of this target. 00:03:24.529 --> 00:03:29.510 If my head was here on the bulls eye this guy would eventually kill me. 00:03:29.510 --> 00:03:36.370 If my head was here then I would be pretty safe because this guy would never hit me. 00:03:36.370 --> 00:03:42.260 So the lack of validity is more problematic than lack of reliability because an invalid 00:03:42.260 --> 00:03:48.299 measure is always incorrect and unreliable measure can sometimes provide you the correct 00:03:48.299 --> 00:03:50.430 value if it's valid. 00:03:50.430 --> 00:03:59.420 The idea of no validity without reliability basically refers to - if you are just looking 00:03:59.420 --> 00:04:03.959 at one of these hits. 00:04:03.959 --> 00:04:12.199 These hits individually are not very valuable because they are so disperse they are so unreliable. 00:04:12.199 --> 00:04:19.030 So in that sense if you just look at one hit one the target it's unlikely to be close to 00:04:19.030 --> 00:04:20.030 the bulls eye. 00:04:20.030 --> 00:04:24.819 So that's the argument for no validity without reliability. 00:04:24.819 --> 00:04:28.319 But if you look at these as a collection. 00:04:28.319 --> 00:04:34.930 Let's say these are five repeated studies and then after those five studies have been 00:04:34.930 --> 00:04:39.250 done then we are trying to aggregate those somehow. 00:04:39.250 --> 00:04:46.910 Then as a collection these five hits valid because they are on average on the bulls eye. 00:04:46.910 --> 00:04:55.240 So reliability is a problem if you just do an individual measurement or individual study. 00:04:55.240 --> 00:05:00.050 Reliability can be less of a problem if you get to do multiple measurements. 00:05:00.050 --> 00:05:05.310 And multiple unreliable measurements actually could produce you valid inference.