Experimental Design: Randomness & Signal/Noise

Signal/Noise

What are the different major types of research designs? We can classify designs into a simple threefold classification by asking some key questions. First, does the design use random assignment to groups? [Don't forget that random assignment is not the same thing as random selection of a sample from a population!] If random assignment is used, we call the design a randomized experiment or true experiment. If random assignment is not used, then we have to ask a second question: Does the design useeither multiple groups or multiple waves of measurement? If the answer is yes, we would label it a quasi-experimental design. If no, we would call it a non-experimental design. This threefold classification is especially useful for describing the design with respect to internal validity. A randomized experiment generally is the strongest of the three designs when your interest is in establishing a cause-effect relationship. A non-experiment is generally the weakest in this respect. I have to hasten to add here, that I don't mean that a non-experiment is the weakest of the the three designs overall, but only with respect to internal validity or causal assessment. In fact, the simplest form of non-experiment is a one-shot survey design that consists of nothing but a single observation O. This is probably one of the most common forms of research and, for some research questions -- especially descriptive ones -- is clearly a strong design. When I say that the non-experiment is the weakest with respect to internal validity, all I mean is that it isn't a particularly good method for assessing the cause-effect relationship that you think might exist between a program and its outcomes.

To illustrate the different types of designs, consider one of each in design notation. The first design is a posttest-only randomized experiment. You can tell it's a randomized experiment because it has an R at the beginning of each line, indicating random assignment. The second design is a pre-post nonequivalent groups quasi-experiment. We know it's not a randomized experiment because random assignment wasn't used. And we know it's not a non-experiment because there are both multiple groups and multiple waves of measurement. That means it must be a quasi-experiment. We add the label "nonequivalent" because in this design we do not explicitly control the assignment and the groups may be nonequivalent or not similar to each other (see nonequivalent group designs). Finally, we show a posttest-only nonexperimental design. You might use this design if you want to study the effects of a natural disaster like a flood or tornado and you want to do so by interviewing survivors. Notice that in this design, you don't have a comparison group (e.g., interview in a town down the road the road that didn't have the tornado to see what differences the tornado caused) and you don't have multiple waves of measurement (e.g., a pre-tornado level of how people in the ravaged town were doing before the disaster). Does it make sense to do the non-experimental study? Of course! You could gain lots of valuable information by well-conducted post-disaster interviews. But you may have a hard time establishing which of the things you observed are due to the disaster rather than to other factors like the peculiarities of the town or pre-disaster characteristics.

http://www.socialresearchmethods.net/kb/expclass.php