PPA 696 RESEARCH METHODS

QUASI-EXPERIMENTAL RESEARCH DESIGNS

Comparison Group Design
Threats to Internal Validity
Threats to External Validity
Interrupted Time Series Design
Interrupted Time Series with Comparison Group
Quasi-experimental designs came about because of:
1) difficulty of applying the classical natural science method to the social sciences
2) overemphasis on theory testing and development
3) high cost of classic natural science methods
4) development of new statistical tools that allowed for statistical control
 
There are several types of quasi-experimental designs, including:
time series design
equivalent time series samples
equivalent samples materials design
non-equivalent control group
counterbalanced designs
separate sample pre-test/post-test
separate sample pre-test/post-test control group
multiple time series design
institutional cycle design
regression-discontinuity design

Comparison Group Pre-test/Post-test Design

In a quasi-experimental design, the research substitutes statistical "controls" for the absence of physical control of the experimental situation.  The most common quasi-experimental design is the Comparison Group Pre-test/Post-test Design.  This design is the same as the classic controlled experimental design except that the subjects cannot be randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment.  In other words, participants do not all have the same chance of being in the control or the experimental groups, or of receiving or not receiving the treatment.

This design can be diagrammed as follows:

    O1    X    O2
    O1            O2

This can be illustrated by the following research study to determine whether a home weatherization program for low income families reduced home energy consumption.
 
Were homes weatherized? Average energy consumption per home (first measurement) Was home weatherized? Average energy consumption per home (second measurement)
Yes BTUs used per month; 
BTUs per sf per degree day; Cost per degree day		 Home was weatherized BTUs used per month; 
BTUs per sf per degree day; Cost per degree day
No (on the waiting list) BTUs used per month; 
BTUs per sf per degree day; Cost per degree day 		 BTUs used per month; 
BTUs per sf per degree day; Cost per degree day
The weatherized homes saved an average of 10.95% of the BTUs used per month before weatherization; saved an average of 1.68 BTUs per degree day per square foot; and saved an average of $.006 per degree day.

The non-weatherized homes used more BTUs per month (+2.48% more) over the same time period compared to the weatherized homes.  The non-weatherized homes also used more BTUs per degree day per square foot; and saved less money (an average of $.001 per degree day).

The difference between the change in the experimental group (down 10.95%) and the change in the control group (up 2.48%) is +13.43%.   This is the amount of savings that can be attributed to the weatherization program.

The quasi-experimental design is not as strong in controlling for threats to the internal and external validity of the study as the true controlled experimental design.
 

Controlling for Threats to Internal Validity

1) History:  did some other current event effect the change in the dependent variable?  No, because both groups experienced the same current events.

2) Maturation:  were changes in the dependent variable due to normal developmental processes?  No, because both groups experienced the same developmental processes.

3) Statistical Regression:  did subjects come from low or high performing groups?  Both groups were low income families but not necessarily high energy users.

4) Selection:  were the subjects self-selected into experimental and control groups, which could affect the dependent variable?  No, both groups had applied to the weatherization program, and had similar floor space, number of occupants, and percent owner-occupied.

5) Experimental Mortality:  did some subjects drop out?  did this affect the results?  There were some homes eliminated because of moves, being away from home, or unable to get accurate fuel records.

6) Testing:  Did the pre-test affect the scores on the post-test?  No, both groups supplied energy records.

7) Instrumentation:  Did the measurement method change during the research?  No, both groups supplied energy records.

8) Design contamination:  did the control group find out about the experimental treatment?  did either group have a reason to want to make the research succeed or fail?  None noted.
 

Controlling for Threats to External Validity

1) Unique program features:  None noted.

2)  Effects of Selection:  None noted.

3) Effects of Setting:  Study was done at one location in Minnesota;

4) Effects of History:  Study was done during a period of high energy costs;

5) Effects of Testing:  None noted.

6)  Reactive effects of experimental arrangements:  Need to replicate the findings in other locations and other time periods.

Quasi-experimental designs may be weak in controlling for threats to internal validity, but can be quite strong in controlling for threats to external validity.  It may be difficult to control which police are switched to a four-day ten-hour shift, or which children are given a new method of learning a foreign language.  However, since the research takes place in a natural setting, it may have wide applicability to other similar settings.
 

Interrupted Time Series Design

This design uses several waves of observation before and after the introduction of the independent (treatment) variable X.  It is diagrammed as follows:

O1    O2    O3    O4    X    O5    O6    O7    O8

This may be illustrated by a study designed to test whether the implementation of a crackdown on speeding in a given state reduces the traffic fatality rate in that state.
 
T-4 T-3 T-2 T-1 X T+1 T+2 T+3 T+4
Fatality Rate 
(4 yrs before)		  Fatality Rate 
 (3 yrs before)		 Fatality Rate 
 (2 yrs before)		 Fatality Rate 
(Year before) 		 Crack 
-down		 Fatality Rate 
(Year after )		 Fatality Rate 
(2 yrs after) 		 Fatality Rate 
(3 yrs after)		 Fatality Rate 
(4 yrs after)
This type of design works best if the treatment (independent variable) is expected to have an immediate, marked effect, and if the treatment is introduced (implemented) all at once in all relevant situations.  However, the results may be difficult to interpret, especially if no statistically significant differences are found.  Researchers must often collect qualitative data to supplement and interpret the quantitative measurements.
 

The design is not particularly strong at controlling for threats to internal validity:
1) History:  did some other current event effect the change in the dependent variable?  Researcher must gather qualitative data on possible events that could have affected the fatality rate.
2) Maturation:  were changes in the dependent variable due to normal developmental processes?  No.
3) Statistical Regression:  did subjects come from low or high performing groups?  Statistical analysis is used to determine whether changes are due to statistical regression or the independent variable.
4) Selection:  were the subjects self-selected into experimental and control groups, which could affect the dependent variable?  Researcher must determine whether there were any major changes in the population between the before and after measures.
5) Experimental Mortality:  did some subjects drop out?  did this affect the results?  Researcher must check whether some of the population dropped out after the implementation of the crackdown.
6) Testing:  Did the pre-test affect the scores on the post-test?  No effects.
7) Instrumentation:  Did the measurement method change during the research?  Researcher must ensure that fatalities were measured in the same way in all the years considered.
8) Design contamination:  did the control group find out about the experimental treatment?  did either group have a reason to want to make the research succeed or fail?  None noted.
Nor is this design strong on controlling for threats to external validity.
All the possible threats must be considered, in particular any interaction between the selection of this population and the particular treatment (crackdown) applied.  These concerns include a) unique program features; b) effects of selection; c) effects of setting; d) effects of history; e) effects of testing; f) reactive effects of experimental arrangements.

Interrupted Time Series Design with Comparison Group

The addition of a second time series for a comparison group helps to provide a check on some of the threats to validity of the Single Interrupted Time Series Design discussed above, especially history.

This design uses several waves of observation in both groups (treatment and comparison groups) before and after the introduction of the independent variable X in the treatment group.  It is diagrammed as follows:

State A:    O1    O2    O3    O4    X    O5    O6    O7    O8
State B:    O1    O2    O3    O4     -     O5    O6    O7    O8

This may be illustrated by a study to assess the effect of a crackdown on drunk driving on automobile fatalities in one state, compared to automobile fatalities in another state without a similar crackdown.
 
State  T-3 T-2 T-1 X T+1 T+2 T+3
A Fatality 
Rate		 Fatality 
Rate		 Fatality 
Rate		 Crack 
down		 Fatality 
Rate		 Fatality 
Rate		 Fatality 
Rate
B Fatality 
Rate		 Fatality 
Rate		 Fatality 
Rate		 - Fatality 
Rate		 Fatality 
Rate		 Fatality 
Rate