PPA 696 RESEARCH METHODS

Pre-experimental Design
Cross-Sectional Design
Longitudinal Designs
Time Series Design
Panel Design
Case Study Design
Control over Sources of Invalidity
How to Improve Designs for Description

PRE-EXPERIMENTAL DESIGNS

In general, a research design is like a blueprint for the research.  A research design is a plan that guides the decision as to:
-when and how often to collect data
-what data to gather and from whom
-how to analyze the data

More specifically, a research design refers to the type of study that will be conducted, whether it will be pre-experimental, quasi-experimental, or true experimental.

Pre-experimental designs include:
-case study design
-one group pre-test/post-test design
-static group comparison design (cross-sectional study)
Quasi-experimental designs include:
-time series design (may include panel design)
-equivalent time samples design
-equivalent materials design
-nonequivalent control group (comparison group) design
-counterbalanced design
-separate sample pre-test/post-test design
-separate sample pre-test/post-test control group design
-multiple time-series design
-recurrent institutional cycle design
-regression/discontinuity analysis
True experimental designs include:
-pre-test/post-test control group design
-Solomon four-group design
-post-test only control group design
Research Methodology concerns how the design is implemented, how the research is carried out.  The methodology employed often determines the quality of the data set produced.
Methodology is concerned with:
-when and how often to collect data
-construction of data collection measures
-identification of the sample or test population
-choice of strategy for contacting subjects
-selection of statistical tools
-presentation of the findings

Pre-Experimental Designs for Description

Descriptive research can provide data for monitoring and evaluating policies and programs.  These designs are concerned with how to answer questions such as:
-how many?
-how much?
-how efficient?
-how effective?
-how adequate?
Cross-Sectional Design
A cross-sectional design is used for research that collects data on relevant variables one time only from a variety of people, subjects, or phenomena.  The data are collected all at the same time (or within a short time frame).

A cross-sectional designs provides a snapshot of the variables included in the study, at one particular point in time.  It may reveal how those variables are represented in a cross-section of a population.  Cross-sectional designs generally use survey techniques to gather data, for example, the U.S. Census.

 Advantages and Disadvantages of Cross-Sectional Designs
Advantages  Disadvantages
data on many variables increased chances of error
data from a large number of subjects increased cost with more subjects
data from dispersed subjects increased cost with each location
data on attitudes and behaviors cannot measure change
answers questions on who, what, when, where cannot establish cause and effect
good for exploratory research no control of independent variable
generates hypotheses for future research difficult to rule out rival hypotheses
data useful to many different researchers static, time bound
 
Longitudinal Designs

A longitudinal design collects data over long periods of time.  Measurements are taken on each variable over two or more distinct time periods.  This allows the researcher to measure change in variables over time.  There are two different types of longitudinal designs:  time series and panel.

A Time Series Design collects data on the same variable at regular intervals (weeks, months, years, etc.) in the form of aggregate measures of a population.  For example, the Consumer Price Index (CPI), the FBI Uniform Crime Rate, unemployment rates, poverty rates, etc.

Time series designs are useful for:
-establishing a baseline measure
-describing changes over time
-keeping track of trends
-forecasting future (short term) trends
Time series data are nearly always presented in the form of a chart or graph.  The horizontal (or x) axis is divided into time intervals, and the vertical (y) axis shows the values of the dependent variable as they fluctuate over time.
 
Researchers inspect a time series graph to look for four types of patterns:
-long term trends (increases or decreases over the whole time span);
-cyclical variations (short-term, valley-to-valley or peak-to-peak cycles);
-seasonal variations (due holidays or weather);
-irregular fluctuations (none of the above).
 Advantages and Disadvantages of Longitudinal Designs
Advantages   Disadvantages
data easy to collect  data collection method may change over time
easy to present in graphs difficult to show more than one variable at a time
easy to interpret   needs qualitative research to explain fluctuations
can forecast short term trends assumes present trends will continue unchanged

Panel Designs collect repeated measurements from the same people or subjects over time.  Panel studies reveal changes at the individual level, for example, when a particular person was employed or unemployed, or when they were on or off of welfare.

Panel data can show different patterns from time series data.  For example, about 5% of the elderly are institutionalized at any one time, but it is not always the same people.  So elderly people have a 20% chance of being institutionalized at some point.
 
 Advantages and Disadvantages of Panel Designs
Advantages Disadvantages
reveals individual level changes difficult to obtain initial sample of subjects
 establishes time order of variables difficult to keep the same subjects over time
can show how relationships emerge  repeated measures may influence subjects behavior
 
 
Case Study Design

Case studies examine some phenomenon in depth, e.g., people, programs, policies, decisions, organizations, etc.  Case studies are useful for learning about:
-policies or programs with remarkable successes
-policies or programs with ambiguous or unexpected outcomes
-situations where actors have discretionary behavior (e.g., street-level bureaucrats)
A case study weave together data from documents, archives, interviews, participation, observation, artifacts, etc.  It attempts to document not only the "what" but also the "why."
 
Advantages and Disadvantages of Case Study Design
Advantages  Disadvantages
includes data from multiple perspectives limited to contemporary phenomena
combines data from different sources  need direct access to subjects
need diverse sources of information
need skills in many techniques
can be an intense experience
difficult to replicate findings
insiders can be biased
outsiders can be naive
difficult to draw boundaries
 
Focus Groups
Focus groups are a method of group interviewing for obtaining qualitative data.  It is not so much a research design as a data collection method.  More will be said about focus groups in the section on data collection.

Meta-Analysis
A meta-analysis is a quantitative analysis of a sample of existing research studies on a particular topic.  It is used to draw conclusions about the topic from a range of studies, for example, identify aspects of a program associated with program success.  A meta-analysis may also generate new hypotheses for future research.

Problems with doing a meta-analysis are:
-locating suitable studies
-studies may not all have the same dependent variable
-only findings of statistical significance are usually published
-different studies may have many dissimilar aspects making them difficult to compare

Control Over Sources of Invalidity in Designs for Description 

Source Case study Cross-section Panel Time-series
Internal Validity
History weak weak weak weak
Maturation weak weak strong strong
Testing strong strong
Instrumentation weak ? ?
Regression weak strong strong
Selection weak weak weak strong
Mortality weak weak strong
Design Contamination ? strong
External Validity
Testing weak weak
Selection weak weak weak ?
Experimental Arrangements ? ?
Multiple Interactions
 
 

How to Improve the Validity of Designs for Description

1.  Case study design
 
X O2
 

2.  One-group Pre-Test/Post-Test Design
 
O1 X O2
 

3.  Non-randomized comparison group Pre-Test/Post-Test Design
 
O1 X O2
O1 O2
 

4.  One-group Time Series Design
 
O1 O2 O3 X O4 O5 O6
 
 

Another way to improve the validity of designs for description:

1.  Case study design
 
X O2
 

2.  Randomized Control Group Post-Test only design
 
X O2
O2
 

3.  Randomized Control Group Pre-Test/Post-test design
 
O1 X O2
O1 O2
 

4.  Control group Time Series Design
 
O1 O2 O3 X O4 O5 O6
O1 O2 O3 O4 O5 O6