Research involving administrative datasets or large
national surveys typically lacks one or more of the three design
criteria that define rigorous "experimental research" designs:
manipulation, randomization, and control. While randomized controlled
trials (RCTs) are the epitome of experimental research and remain the
gold standard for inferring causation, methodology advances over the
past 20 years have greatly increased our interest in and understanding
of quasi-experimental or "observational "research. A major advantage
of existing claims or survey data is that they reflect routine
practice for large and representative populations, in contrast to the
much smaller and often healthier patient populations recruited in
clinical trials. In other words, these datasets capture the
characteristics and experiences of everyday patients in everyday
clinical settings. Moreover, these resources provide the only way to
assess policy- or practice-related changes, the so-called "natural
experiments."
The fundamental strength of RCTs is the primary
criticism of quasi-experimental research: internal validity - the
degree to which the relationship between the treatment and outcome is
free from the effects of extraneous factors. However, treatment
decisions in practice are not randomly assigned. Rather, factors such as
prognosis, patient - and provider-preferences, insurance coverage,
and out-of-pocket costs influence who gets what treatment. Thus,
socio-demographic and clinical characteristics are not balanced between
treated and untreated cohorts. External validity - degree to which
the results can be generalized to persons or settings outside the
experimental situation - is generally less of a concern in observational
studies since the experimental situation is routine patients
receiving routine care.
When independent variable manipulation and
random assignment are beyond the control of the investigator, there
are four other design parameters that can strengthen a study's
internal validity:
- Cohort identification (incident vs. prevalent users)
- Control or "counterfactual" group
- Pre-period measurement
- Post-period measurement
Citations
NF Marko & RJ Well (2010) The role of observational
investigations in comparative effectiveness research. Value in Health.
13(8): 989-997.
S Schneeweiss & J Avorn (2005) A review of uses of health
care utilization databases for epidemiologic research on
therapeutics. J Clin Epidemiol. 58: 323-337.
E von Elm et al. (2007) The Strengthening the Reporting of
Observational Studies in Epidemiology (STROBE) statement: guidelines
for reporting observational studies. Ann Intern Med. 147(8): 573-577.
Other Links
STROBE Statement Website STROBE
stands for an international, collaborative initiative of
epidemiologists, methodologists, statisticians, researchers and journal
editors involved in the conduct and dissemination of observational
studies, with the common aim of STrengthening the Reporting of OBservational studies in Epidemiology.
Video
Rigorous
Quasi-Experimental Comparative Effectiveness Research Study Design
by Professor Matthew Maciejewski from Duke University and the Center
for Health Services Research at Durham VA Medical Center. This 60
min video recorded during the Comparative Effectiveness Research
with Population-Based Data conference in the Baker Institute at Rice
University on July 13, 2012.