Objective of epidemiologic research

“To obtain a valid, precise, and generalizable estimate of the effect of an exposure on the occurrence of an outcome (e.g., disease)”

• Design → enhance the precision and validity of the effect estimate
• No study is perfect
  • Interpret the precision and validity of the effect estimate

T. Lash, M. Fox. Applying Quantitative Bias Analysis to Epidemiologic Data

Counterfactual vs factual worlds

• Counterfactual comparisons of exposed and unexposed → perfect and impossible
• RCTs → closest we can get to the counterfactual comparison of exposed and unexposed

Hernán MA, Robins JM (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC.

Estimating magnitude of effect in epidemiologic studies

• Design
• Data
• Statistical analyses

Error in epidemiologic research

• Random error
  • Sampling (random) variability
  • Chance
• Systematic error (bias)
  • Selection bias
  • Confounding
  • Measurement (information) bias

Validity

• No bias
  • Valid study = Study with minimal bias (systematic error)

• Internal validity → external validity

Accuracy

• Value of the parameter is estimated with little error
• Errors in estimation: random or systematic

Had we known the true effect of the exposure on the outcome (simulation):

Random error

• Can chance be solely responsible for an observed association?
• Reducing random error → increasing precision of an estimate
• Enlarging the size of the study
  • “The statistical dictum that there is no sampling error if an entire population (as opposed to a sample of it) is studied does not apply to epidemiologic studies, even if an entire population is included in the study”

• Planning study size based on desired precision of an estimate
  • Envision stratification analyses

Precision

• No random error

Confidence intervals

How CIs work?

See here

Estimate and confidence level

• Association measure estimate

  • Risk difference
  • Risk ratio
  • Odds ratio
  • Rate difference
  • Rate ratio
  • Hazard ratio

• 95% confidence interval (CI)

  • How often 95 % confidence intervals computed from many studies would contain the true effect size if all the assumptions used to compute the intervals were correct

  • Uncertainty in an epidemiologic result under assumption of random error alone → often unrealistic

Study size and precision

Type I and II errors

• 20th century

  The requirement of randomization in experimental design was first stated by R. A. Fisher, statistician and geneticist, in 1925 in his book Statistical Methods for Research Workers. Fisher's dictum was that randomization eliminates bias and permits a valid test of significance.

• Type I error rate

  • a “false-positive” mistake
  • is usually represented by α, and is conventionally set to 0.05
  • probability of rejecting the null hypothesis when it is true

• Type II error rate

  • a “false-negative” mistake
  • probability of failing to reject the null hypothesis when it is false

Type I and II errors

Null hypothesis significance testing

Null hypothesis

• Is a hypothesis of no association between two variables in a hypothetical superpopulation
  • Compared groups were sampled in a random fashion from the hypothetical superpopulation
• Is not about the observed study groups

The difference is not "statistically significant"

• One cannot reject the null hypothesis that the superpopulation groups are different
  • It does not mean that the two observed groups are the same

Rothman, K., Greenland, S., & Lash, TL. (2008). Modern Epidemiology, 3rd Edition. Philadelphia, PA: Lippincott Williams & Wilkins.

What p-value is not

P-value

The probability that a test statistic computed from the data would be greater than or equal to its observed value, assuming that the test hypothesis is correct

• Indicates the degree to which the data conform to the pattern predicted by the test hypothesis and all the assumptions used for statistical modeling

P-value

In any given study, the observed difference can be "statistically significant"

• When the model used to compute it is wrong
  • Many sources of uncontrolled bias
• Due to chance
  • 0.05 alpha level
  • false-positive statistically significant difference 5% of the time due to chance
• Dichotomization of study results based on p-values is harmful
  • Cherry-picking of “significant” results
  • Lack of studies reproducibility

Good practices

http://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html

Let's discuss

• Estimation of the magnitude of the association
  • 95% Confidence Interval
• Do not treat confidence interval as a substitute for a null hypothesis significance test
  • Interpret 95% CI width, not whether it crosses the null value

Are these results pointing to the same conclusion or not?

• RR = 1.20 (95% CI: 0.97–1.48)
• RR = 1.20 (95% CI: 1.09–1.33)

95% Confidence Interval misinterpretation and p-value function

What's wrong here?

15 min break

Threat to validity: Bias

• Can not be remedied by large (infinite) sample size
• Can not be remedied by pooling studies
• Often is larger than random error

Bias

• Selection bias (=collider stratification bias)
• Confounding
• Measurement (= information) bias
• Misclassification (if discrete variables)
  • Exposure, outcome, covariables
  • Differential/ non-differential
  • Dependent/ independent

Selection

Rothman KJ, Gallacher JE, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42(4):1012-4

Selection bias

Errors due to systematic differences in characteristics between those who are selected for study and those who are not

survivorship bias

Selection bias

• Both exposure and outcome affect participation in the study
  • At enrollment
  • At follow-up
• Association is measured among participants

Selection bias

• Risk of the outcome in exposed and unexposed in all eligible?
• 100/1000 = 10% (same outcome risk in exposed and unexposed in all eligible)
• Exposure-Outcome association in all eligible: RR = ?
• (100/1000) / (100/1000) = 1.0
• Exposure-Outcome association in participants: RR = ?
• (80/480) / (60/260) = 0.72
• Exposure-Outcome association in nonparticipants: RR = ?
• (20/520) / (40/740) = 0.71
• Way of thinking: association in all eligible and in participants is different

DAGs

Collider stratification bias

Selection bias

• Differential participation
• Case-control study of residence proximity (A) to farms and cancer (Y)
• By definition, we start from selecting cases
  • Outcome → participation (S=selection)
• People who live close to the farms are more likely to participate than those who do not live near
  • Exposure → participation (S=selection)
• By conditioning on S, we open a biasing path

Selection bias

• Case-control study of association between hormone therapy and cancer
  • Cancer cases
  • Cancer controls are women hospitalized with hip fracture
  • Exposure → participation?
• Would there be a selection bias in this study?

Selection bias

• There is a negative association between hormone replacement therapy (HRT) and hip fracture: HRT → ↓ risk hip fracture
• Controls are less healthy than cases: HRT seems protective against cancer while it is not (spurious association)

Selection bias

• Differential loss to follow-up (informative censoring)
• Follow-up studies (observational and RCTs)
• Effect in those who was not lost to follow-up
• Think "survivorship bias"

Selection bias

Birth weight paradox

The birth weight "paradox" uncovered?

Selection bias

Birth weight paradox

The birth weight "paradox" uncovered?

Selection bias has many names

• Inappropriate selection of controls (case-control studies)
• Competing risks
• Differential loss-to-follow-up (informative censoring)
• Berkson’s bias
• Self-selection bias/Volunteer Bias
• Healthcare access bias
• Healthy worker bias
• Healthy adherer bias
• Sick stopper bias
• Non-response bias
• Matching (case-control studies)

Let's discuss

• Effect of smoking on diabetes
• What to adjust for? HarvardX PH559x Causal Diagrams: Draw Your Assumptions Before Your Conclusions

Let's discuss

• Observational cohort study of effect of hormone replacement therapy (HRT) on the risk of coronary heart disease (CHD)
  • Prevalent and new users of HRT are included in the study
  • Follow-up starts at enrollment for both initiators and prevalent users
  • HR for CHD for current users HRT vs. never users HRT was 0.72 (Nurses’ Health Study, Grodstein, 2006)
• RCT of effect of HRT on the risk of CHD
  • Intention-to-treat analysis (effect of randomization)
  • Follow-up starts from the date of HRT initiation
  • HR for CHD for users HRT vs. not users HRT was 1.24 (Women’s Health Initiative, Manson, 2003)
• What went wrong in the observational study?

Hernan MA, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766-79

Let's discuss

• When including prevalent and new users of HRT the association with CHD is attenuated due to the effect of prevalent users of HRT who are "long-term survivors of CHD" ("super women" who never developed CHD while on HRT)
  • Healthier individuals are more likely to initiate and to stay on preventive treatment
  • Treatment initiators with early side effects discontinue HRT ("depletion of susceptible women") → selection mechanism
• When asking a proper research question "what is the effect of starting HRT?" the observational cohort study shows the same results as the RCT

Selection bias

Take home message

Confounding

• Mixing of the effects
• The apparent effect of the exposure of interest is distorted because the effect of an extraneous factor is mistaken for or mixed with the actual exposure effect
• Overestimation/Underestimation
• Changing direction of the effect
• Known/Unknown
• Measured/Unmeasured
• Residual

Confounding

Properties of a confounder

• Associated with the exposure
• Associated with the outcome
• Does not lie on the pathway between exposure and outcome
• Unequally distributed among the exposed and the unexposed

Confounding vs confounders

Confounding

Examples from your research

Confounding is not mediation

Confounding exercise

• Assume that this causal structure is true
• According to this DAG
  • Is there a confounding for an association between physical activity and cervical cancer ?
  • Would you stratify by or adjust for PAP-test results?
  • Would you restrict the study population to women with PAP+?

Confounding exercise

Causal Diagrams: Draw Your Assumptions Before Your Conclusions, Hernan

Controlling confounding

• Design
  • Randomization (intervention)
  • Restriction (observational)
  • Matching (observational)
• Analysis
  • Stratification
  • Standardization
  • Multivariable outcome regression
  • Causal modeling (e.g., marginal structural models)

Sensitivity analyses & triangulation

• Evaluate statistical and other assumptions made when conducting the study

• Challenge results

  • Sensitivity analyses
  • Changing definitions of exposure/outcome
  • Checking possible violations of your (model) assumptions
  • Bias analysis (for unmeasured or residual confounding, selection bias, etc.)

• Triangulation

  • Integrating results from different study designs investigating the same research question
  • Each design has different sources of potential bias that are unrelated

Confounding

Take home message

Let's discuss

Example 1

discussion

Let's discuss

Example 2

• Do you agree or disagree with conclusion?
  • Why or why not?
• Confounding by indication: the association can be explained by insufficient control of the confounding by depression severity
• Bias (e.g. residual confounding by indication) is a threat to validity and needs to be considered when interpreting results
• E-value: https://www.evalue-calculator.com/
  • The strength of an association an unmeasured confounder would need to have with exposure and the outcome beyond measured covariables to explain away the observed association

Sources for selection bias and confounding

1. Confidence interval viz
2. Causal Diagrams: Draw Your Assumptions Before Your Conclusions, Hernan
3. The ASA's Statement on p-Values: Context, Process, and Purpose
4. Statistical tests, P-values, confidence intervals, and power: a guide to misinterpretations 1.A Structural Approach to Selection Bias
5. The birth weight "paradox" uncovered?

Measurement (information) bias

• Distorts the true strength of association
• Measurement bias of binary variable → misclassification
• Magnitude → Comparison with the classification obtained without error
• Differential vs non-differential
  • Differential → depends on values of other variables
  • Exposure assessment affected by the outcome
  • Outcome assessment affected by the exposure
  • Confounder assessment affected by the exposure, etc.
• Dependent vs independent
  • Depends on the error in measuring other variables
  • Individuals misclassified on exposure are more likely than individuals correctly classified on exposure to be misclassified on a second variable (e.g., disease status)

Non-differential misclassification

Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC. Chapter 9

Non-differential misclassification

Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC. Chapter 9

Differential misclassification

Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC. Chapter 9

Differential misclassification

Recall bias

Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC. Chapter 9

Differential misclassification

Detection bias

• Differential misclassification of the outcome in regard with exposure

Shahar E. Causal diagrams for encoding and evaluation of information bias. J Eval Clin Pract. 2009;15(3):436-40

Differential misclassification

Detection bias

Shahar E. Causal diagrams for encoding and evaluation of information bias. J Eval Clin Pract. 2009;15(3):436-40

Differential misclassification

Detection bias in RCT

Mansournia MA, Higgins JP, Sterne JA, Hernan MA. Biases in Randomized Trials: A Conversation Between Trialists and Epidemiologists. Epidemiology. 2017;28(1):54-9.

Exposure misclassification

• Non-differential misclassification
  • True non-null exposure-outcome association & dichotomous exposure → bias towards the null (dilution of the effect)
  • Can produce bias away from the null by chance for outcomes with more than 2 levels
• Differential → unpredictable
  • Can simulate scenarios (simulation study)

Chubak J, Pocobelli G, Weiss NS. Tradeoffs between accuracy measures for electronic health care data algorithms. J Clin Epidemiol. 2012;65(3):343-9 e2.

Time-related bias

• It is essentially exposure misclassification

  Immortal time: intuition

  • Is being awarded Oscar prolongs life?
  • Immortal time bias: an actor or director had to stay alive long enough to be classified as “Oscar winner”

Immortal time bias

Immortal time

• Time during which the outcome could not have occurred
• Usually occurs before the person initiates treatment
• Patient must remain event-free in order to be classified as treated
• Incorrect consideration of this (untreated) time → immortal time bias

Lead time bias

• The added time of illness if the diagnosis is caught during its latency period

Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol Community Health. 2004;58(8):635-41.

Outcome misclassification

• Non-differential misclassification → bias towards the null
  • If high specificity → relatively unbiased

“If the disease can be defined such that there are no false-positives (i.e., specificity is 100%) and the misclassification of true positives (i.e., sensitivity) is nondifferential with respect to exposure, then risk ratio measures of effect will not be biased”

• Differential misclassification → unpredictable
  • Can do simulation study

Lash TL, Fox MP, Fink AK. Applying Quantitative Bias Analysis to Epidemiologic Data. Springer. 2009

Confounder misclassification

• Non-differential misclassification of confounder
  • Jeopardize confounding control → residual confounding
  • Bias in the direction of confounding
• Differential misclassification of confounder
  • Unpredictable

Validation studies

• Magnitude of misclassification
• Study validity
• Feed-back to clinicians
• Frequently cited papers
• Lots of work

Validation studies

• Gold standard: medical records
• Algorithm
  • E.g. combination of diagnostic ICD-10 codes E10-E11 and medication ATC codes A10 → patients with diabetes
  • Proportion = probability (bound to be between 0 and 1)

Sensitivity

• The proportion of patients with the disease of interest who are algorithm-positive
• The probability that a subject who was truly exposed was correctly classified as exposed is the classification scheme’s sensitivity (SE)

Specificity

• The proportion of persons without the disease who are algorithm-negative
• The probability that a subject who was truly unexposed was correctly classified as unexposed is the classification scheme’s specificity (SP)

FN & FP

• The proportion truly +ve who were incorrectly classified as -ve is the false-negative proportion (FN)
• The proportion incorrectly classified as -ve is the false-positive proportion (FP)

Validation studies

PPV

• Positive predictive value
• The proportion of algorithm-positive patients who truly have the disease of interest
• The probability of truly being exposed if classified as exposed

  NPV

• Negative predictive value
• The proportion of algorithm-negative persons without the disease of interest
• The probability of truly being unexposed if classified as unexposed

PPV and NPV are prevalence-dependent

Prevalence → PPV & NPV

https://micncltools.shinyapps.io/TestAccuracy/

Prevalence → PPV & NPV

• PPV?
• PPV = 90/(180+90) = 0.3333333
• NPV?
• NPV = 720/(720+10) = 0.9863014

Prevalence → PPV & NPV

https://micncltools.shinyapps.io/TestAccuracy/

Prevalence → PPV & NPV

Prevalence → Sensitivity

Prevalence → Specificity

Bias analysis

Non-differential misclassification of the exposure

• SE and SP parameters of the exposure are same among outcome+ and outcome- → exposure is misclassified non-differentially in regard to the outcome

https://graeme-pmott.shinyapps.io/prob_bias_analysis/

Non-differential misclassification of the outcome