Confidence Interval Calibration Evaluation Study

Objective: This study aims to evaluate confidence interval calibration.

Rationale: Observational studies are prone to bias, but unfortunately this bias is often brushed aside with a single comment in the discussion of a paper and is never quantified. In the past we have proposed using negative controls (exposure-outcome pairs where the true relative risk is believed to be one) to produce an empirical bias distribution, and subsequently calibrate p-values. Here we propose to extend this approach to calibrated confidence intervals, which requires the use of positive controls. Since real positive controls are problematic, we will generate artificial positive controls by injection additional (simulated) outcomes on top of negative controls. To demonstrate and evaluate confidence interval calibration, we will reproduce two pairs of observational studies that have produced conflicting results. We will try to demonstrate that our calibration procedure has good internal validity by showing coverage of the confidence interval improves as measured using our negative and positive controls. We will try to show external validity by showing that after calibration our conflicting studies are no longer in disagreement.

Project Lead(s): Martijn Schuemie, Marc Suchard, George Hripcsak, Patrick Ryan, David Madigan

Coordinating Institution(s): Janssen R&D

Participating Institution(s): Janssen R&D

Full Protocol: CI Calibration protocol

Initial Proposal Date: 1/11/2016

Launch Date: 1/11/2016

Receive Results for Analysis Date: NA

Study Closure Date: 1/5/2017 (Study closed)

Results Submission: NA