Background:
Albogami et al. conducted an observational study in the IBM MarketScan Commercial claims database, and found that among that patients with type 2 diabetes and chronic lower respiratory disease (CLRD, e.g. asthma or COPD), persons initiating Glucagon-Like Peptide 1 Receptor Agonists (GLP-1RA) had a lower risk of CLRD hospitalizations and exacerbations compared to new users of dipeptidyl peptidase 4 inhibitors (DPP-4I). The study applied a comparative cohort design, with a target cohort of new users of GLP-1RA and comparator cohort of new users of DPP-4i, propensity-score adjustment for baseline confounding, and separate outcome models and two endpoints: Cox proportional hazards model to estimate hazard ratio of CLRD hospitalization, and a Poisson regression to estimate incidence rate ratio of CLRD exacerbation. The publication, “Glucagon-Like Peptide 1 Receptor Agonists and Chronic Lower Respiratory Disease Exacerbations Among Patients With Type 2 Diabetes”, is an excellent example of current best practice in pharmacoepidemiology, led by leaders in the field and published in a high-impact journal, Diabetes Care in April 2021 (https://care.diabetesjournals.org/content/44/6/1344). The results, if confirmed, could potentially impact clinical care for the large number of patients who have comorbid diabetes and CLRD.
Challenge:
In Chapter 14 of “The Book of OHDSI”, reproducibility is identified as one of the desired attributes for generating reliable evidence.
“Reliable evidence should be reproducible such that a different researcher should be able to perform the same task of executing a given analysis on a given database and expect to produce an identical result as the first researcher. Reproducibility requires that the process is fully-specified, generally in both human-readable and computer-executable form such that no study decisions are left to the discretion of the investigator. (https://ohdsi.github.io/TheBookOfOhdsi/EvidenceQuality.html)”.
A key barrier to achieving reproducibility in observational studies today is that more times than not, neither the data nor the analysis are made publicly available to enable different researchers to ask the identical question and verify that they produce identical results to the original researcher. Currently, the prevailing evidence dissemination strategy for observational research is peer-review publications, which provide free text description of methods, sometimes accompanied by some supplemental materials, but are often constrained by word counts and inconsistent in the reporting of key analytic details. In their 2017 publication, “Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0”, the ISPE-ISPOR Joint Taskforce wrote, “The ultimate measure of transparency is whether a study could be directly replicated by a qualified independent investigator based on publically reported information. While sharing data and code should be encouraged whenever data use agreements and intellectual property permit, in many cases this is not possible. Even if data and code are shared, clear, natural language description would be necessary for transparency and the ability to evaluate the validity of scientific decisions. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5639362/)”. There are numerous viewpoints and perspectives published arguing for the field to improve its pre-specification of analyses through public registration and posting of protocols and more details reporting of methodological design decisions in manuscripts. Some efforts, such as the REPEAT Initiative (http://www.encepp.eu/encepp/viewResource.htm?id=19637), have sought to evaluate the level of reproducibility in published pharmacoepidemiology studies. A critical challenge in all of this work is the lack of specificity that can come with ‘natural language description’ and, in the absence of source code to fully repeat an analysis, the investigator-induced error that may occur with the interpretation of ‘natural language descriptions’ into a new implementation.
In this workshop, the ‘OHDSI reproducibility challenge’ will aim to reproduce the populations (exposures and outcomes) used in the analysis by Albogami et al. We will also aim to quantify the heterogeneity of interpretations that qualified researchers may produce when attempting to reproduce the same study. The Albogami study was purposefully selected as an example of current best practice, led by a leading research team in the field and published in a top clinical journal. The OHDSI reproducibility challenge is being conducted in collaboration with the lead author of the original study, though for the purposes of the workshop, only the information in the publication and supplemental materials will be used. Teams will independently work on two study design tasks, to be implemented in OHDSI ATLAS: 1) define the target exposure cohort, and 2) create cohorts for the primary and secondary outcomes. All teams will be provided access to an ATLAS instance with an empty cohort definition template, pre-populated with the necessary conceptsets to produce the cohorts, such that the focus of the reproducibility is limited to the logic of the cohort definitions and not selection of codes. Teams will separately implement the cohorts based on their interpretation of the publication. Final cohorts will then be shared and executed against the IBM MarketScan Commercial claims database (the same data as used in the original publication). Variation in cohort definition logic between teams will be reviewed, as well as empirical consequences of those differences will be quantified in terms of both cohort counts and the prevalence of baseline characteristics (demographics, prior conditions/drugs/procedures/measurements).
Results for the OHDSI reproducibility challenge will be summarized for presentation and publication.
Participation:
To accommodate as many OHDSI community members who are interested in this unique event, we have created registration for two types of participants: collaborators and observers.
Registration for Collaborators is based on a first-come, first-served basis. We ask that you be fully committed if you sign up to be a collaborator. The deadline to sign up is September 1 or until we have reached our max number of collaborators.
WORKSHOP COLLABORATORS
-
- Must agree to read Albogami et al, “Glucagon-Like Peptide 1 Receptor Agonists and Chronic Lower Respiratory Disease Exacerbations Among Patients With Type 2 Diabetes”, Diabetes Care, 2021, and supplemental materials prior to the workshop
- Must be committed to participate in full-day workshop
- Must be familiar with OMOP common data model and OHDSI vocabularies (EHDEN Academy course on vocabulary would be useful refresher for those less comfortable)
- Must be familiar with using ATLAS to build cohort definitions, and willing to use an ATLAS instance during workshop (either a local instance or the public demo instance)
- Will be placed in small teams of 4-5 collaborators, who agree to complete two breakout reproducibility exercises in the time allotted, and present their results to the broader group.
- Agree for work generated during the workshop to be used as part of publication about the OHDSI reproducibility challenge
- Will be encouraged to participate in co-authorship (subject to satisfying ICJME authorship guidelines) of publication summarizing the results of the OHDSI reproducibility challenge
TO SIGN UP TO BE AN OBSERVER IN THE WORK SHOP, PLEASE SEE BELOW.
WORKSHOP OBSERVERS
-
- Must agree to read Albogami et al, “Glucagon-Like Peptide 1 Receptor Agonists and Chronic Lower Respiratory Disease Exacerbations Among Patients With Type 2 Diabetes”, Diabetes Care, 2021, and supplemental materials prior to the workshop
- May listen to any or all portions of the full-day workshop
- Will not be assigned to teams, so can follow along with any of the collaborator teams during breakouts and join for group review sessions
- Will not be considered sufficient contributors to warrant co-authorship in resulting publications
Workshop Fees: There is no fee to be a workshop observer
While we are excited to create this opportunity for you to observe this workshop for free, there are costs associated with coordinating all OHDSI community activities. To help offset these costs, we provide the additional optional opportunity for participants to support the OHDSI community through ‘ “workshop optional registration fee’ tickets. You may cancel your free registration ticket at any time; however, contributions are not refundable (should you need a tax receipt for your contribution, please contact symposium@ohdsi.org BEFORE making your registration contribution through this website); anyone can register and there are no workshop registration limits.
**All participants are required to read Albogami et al, “Glucagon-Like Peptide 1 Receptor Agonists and Chronic Lower Respiratory Disease Exacerbations Among Patients With Type 2 Diabetes”, Diabetes Care, 2021, and supplemental materials PRIOR to the workshop. The entire workshop depends on full understanding the contents of the publication, and there will NOT be time alotted for those who come unprepared to get up-to-speed**
TO REGISTER TO BE A WORKSHOP OBSERVER, PLEASE CLICK THE BLUE BUTTON BELOW
Workshop Registration for Observers
