Comparative Effectiveness Review Methods

The Agency for Healthcare Research and Quality (AHRQ) commissioned the RTI International-University of North Carolina at Chapel Hill (RTI-UNC) Evidence-based Practice Center (EPC) to explore how systematic review groups have dealt with clinical heterogeneity and to seek out best practices for addressing clinical heterogeneity in systematic reviews (SRs) and comparative effectiveness reviews (CERs). Such best practices, to the extent they exist, may enable AHRQ's EPCs to address critiques from patients, clinicians, policymakers, and other proponents of health care about the extent to which "average" estimates of the benefits and harms of health care interventions apply to individual patients or to small groups of patients sharing similar characteristics. Such users of reviews often assert that EPC reviews typically focus on broad populations and, as a result, often lack information relevant to patient subgroups that are of particular concern to them. More important, even when EPCs evaluate literature on homogeneous groups, there may be varying individual treatment for no apparent reason, indicating that average treatment effect does not point to the best treatment for any given individual. Thus, the health care community is looking for better ways to develop information that may foster better medical care at a "personal" or "individual" level. To address our charge for this methods project, the EPC set out to answer six key questions (KQ). Key questions for methods report on clinical heterogeneity include: 1. What is clinical heterogeneity? a. How has it been defined by various groups? b. How is it distinct from statistical heterogeneity? c. How does it fit with other issues that have been addressed by the AHRQ Methods Manual for CERs? 2. How have systematic reviews dealt with clinical heterogeneity in the key questions? a. What questions have been asked? b. How have they pre-identified population subgroups with common clinical characteristics that modify their intervention-outcome association? c. What are best practices in key questions and how these subgroups have been identified? 3. How have systematic reviews dealt with clinical heterogeneity in the review process? a. What do guidance documents of various systematic review groups recommend? b. How have EPCs handled clinical heterogeneity in their reviews? c. What are best practices in searching for and interpreting results for particular subgroups with common clinical characteristics that may modify their intervention-outcome association? 4. What are critiques in how systematic reviews handle clinical heterogeneity? a. What are critiques from specific reviews (peer and public) on how EPCs handled clinical heterogeneity? b. What general critiques (in the literature) have been made against how systematic reviews handle clinical heterogeneity? 5. What evidence is there to support how to best address clinical heterogeneity in a systematic review? 6. What questions should an EPC work group on clinical heterogeneity address? Heterogeneity (of any type) in EPC reviews is important because its appearance suggests that included studies differed on one or more dimensions such as patient demographics, study designs, coexisting conditions, or other factors. EPCs then need to clarify for clinical and other audiences, collectively referred to as stakeholders, what are the potential causes of the heterogeneity in their results. This will allow the stakeholders to understand whether and to what degree they can apply this information to their own patients or constituents. Of greatest importance for this project was clinical heterogeneity, which we define as the variation in study population characteristics, coexisting conditions, cointerventions, and outcomes evaluated across studies included in an SR or CER that may influence or modify the magnitude of the intervention measure of effect (e.g., odds ratio, risk ratio, risk difference).