Are You Calling Me a Poor Performer? Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs
Karl Y. Bilimoria*1,2, Ryan P. Merkow3, Mark E. Cohen1, David J. Bentrem2, Bruce L. Hall4, Clifford Y. Ko1,5
1Divison of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; 2Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, IL; 3Departmet of Surgery, University of Colorado, Denver, CO; 4Department of Surgery, Washington University St. Louis, St. Lous, MO; 5Department of Surgery, UCLA, Los Angeles, MO
BACKGROUND: Surgeons and hospitals are being increasingly assessed by oversight agencies and insurers regarding surgical quality, and much of this information is beginning to be reported publicly. Our objective was to compare various methods used to classify hospitals as outliers (poor performers) in established quality assessment programs.METHODS: From the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP, 2007), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 183 hospitals (n=164,153) and for colorectal surgery at 108 hospitals (n=17,050). Outlier detection methods currently employed in established quality improvement programs were compared.RESULTS: Using the most common criterion of a p-value <0.05 (used by ACS NSQIP, New York and California CABG programs, ASCO’s QOPI) identified 33 outliers for morbidity and 5 for mortality. Using the method employed by Medicare (lowest quintile), 33 outliers were identified for morbidity and mortality. Hierarchical models identified 31 morbidity outliers and 5 mortality outliers. Alternate, more rigorous outlier detection methods identified fewer hospitals as poor performers (Bonferroni correction=13 outliers for morbidity and 0 outliers for mortality; false detection rate=27 outliers for morbidity and 0 outliers for mortality). Similar results were observed for colorectal surgery (table).CONCLUSIONS: There was considerable variation in the number of outliers identified using different detection criterion. Quality programs need to justify their outlier detection methodology based on the intent of the program (i.e. quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.
Back to Program
