Validating the Patient Safety Indicators in the Veterans Health Administration

BACKGROUND: The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) use administrative data to detect potentially preventable in-hospital adverse events. However, few studies have determined how accurately the PSIs identify true safety events. OBJECTIVES: We examined the criterion validity, specifically the positive predictive value (PPV), of 12 selected PSIs using clinical data abstracted from the Veterans Health Administration (VA) electronic medical record as the gold standard. METHODS: We identified PSI-flagged cases from 28 representative hospitals by applying the AHRQ PSI software (v.3.1a) to VA fiscal year 2003 to 2007 administrative data. Trained nurse-abstractors used standardized abstraction tools to review a random sample of flagged medical records (112 records per PSI) for the presence of true adverse events. Interrater reliability was assessed. We evaluated PPVs and associated 95% confidence intervals of each PSI and examined false positive (FP) cases to determine why they were incorrectly flagged and gain insight into how each PSI might be improved. RESULTS: PPVs ranged from 28% (95% CI, 15%-43%) for Postoperative Hip Fracture to 87% (95% CI, 79%-92%) for Postoperative Wound Dehiscence. Common reasons for FPs included conditions that were present on admission (POA), coding errors, and lack of coding specificity. PSIs with the lowest PPVs had the highest proportion of FPs owing to POA. CONCLUSIONS: Overall, PPVs were moderate for most of the PSIs. Implementing POA codes and using more specific ICD-9-CM codes would improve their validity. Our results suggest that additional coding improvements are needed before the PSIs evaluated herein are used for hospital reporting or pay for performance

Adverse effect · Coding (social sciences) · Confidence interval · Health care · Inter-rater reliability · Medical record · Patient safety · Emergency Medicine · Internal Medicine · Medical Coding and Health Information · Medicine · Patient Safety and Medication Errors · Psychology · Sepsis Diagnosis and Treatment

• Temporal Variation in Surgical Mortality Within French Hospitals

  Antoine Duclos, Stéphanie Polazzi et al.•Medical Care•2013

  Add to list

• Improved Coding of Postoperative Deep Vein Thrombosis and Pulmonary Embolism in Administrative Data (AHRQ Patient Safety Indicator 12) After Introduction of New ICD-9-CM Diagnosis Codes

  Banafsheh Sadeghi, Richard H White et al.•Medical Care•2015

  Add to list

• Validity of the Agency for Health Care Research and Quality Patient Safety Indicators and the Centers for Medicare and Medicaid Hospital-acquired Conditions

  Bradford D Winters, Aamir Bharmal et al.•Medical Care•2016

  Add to list

• How Valid is the ICD-9-CM Based AHRQ Patient Safety Indicator for Postoperative Venous Thromboembolism

  Richard H White, Banafsheh Sadeghi et al.•Medical Care•2009

  Add to list

• Evaluating the Patient Safety Indicators

  Amy K Rosen, Peter Rivard et al.•Medical Care•2005

  Add to list

• Effects of Resident Duty Hour Reform on Surgical and Procedural Patient Safety Indicators Among Hospitalized Veterans Health Administration and Medicare Patients

  Amy K Rosen, Susan Loveland et al.•Medical Care•2009

  Add to list

• Identification of In-Hospital Complications From Claims Data

  Ann G Lawthers, Ellen P McCarthy et al.•Medical Care•2000

  Add to list

• Agreement Between Administrative Files and Written Medical Records

  T MICHAEL KASHNER•Medical Care•1998

  Add to list

• Tracking Rates of Patient Safety Indicators Over Time

  Amy K Rosen, Shibei Zhao et al.•Medical Care•2006

  Add to list