Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Evaluating Alarm Classifiers with High-confidence Data Programming

Evaluating Alarm Classifiers with High-confidence Data Programming Classification of clinical alarms is at the heart of prioritization, suppression, integration, postponement, and other methods of mitigating alarm fatigue. Since these methods directly affect clinical care, alarm classifiers, such as intelligent suppression systems, need to be evaluated in terms of their sensitivity and specificity, which is typically calculated on a labeled dataset of alarms. Unfortunately, the collection and particularly labeling of such datasets requires substantial effort and time, thus deterring hospitals from investigating mitigations of alarm fatigue. This article develops a lightweight method for evaluating alarm classifiers without perfect alarm labels. The method relies on probabilistic labels obtained from data programming—a labeling paradigm based on combining noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method produces confidence bounds for the sensitivity/specificity values from a hypothetical evaluation with manual labeling. Our experiments on five alarm datasets collected at Children’s Hospital of Philadelphia show that the proposed method provides accurate bounds on the classifier’s sensitivity/specificity, appropriately reflecting the uncertainty from noisy labeling and limited sample sizes. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Computing for Healthcare (HEALTH) Association for Computing Machinery

Evaluating Alarm Classifiers with High-confidence Data Programming

Download PDF
24 pages

Loading next page...
 
/lp/association-for-computing-machinery/evaluating-alarm-classifiers-with-high-confidence-data-programming-h9Zo98LMse
Publisher
Association for Computing Machinery
Copyright
Copyright © 2022 Association for Computing Machinery.
ISSN
2691-1957
eISSN
2637-8051
DOI
10.1145/3549942
Publisher site
See Article on Publisher Site

Abstract

Classification of clinical alarms is at the heart of prioritization, suppression, integration, postponement, and other methods of mitigating alarm fatigue. Since these methods directly affect clinical care, alarm classifiers, such as intelligent suppression systems, need to be evaluated in terms of their sensitivity and specificity, which is typically calculated on a labeled dataset of alarms. Unfortunately, the collection and particularly labeling of such datasets requires substantial effort and time, thus deterring hospitals from investigating mitigations of alarm fatigue. This article develops a lightweight method for evaluating alarm classifiers without perfect alarm labels. The method relies on probabilistic labels obtained from data programming—a labeling paradigm based on combining noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method produces confidence bounds for the sensitivity/specificity values from a hypothetical evaluation with manual labeling. Our experiments on five alarm datasets collected at Children’s Hospital of Philadelphia show that the proposed method provides accurate bounds on the classifier’s sensitivity/specificity, appropriately reflecting the uncertainty from noisy labeling and limited sample sizes.

Journal

ACM Transactions on Computing for Healthcare (HEALTH) – Association for Computing Machinery

Published: Nov 3, 2022

Keywords: Physiological alarms

References

  • Reduction of false alarms in the intensive care unit using an optimized machine learning based approach
    [,
  • Confidence interval estimation of an ROC curve: An application of Generalized Half Normal and Weibull distributions
    [,
  • Video analysis of factors associated with response time to physiologic monitor alarms in a children’s hospital
    [,
  • Confidence intervals for the area under the ROC curve
    [,
  • Insights into the problem of alarm fatigue with physiologic monitor devices: A comprehensive observational study of consecutive intensive care unit patients
    [,
  • Computer aided clinical trials for implantaule cardiac devices
    [,
  • Development and application of a clinical microsystem simulation methodology for human factors-based research of alarm fatigue
    [,
  • Suppression of intensive care unit false alarms based on the arterial blood pressure signal
    [,
  • Research: Acceptability, feasibility, and cost of using video to evaluate alarm fatigue
    [,
  • Reducing pulse oximetry false alarms without missing life-threatening events
    [,
  • Systematic review of physiologic monitor alarm characteristics and pragmatic interventions to reduce alarm frequency
    [,
  • Adversarial data programming: Using gans to relax the bottleneck of curated labeled data
    [,
  • Generative adversarial data programming
    [,
  • High-confidence data programming for evaluating suppression of physiological alarms
    [,
  • Snorkel: Rapid training data creation with weak supervision
    [,
  • Data programming: Creating large training sets, quickly
    [,
  • Training Complex Models with Multi-Task Weak Supervision
    [,
  • Technological distractions (part 2): A summary of approaches to manage clinical alarms with intent to reduce alarm fatigue
    [,