Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies

The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation... Given an unsupervised outlier detection task, how should one select i) a detection algorithm, and ii) associated hyperparameter values (jointly called a model)? E ective outlier model selection is essential as di erent algorithms may work well for varying detection tasks, and moreover their performance can be quite sensitive to the values of the hyperparameters (HPs). On the other hand, unsupervised model selection is notoriously difficult, in the absence of hold-out validation data with ground-truth labels. Therefore, the problem is vastly understudied in the outlier mining literature. There exists a body of work that propose internal model evaluation strate- gies for selecting a model. These so-called internal strategies solely rely on the input data (without labels) and the output (outlier scores) of the candidate models. In this paper, we rst survey internal model evaluation strategies including both those proposed speci cally for outlier detection, as well as those that can be adapted from the unsupervised deep representation learning literature. Then, we investigate their e ectiveness empirically in comparison to simple baselines such as random selection and the popular state-of-the-art detector Isolation Forest (iForest) with default HPs. To this end, we set up (and open-source) a large testbed with 39 detection tasks and 297 candidate models comprised of 8 different detectors and various HP con gurations. We evaluate internal strategies from 7 di erent families on their ability to discriminate between models w.r.t. detection performance, without using any labels. Our study reports a striking nding, that none of the existing and adapted strategies would be practically useful: stand-alone ones are not signi cantly di erent from random, and consensus-based ones do not outperform iForest (w/ default HPs) while being more expensive (as all candidate models need to be trained for evaluation). Our survey stresses the importance of and the standing need for e ective unsupervised outlier model selection, and acts as a call for future work on the problem. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM SIGKDD Explorations Newsletter Association for Computing Machinery

The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies

Loading next page...
 
/lp/association-for-computing-machinery/the-need-for-unsupervised-outlier-model-selection-a-review-and-rcauFFxS1N

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Association for Computing Machinery
Copyright
Copyright © 2023 Copyright is held by the owner/author(s)
ISSN
1931-0145
eISSN
1931-0153
DOI
10.1145/3606274.3606277
Publisher site
See Article on Publisher Site

Abstract

Given an unsupervised outlier detection task, how should one select i) a detection algorithm, and ii) associated hyperparameter values (jointly called a model)? E ective outlier model selection is essential as di erent algorithms may work well for varying detection tasks, and moreover their performance can be quite sensitive to the values of the hyperparameters (HPs). On the other hand, unsupervised model selection is notoriously difficult, in the absence of hold-out validation data with ground-truth labels. Therefore, the problem is vastly understudied in the outlier mining literature. There exists a body of work that propose internal model evaluation strate- gies for selecting a model. These so-called internal strategies solely rely on the input data (without labels) and the output (outlier scores) of the candidate models. In this paper, we rst survey internal model evaluation strategies including both those proposed speci cally for outlier detection, as well as those that can be adapted from the unsupervised deep representation learning literature. Then, we investigate their e ectiveness empirically in comparison to simple baselines such as random selection and the popular state-of-the-art detector Isolation Forest (iForest) with default HPs. To this end, we set up (and open-source) a large testbed with 39 detection tasks and 297 candidate models comprised of 8 different detectors and various HP con gurations. We evaluate internal strategies from 7 di erent families on their ability to discriminate between models w.r.t. detection performance, without using any labels. Our study reports a striking nding, that none of the existing and adapted strategies would be practically useful: stand-alone ones are not signi cantly di erent from random, and consensus-based ones do not outperform iForest (w/ default HPs) while being more expensive (as all candidate models need to be trained for evaluation). Our survey stresses the importance of and the standing need for e ective unsupervised outlier model selection, and acts as a call for future work on the problem.

Journal

ACM SIGKDD Explorations NewsletterAssociation for Computing Machinery

Published: Jun 26, 2023

References