Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Finding Multidimensional Simpson's Paradox

Finding Multidimensional Simpson's Paradox Finding and analyzing Simpson's paradox, a well known statistical phenomenon, has found many applications. While the existing literature focuses on only analyzing the causes of identi ed Simpson's paradox, there is no systematic analysis on Simpson's paradox in multidimensional spaces. In this paper, we develop a simple yet practical approach to automatically identify all Simpson's paradox instances formed by various sub-populations and separator attributes in a multidimensional data set. Moreover, we analyze the distribution of the multidimensional Simpson's paradox instances on three real data sets with respect to dimensionality, size of sub-populations, participation of individual records, redundancy, and more. We obtain a series of interesting observations about a few questions that have never been asked before. The results open doors to a few interesting directions for future study. Moreover, this paper is an outcome from a high-school student summer research internship. It re ects our on-going e ort in promoting data science research to youth and high school students. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM SIGKDD Explorations Newsletter Association for Computing Machinery

Finding Multidimensional Simpson's Paradox

Xu, Jay; Pei, Jian; Cong, Zicun
ACM SIGKDD Explorations Newsletter , Volume 24 (2): 13 – Dec 5, 2022
Download PDF
13 pages

Loading next page...
 
/lp/association-for-computing-machinery/finding-multidimensional-simpson-s-paradox-StSTvdN9eo
Publisher
Association for Computing Machinery
Copyright
Copyright © 2022 Copyright is held by the owner/author(s)
ISSN
1931-0145
eISSN
1931-0153
DOI
10.1145/3575637.3575645
Publisher site
See Article on Publisher Site

Abstract

Finding and analyzing Simpson's paradox, a well known statistical phenomenon, has found many applications. While the existing literature focuses on only analyzing the causes of identi ed Simpson's paradox, there is no systematic analysis on Simpson's paradox in multidimensional spaces. In this paper, we develop a simple yet practical approach to automatically identify all Simpson's paradox instances formed by various sub-populations and separator attributes in a multidimensional data set. Moreover, we analyze the distribution of the multidimensional Simpson's paradox instances on three real data sets with respect to dimensionality, size of sub-populations, participation of individual records, redundancy, and more. We obtain a series of interesting observations about a few questions that have never been asked before. The results open doors to a few interesting directions for future study. Moreover, this paper is an outcome from a high-school student summer research internship. It re ects our on-going e ort in promoting data science research to youth and high school students.

Journal

ACM SIGKDD Explorations Newsletter – Association for Computing Machinery

Published: Dec 5, 2022

References

  • Can you trust the trend? discovering simpson's paradoxes in social data
    Alipourfard, Nazanin; Fennell, Peter G.; Lerman, Kristina
  • Bottom-up computation of sparse and iceberg cube
    Beyer, Kevin; Ramakrishnan, Raghu
  • Sex bias in graduate admissions: Data from berkeley: Measuring bias is harder than is usually assumed, and the evidence is sometimes contrary to expectation
    Bickel, Peter J; Hammel, Eugene A; O'Connell, J William
  • Epidemia COVID-19: Aggiornamento nazionale, 09 Marzo 2020 { ore 16:00
    di Sanita, Istituto Superiore
  • Discovering surprising instances of simpson's paradox in hierarchical multidimensional data
    Fabris, Carem C.; Freitas, Alex A.
  • Data cube: a relational aggregation operator generalizing group-by, cross-tab, and sub-totals
    Gray, J.; Bosworth, A.; Lyaman, A.; Pirahesh, H.
  • Data Min- ing: Concepts and Techniques
    Han, Jiawei; Pei, Jian; Tong, Hanghang
  • Quotient cube: How to summarize the semantics of a data cube
    Lakshmanan, Laks V. S.; Pei, Jian; Han, Jiawei
  • Discovering frequent closed itemsets for association rules
    Pasquier, Nicolas; Bastide, Yves; Taouil, Ra k; Lakhal, Lot
  • Comment: Understanding simpsons paradox
    Pearl, Judea
  • Vi
    Pearson, Karl; Lee, Alice; Bramley-Moore, Leslie
  • The interpretation of interaction in contingency tables
    Simpson, E. H.
  • Simpson's Paradox
    Sprenger, Jan; Weinberger, Naftali
  • Simpson's paradox in Covid-19 case fatality rates: a mediation analysis of age-related causal e ects
    von Kugelgen, Julius; Gresele, Luigi; Scholkopf, Bernhard
  • How much do minority lives matter? Jour- nal of Statistics Education, 23(2), 2015
    Witmer, Je
  • Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72,314 Cases From the Chinese Center for Disease Control and Prevention
    ZunyouWu, ; McGoogan, Jennifer M.
  • Notes on the theory of association of attributes in statistics
    Yule, G. Udny
  • An array-based algorithm for simultaneous multidimensional aggregates
    Zhao, Yihong; Deshpande, Prasad M.; Naughton, Je rey F.