Help or Hinder? Evaluating the Impact of Fairness Metrics and Algorithms in Visualizations for Consensus Ranking
Open Access
- 12 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1685-1698
Abstract
For applications where multiple stakeholders provide recommendations, a fair consensus ranking must not only ensure that the preferences of rankers are well represented, but must also mitigate disadvantages among socio-demographic groups in the final result. However, there is little empirical guidance on the value or challenges of visualizing and integrating fairness metrics and algorithms into human-in-the-loop systems to aid decision-makers. In this work, we design a study to analyze the effectiveness of integrating such fairness metrics-based visualization and algorithms. We explore this through a task-based crowdsourced experiment comparing an interactive visualization system for constructing consensus rankings, ConsensusFuse, with a similar system that includes visual encodings of fairness metrics and fair-rank generation algorithms, FairFuse. We analyze the measure of fairness, agreement of rankers’ decisions, and user interactions in constructing the fair consensus ranking across these two systems. In our study with 200 participants, results suggest that providing these fairness-oriented support features nudges users to align their decision with the fairness metrics while minimizing the tedious process of manually having to amend the consensus ranking. We discuss the implications of these results for the design of next-generation fairness oriented-systems and along with emerging directions for future research.Keywords
All Related Versions
Funding Information
- National Science Foundation (2007932)
This publication has 37 references indexed in Scilit:
- FairTest: Discovering Unwarranted Associations in Data-Driven Applications2017
- Collaborative information seekingJournal of the Association for Information Science and Technology, 2013
- A Multi-Level Typology of Abstract Visualization TasksIEEE Transactions on Visualization and Computer Graphics, 2013
- Taxonomy-Based Glyph Design—with a Case Study on Visualizing Workflows of Biological ExperimentsIEEE Transactions on Visualization and Computer Graphics, 2012
- D-SightInternational Journal of Decision Support System Technology, 2012
- A Nested Model for Visualization Design and ValidationIEEE Transactions on Visualization and Computer Graphics, 2009
- A new method for scoring additive multi‐attribute value models using pairwise rankings of alternativesJournal of Multi-Criteria Decision Analysis, 2008
- Web-Hipre: Global Decision Support By Value Tree And AHP AnalysisINFOR: Information Systems and Operational Research, 2000
- Mathematical Derivation of an Election SystemIsis, 1953
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938