Jessica Hullman
Ginni Rometty Professor of Computer Science
Ph.D., Information Science, University of Michigan School of Information, 2014
Jessica Hullman is a computer science who develops interface tools and theoretical frameworks that help people combine their knowledge with statistical models. Advances in computer science have led to a plethora of powerful new tools for learning from data. Hullman’s work investigates how to best design workflows for AI-advised decision-making, explanatory and predictive statistical modeling, and policy decisions under uncertainty. Methodologically, she frequently draws on formal models of rational inference such as statistical decision theory to provide benchmarks for evaluating decision-making workflows.
Her research on uncertainty visualization and interactive data analysis has explored how to best align data-driven interfaces and summaries with human reasoning capabilities, with applications to data privacy decisions, network analysis, causal inference, meta-analysis, strategic games, and scholarly communication of uncertainty, among others.
Hullman’s work has been awarded with multiple best paper and honorable mention awards at top visualization and HCI venues. She was named a Microsoft Faculty Fellow in 2019 and is funded by NSF CAREER, Medium, and Small awards as PI, Adobe, Google, and the U.S. Navy, among others. She frequently speaks and blogs on topics related to uncertainty quantification, data-driven decision-making, and interactive interfaces.
Current Research
A Decision Theoretic Framework for Measuring AI Reliance
Humans frequently make decisions with the aid of artificially intelligent (AI) systems. A common pattern is for the AI to recommend an action to the human who retains control over the final decision. Researchers have identified ensuring that a human has appropriate reliance on an AI as a critical component of achieving complementary performance. We argue that the current definition of appropriate reliance used in such research lacks formal statistical grounding and can lead to contradictions. We propose a formal definition of reliance, based on statistical decision theory, which separates the concepts of reliance as the probability the decision-maker follows the AI’s recommendation from challenges a human may face in differentiating the signals and forming accurate beliefs about the situation. Our definition gives rise to a framework that can be used to guide the design and interpretation of studies on human-AI complementarity and reliance. Using recent AI-advised decision-making studies from literature, we demonstrate how our framework can be used to separate the loss due to mis-reliance from the loss due to not accurately differentiating the signals. We evaluate these losses by comparing a baseline and a benchmark for complementary performance defined by the expected payoff achieved by a rational decision-maker facing the same decision task as the behavioral decision-makers.
An Interactive Paradigm for Differentially-Private Exploratory Analysis
Differential privacy (DP) has the potential to enable privacy-preserving analysis on sensitive data, but requires analysts to judiciously spend a limited "privacy loss budget" epsilon across queries. Analysts conducting exploratory analyses do not, however, know all queries in advance and seldom have DP expertise. Thus, they are limited in their ability to specify epsilon allotments across queries prior to an analysis. To support analysts in spending epsilon efficiently, we propose a new interactive analysis paradigm, Measure-Observe-Remeasure, where analysts "measure" the database with a limited amount of epsilon, observe estimates and their errors, and remeasure with more epsilon as needed. We instantiate the paradigm in an interactive visualization interface which allows analysts to spend increasing amounts of epsilon under a total budget. To observe how analysts interact with the Measure-Observe-Remeasure paradigm via the interface, we conduct a user study that compares the utility of epsilon allocations and findings from sensitive data participants make to the allocations and findings expected of a rational agent who faces the same decision task. We find that participants are able to use the workflow relatively successfully, including using budget allocation strategies that maximize over half of the available utility stemming from epsilon allocation. Their loss in performance relative to a rational agent appears to be driven more by their inability to access information and report it than to allocate epsilon.
Causal Quartets
The average causal effect can often be best understood in the context of its variation. We demonstrate with two sets of four graphs, all of which represent the same average effect but with much different patterns of heterogeneity. As with the famous correlation quartet of Anscombe, these graphs dramatize the way in which real-world variation can be more complex than simple numerical summaries. The graphs also give insight into why the average effect is often much smaller than anticipated.
Selected Publications
Guo, Z., Y. Wu, J. Hartline, and J. Hullman. 2024. A Decision Theoretic Framework for Measuring AI Reliance. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency in Artificial Intelligence 221–36.
Nanayakkara, P., H. Kim, Y. Wu, A. Sarvghad, N. Mahyar, G. Miklau, and J. Hullman. 2024. Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory Analysis. Proceedings of IEEE Security & Privacy (S&P) 231.
Zhang, D., A. Chatzimparmpas, N. Kamali, and J. Hullman. 2024. Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling. Proceedings of the ACM Conference on Computer-Human Interaction (CHI) 1–19.
Gelman, A., J. Hullman, and L. Kennedy. 2023. Casual Quartets: Different Ways to Achieve the Same Average Treatment Effect. The American Statistician 78(3): 267–72.
Wu, Y., Z. Guo, J. Hartline, and J. Hullman. 2023. The Rational Agent Benchmark for Data Visualization. IEEE Transactions on Visualization and Computer Graphics 30(1): 338–47.
Nanayakkara, P., J. Bater, X. Hu, J. Hullman, and J. Rogers. 2022. Visualizing Privacy-Utility Trade-Offs in Differentially Private Data Releases. Proceedings of Privacy Enhancing Technologies (2): 601–18.
Hullman, J. and A. Gelman. 2021. Designing for Interactive Exploratory Data Analysis Requires Theories of Graphical Inference. Harvard Data Science Review 3(3).
