arXiv subjects

Nicholas Biddle

Publications and source records attributed to Nicholas Biddle.

4 recordsLinked to original sources

arXiv2024

Heterogeneous treatment effect estimation with high-dimensional data in public policy evaluation -- an application to the conditioning of cash transfers in Morocco using causal machine learning

Causal machine learning methods can be used to search for treatment effect heterogeneity in high-dimensional datasets even where we lack a strong enough theoretical framework to select variables or make parametric assumptions about data. This paper uses causal machine learning methods to estimate heterogeneous treatment effects in the case of an experimental study carried out in Morocco which evaluated the effect of conditionalizing a cash transfer program on several outcomes including maths test scores which is the focus of this work. We explore treatment effects across a dataset of 1936 pre-treatment variables. For the most part, heterogeneity is modelled by two different factors, participation in education (at the baseline) and more general measures of poverty. Those who are more disadvantaged at the baseline benefit less from any treatment. While conditioning generally has a negative effect this more disadvantaged group is also hurt more by conditioning. The second purpose of this paper is to demonstrate and reflect upon a causal machine learning approach to policy evaluation. We propose a novel causal tree method for interpretable modelling of causal effects and reflect on the difficulty of explaining atheoretical results.

econ.GN↗

arXiv2023

Transparency challenges in policy evaluation with causal machine learning -- improving usability and accountability

Causal machine learning tools are beginning to see use in real-world policy evaluation tasks to flexibly estimate treatment effects. One issue with these methods is that the machine learning models used are generally black boxes, i.e., there is no globally interpretable way to understand how a model makes estimates. This is a clear problem in policy evaluation applications, particularly in government, because it is difficult to understand whether such models are functioning in ways that are fair, based on the correct interpretation of evidence and transparent enough to allow for accountability if things go wrong. However, there has been little discussion of transparency problems in the causal machine learning literature and how these might be overcome. This paper explores why transparency issues are a problem for causal machine learning in public policy evaluation applications and considers ways these problems might be addressed through explainable AI tools and by simplifying models in line with interpretable AI principles. It then applies these ideas to a case-study using a causal forest model to estimate conditional average treatment effects for a hypothetical change in the school leaving age in Australia. It shows that existing tools for understanding black-box predictive models are poorly suited to causal machine learning and that simplifying the model to make it interpretable leads to an unacceptable increase in error (in this application). It concludes that new tools are needed to properly understand causal machine learning models and the algorithms that fit them.

cs.LG↗

arXiv2023

Fairness Implications of Heterogeneous Treatment Effect Estimation with Machine Learning Methods in Policy-making

Causal machine learning methods which flexibly generate heterogeneous treatment effect estimates could be very useful tools for governments trying to make and implement policy. However, as the critical artificial intelligence literature has shown, governments must be very careful of unintended consequences when using machine learning models. One way to try and protect against unintended bad outcomes is with AI Fairness methods which seek to create machine learning models where sensitive variables like race or gender do not influence outcomes. In this paper we argue that standard AI Fairness approaches developed for predictive machine learning are not suitable for all causal machine learning applications because causal machine learning generally (at least so far) uses modelling to inform a human who is the ultimate decision-maker while AI Fairness approaches assume a model that is making decisions directly. We define these scenarios as indirect and direct decision-making respectively and suggest that policy-making is best seen as a joint decision where the causal machine learning model usually only has indirect power. We lay out a definition of fairness for this scenario - a model that provides the information a decision-maker needs to accurately make a value judgement about just policy outcomes - and argue that the complexity of causal machine learning models can make this difficult to achieve. The solution here is not traditional AI Fairness adjustments, but careful modelling and awareness of some of the decision-making biases that these methods might encourage which we describe.

econ.EM↗

arXiv2022

Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation

Methods for learning optimal policies use causal machine learning models to create human-interpretable rules for making choices around the allocation of different policy interventions. However, in realistic policy-making contexts, decision-makers often care about trade-offs between outcomes, not just single-mindedly maximising utility for one outcome. This paper proposes an approach termed Multi-Objective Policy Learning (MOPoL) which combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach to explore the trade-off between multiple outcomes. It does this by building a Pareto frontier of non-dominated models for different hyperparameter settings which govern outcome weighting. The key here is that a low-cost greedy tree can be an accurate proxy for the very computationally costly optimal tree for the purposes of making decisions which means models can be repeatedly fit to learn a Pareto frontier. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya.

cs.LG↗