Research
Google Scholar · arXiv · Software and examples
No papers match these filters. Try a broader search or clear the filters above.
Select Working Papers
From More to Better: Causal Inference and Digital Experimentation in the Age of AI. pre-print
Neyman Orthogonal Activation Steering. pre-print
Always-On Experimentation. pre-print
Increases in national origin disparities in ICE Arrests, 2025–26. pre-print
Bridging Predictions and Interventions: An Integrated Framework for Automated Decision-Systems. pre-print
Locally Equivalent Weights for Multilevel Regression and Poststratification. pre-print
A weighting framework for clusters as confounders in observational studies. pre-print
On nonasymptotic confidence intervals for treatment effects in randomized experiments. pre-print
Challenges in statistics: A Dozen challenges in causality and causal inference. pre-print
The balancing act in causal inference. pre-print
Distribution-free assessment of population overlap in observational studies. pre-print
Forthcoming
Using multiple outcomes to improve the Synthetic Control Method. Review of Economics and Statistics, forthcoming. pre-print
2026
Statistical methods to estimate the impact of gun policy on gun violence. Gun Violence: The Statistical Issues, edited by C. Loeffler, J. L. Rosenberger, and L. Xue. Chapman and Hall/CRC, pp. 138–174. published pre-print
Gun violence is a critical public health and safety concern in the United States. There is considerable variability in policy proposals meant to curb gun violence, ranging from increasing gun availability to deter potential assailants (e.g., concealed carry laws or arming school teachers) to restricting access to firearms (e.g., universal background checks or banning assault weapons). Many studies quantify policy impacts on gun violence using methods that combine state-level variation in the enactment of these policies with repeated measurements of the outcome of interest both before and after the policy change. In this chapter, we highlight the key features of the design and analysis of such evaluations, along with the associated opportunities and challenges, especially for studying gun policies in particular. We focus primarily on difference-in-differences and synthetic controls methods when there is staggered adoption of policies across jurisdictions, estimating the impacts of right-to-carry laws on violent crime as a case study.
Forest Kernel Balancing Weights: Outcome-guided features for causal inference. Statistics in Medicine, 45(20–22): e70720. published pre-print
AI-assisted variance reduction in randomized experiments. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), V.2: 21–32. published
Toward causal field evaluations of AI systems. Harvard Data Science Review, 8(2). published
Abortion bans and maternal, pregnancy-related, and pregnancy-associated mortality in 14 US states, 2016–2023: Estimated impacts amid substantial measurement challenges. American Journal of Public Health, 116(6): 819–828. published
Methodological considerations for investigating the impact of abortion restrictions on outcomes using aggregate panel data. American Journal of Epidemiology, 195(7): 2094–2099. published
Regularizing extrapolation in causal inference. Annual Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 300: 784–792. published pre-print
Many common estimators in machine learning and causal inference are linear smoothers, where the prediction is a weighted average of the training outcomes. Some estimators, such as ordinary least squares and kernel ridge regression, allow for arbitrarily negative weights, which improve feature imbalance but often at the cost of increased dependence on parametric modeling assumptions and higher variance. By contrast, estimators like importance weighting and random forests (sometimes implicitly) restrict weights to be non-negative, reducing dependence on parametric modeling and variance at the cost of worse imbalance. In this paper, we propose a unified framework that directly penalizes the level of extrapolation, replacing the current practice of a hard non-negativity constraint with a soft constraint and corresponding hyperparameter. We derive a worst-case extrapolation error bound and introduce a novel “bias-bias-variance” tradeoff, encompassing biases due to feature imbalance, model misspecification, and estimator variance; this tradeoff is especially pronounced in high dimensions, when positivity is poor. We then develop an optimization procedure that regularizes this bound while minimizing imbalance and outline how to use this approach as a sensitivity analysis for dependence on parametric modeling assumptions. We demonstrate the effectiveness of our approach through synthetic experiments and a real-world application, involving the generalization of randomized controlled trial estimates to a target population of interest.
Deconfounding scores and representation learning for causal effect estimation with weak overlap. Annual Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 300: 1288–1296. published pre-print
Augmented balancing weights as linear regression (with discussion). Journal of the Royal Statistical Society (Series B), 88(3): 699–723. published pre-print ★ Read paper, Royal Statistical Society Research Section
We provide a novel characterization of augmented balancing weights, also known as automatic debiased machine learning. These popular doubly robust estimators combine outcome modelling with balancing weights—weights that achieve covariate balance directly instead of estimating and inverting the propensity score. When the outcome and weighting models are both linear in some (possibly infinite) basis, we show that the augmented estimator is equivalent to a single linear model with coefficients that combine those of the original outcome model with those from unpenalized ordinary least-squares (OLS). Under certain choices of regularization parameters, the augmented estimator in fact collapses to the OLS estimator alone. We then extend these results to specific outcome and weighting models. We first show that the augmented estimator that uses (kernel) ridge regression for both outcome and weighting models is equivalent to a single, undersmoothed (kernel) ridge regression—implying a novel analysis of undersmoothing. When the weighting model is instead lasso–penalized, we demonstrate a familiar ‘double selection’ property. Our framework opens the black box on this increasingly popular class of estimators, bridges the gap between existing results on the semiparametric efficiency of undersmoothed and doubly robust estimators, and provides new insights into the performance of augmented balancing weights.
2025
Ridge boosting is both robust and efficient. Conference on Neural Information Processing Systems (NeurIPS) (spotlight). pre-print
Leveraging semantic similarity for experimentation with AI-generated treatments. Conference on Neural Information Processing Systems (NeurIPS). pre-print
Handling missing responses under cluster dependence with applications to language model evaluation. Conference on Neural Information Processing Systems (NeurIPS). pre-print
The Impossibility of Fair LLMs. Annual Meeting of the Association for Computational Linguistics (ACL), 105–120. published pre-print
US abortion bans and fertility. Journal of the American Medical Association, 333 (15): 1324-1332. published
Importance Abortion bans may lead to births among those who are unable to overcome barriers to abortion. The population-level effects of these policies, particularly their unequal impacts across subpopulations in the US, remain unclear.
Objective To assess heterogeneity in the association of abortion bans with changes in fertility in the US, within and across states.
Design, Setting, and Participants Drawing from birth certificate and US Census Bureau data from 2012 through 2023 for all 50 states and the District of Columbia, this study used a bayesian panel data model to evaluate state-by-subgroup-specific changes in fertility associated with complete or 6-week abortion bans in 14 US states. The average percent and absolute change in the fertility rate among females aged 15 through 44 years was estimated overall and by state, and within and across states by age, race and ethnicity, marital status, education, and insurance payer.
Exposure Complete or 6-week abortion ban.
Main outcome and Measures Fertility rate (births per 1000 reproductive-aged females) overall and by subgroups.
Results There were an estimated 1.01 (95% credible interval [CrI], 0.45-1.64) additional births above expectation per 1000 females aged 15 through 44 years (reproductive age) in states following adoption of abortion bans (60.55 observed vs 59.54 expected; 1.70% increase; 95% CrI, 0.75%-2.78%), equivalent to 22 180 excess births, with evidence of variation by state and subgroup. Estimated differences above expectation were largest for racially minoritized individuals (≈2.0%), unmarried individuals (1.79%), individuals younger than 35 years (≈2.0%), Medicaid beneficiaries (2.41%), and those without college degrees (high school diploma, 2.36%; some college, 1.58%), particularly in southern states. Differences in race and ethnicity and education across states explain most of the variability in the state-level association between abortion bans and fertility rates.
Conclusion and Relevance These findings provide evidence that fertility rates in states with abortion bans were higher than would have been expected in the absence of these policies, with the largest estimated differences among subpopulations experiencing the greatest structural disadvantages and in states with among the worst maternal and child health and well-being outcomes.
US abortion bans and infant mortality. Journal of the American Medical Association, 333(15): 1315-1323. published ★ Outstanding application award, American Statistical Association
Importance The impact of recent abortion bans on infant mortality is not fully understood. There is also limited evidence on how these bans may interact with long-standing racial and ethnic disparities in infant health.
Objective To examine the association of abortion bans with changes in infant mortality and to compare this association in racial and ethnic groups based on analyses within and across states.
Design, Setting, and Participants This population-based, serial, cross-sectional study used a bayesian panel model to examine infant mortality rates in 14 states that implemented complete or 6-week abortion bans and compared them with predictions of infant mortality rates based on pre-ban mortality rates and states without bans. Data included all live births and infant deaths from all 50 US states and the District of Columbia for 2012 through 2023. Models accounted for temporal trends and state-specific factors, with analyses stratified by race and ethnicity, timing of death, and cause of death.
Exposure Complete or 6-week abortion bans.
Main Outcome and Measures Infant mortality rate, analyzed overall and by subgroups.
Results The analysis found higher than expected infant mortality in states after adoption of abortion bans (observed vs expected, 6.26 vs 5.93 per 1000 live births; absolute increase, 0.33 [95% credible interval (CrI), 0.14-0.51]; relative increase, 5.60% [95% CrI, 2.43%-8.73%]). This resulted in an estimated 478 excess infant deaths in the 14 states with bans during the months affected by bans. The estimated increases were higher among non-Hispanic Black infants compared with other racial and ethnic groups, with 11.81 observed vs 10.66 expected infant deaths per 1000 live births, an absolute increase of 1.15 (95% CrI, 0.53-1.81) and relative increase of 10.98% (95% CrI, 4.87%-17.89%). The observed infant mortality rate due to congenital anomalies was 1.37 vs 1.24 expected (absolute increase, 0.13 [95% CrI, 0.04-0.21]; relative increase, 10.87% [95% CrI, 3.39%-18.08%]), while the rate not due to congenital anomalies was 4.89 observed vs 4.69 expected (absolute increase, 0.20 [95% CrI, 0.02-0.38]; relative increase, 4.23% [95% CrI, 0.49%-8.23%]). Texas had a dominant influence on the overall results and there were larger increases in southern vs nonsouthern states.
Conclusions US states that adopted abortion bans had higher than expected infant mortality after the bans took effect. The estimated relative increases in infant mortality were larger for deaths with congenital causes and among groups that had higher than average infant mortality rates at baseline, including Black infants and those in southern states.
Evaluation and incident prevention in an enterprise AI assistant. Annual Conference on Innovative Applications of Artificial Intelligence (IAAI), 39(28): 28931–28937. published
Estimating racial disparities in emergency general surgery. Journal of the Royal Statistical Society (Series A). 188(4): 1125-1148. published pre-print
Research documents that Black patients experience worse general surgery outcomes than White patients in the U.S. In this paper, we focus on an important but less-examined category: the surgical treatment of emergency general surgery (EGS) conditions, which refers to medical emergencies where the injury is internal, such as a burst appendix. Our goal is to assess racial disparities in outcomes after EGS treatment using administrative data. We also seek to understand the extent to which differences are attributable to patient-level risk factors vs. hospital-level factors, as well as to the decision to operate on EGS patients. To do so, we develop a class of linear weighting estimators that reweight White patients to have a similar distribution of baseline characteristics to Black patients. This framework nests many common approaches, including matching and linear regression, but offers important advantages over these methods in terms of controlling imbalance between groups, minimizing extrapolation, and reducing computation time. Applying this approach to the claims data, we find that disparities estimates that adjust for the admitting hospital are substantially smaller than estimates that adjust for patient baseline characteristics only, suggesting that hospital-specific factors are important drivers of racial disparities in EGS outcomes.
Addressing missing data due to COVID-19: Two early childhood case studies. Journal of Research on Educational Effectiveness. 18(1): 226-245. published
One part of COVID-19’s staggering impact on education has been to suspend or fundamentally alter ongoing education research projects. This article addresses how to analyze the simple but fundamental example of a multi-cohort study in which student assessment data for the final cohort are missing because schools were closed, learning was virtual, and/or assessments were canceled or inconsistently collected due to COVID-19. We argue that current best-practice recommendations for addressing missing data may fall short in such studies because the assumptions that underpin these recommendations are violated. We then provide a new, simple decision-making framework for empirical researchers facing this situation and provide two empirical examples of how to apply this framework drawn from early childhood studies, one a cluster randomized trial and the other a descriptive longitudinal study. Based on this framework and the assumptions required to address missing data, we advise against the standard recommendation of adjusting for missing outcomes (e.g., via imputation or weighting). Instead, we generally recommend changing the target quantity by restricting to fully observed cohorts or by pivoting to focusing on an alternative outcome.
Improving the estimation of site-specific effects and their distribution in multisite trials. Journal of Educational and Behavioral Statistics, 50(5): 731-764. published pre-print
2024
Reducing student absenteeism at scale: Translating social norms and attention interventions. What works, what doesn’t (and when): Case studies in applied behavioral science (pp. 290–312). University of Toronto Press. published
Towards representation learning for weighting problems in design-based causal inference. Conference on Uncertainty in Artificial Intelligence (UAI). published
Continuous treatment effects with surrogate outcomes. International Conference on Machine Learning (ICML). published pre-print
Temporal aggregation for the Synthetic Control Method. AEA Papers & Proceedings 114: 614–617. published pre-print
The synthetic control method (SCM) is a popular approach for estimating the impact of a treatment on a single unit with panel data. Two challenges arise with higher-frequency data (e.g., monthly versus yearly): (i) achieving excellent pretreatment fit is typically more challenging, and (ii) overfitting to noise is more likely. Aggregating data over time can mitigate these problems but can also destroy important signal. In this paper, we bound the bias for SCM with disaggregated and aggregated outcomes and give conditions under which aggregating tightens the bounds. We then propose finding weights that balance both disaggregated and aggregated series.
Randomization tests for peer effects in group formation experiments. Econometrica. 92(2): 567–590. published pre-print
Multilevel calibration weighting for survey data. Political Analysis. 32: 65–83. published pre-print
2023
Interpretable sensitivity analysis for balancing weights. Journal of the Royal Statistical Society (Series A). 186(4): 707–721. published pre-print
Variation in impacts of letters of recommendation on college admissions decisions: Approximate balancing weights for treatment effect heterogeneity in observational studies. Annals of Applied Statistics. 17(4): 2843–2864. published pre-print ★ Winner, JSM student paper competition, ASA sections on survey research, government statistics, and social statistics
In a pilot program during the 2016–17 admissions cycle, the University of California, Berkeley invited many applicants for freshman admission to submit letters of recommendation. This proved controversial within the university, with concerns that this change would further disadvantage applicants from disadvantaged groups. To inform this debate, we use this pilot as the basis for an observational study of the impact of submitting letters of recommendation on subsequent admission, with the goal of estimating how impacts vary across predefined subgroups. Understanding this variation is challenging in an observational setting because estimated impacts reflect both actual treatment effect variation and differences in covariate balance across groups. To address this, we develop balancing weights that directly optimize for “local balance” within subgroups while maintaining global covariate balance between treated and control units. Applying this approach to the UC Berkeley pilot study yields excellent local and global balance, unlike more traditional weighting methods, which fail to balance covariates within subgroups. We find that the impact of letters of recommendation increases with applicant strength. However, we find little average difference for applicants from disadvantaged groups, although this result is more mixed. In the end we conclude that soliciting letters of recommendation from a broader pool of applicants would not meaningfully change the composition of admitted undergraduates.
Is it who you are or where you are? Accounting for compositional differences in cross-site treatment effect variation. Journal of Educational and Behavioral Statistics. 48(4): 420-453. published pre-print
In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture “compositional” differences in the distributions of unit-level features as well as “contextual” differences in site-level features, including possible differences in program implementation. Our goal in this article is to adjust site-level estimates for differences in the distribution of observed unit-level features: If we can reweight (or “transport”) each site to have a common distribution of observed unit-level covariates, the remaining treatment effect variation captures contextual and unobserved compositional differences across sites. This allows us to make apples-to-apples comparisons across sites, parceling out the amount of cross-site effect variation explained by systematic differences in populations served. In this article, we develop a framework for transporting effects using approximate balancing weights, where the weights are chosen to directly optimize unit-level covariate balance between each site and the common target distribution. We first develop our approach for the general setting of transporting the effect of a single-site trial. We then extend our method to multisite trials, assess its performance via simulation, and use it to analyze a series of multisite trials of adult education and vocational training programs. In our application, we find that distributional differences are potentially masking cross-site variation. Our method is available in the balancer R package.
Using supervised learning to estimate inequality in the size and persistence of income shocks. ACM Conference on Fairness, Accountability, and Transparency (FAccT). published
Balancing weights for causal inference. Handbook of Matching and Weighting Adjustments for Causal Inference (Chapman and Hall/CRC). published
Estimating the effects of a California gun control program with Multitask Gaussian Processes. Annals of Applied Statistics. 17(2): 985-1016. published pre-print
Hospital quality risk standardization via approximate balancing weights. Annals of Applied Statistics. 17(2): 901-928. published pre-print
2022
Weak Separation in Mixture Models and Implications for Principal Stratification. Annual Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 151:5416-5458. published pre-print ★ Ten Have Award, Atlantic Causal Inference Conference, “for exceptionally creative or skillful research on causal inference”
Outcome Assumptions and Duality Theory for Balancing Weights. Annual Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 151:11037-11055 published pre-print
Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. BMJ Open. 12(1), e053820. published
Methods. We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.
Results. After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.
Discussion. The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.
A graph-theoretic approach to randomization tests of causal effects under general interference. Journal of the Royal Statistical Society (Series B). 84: 174-204. published pre-print ★ Best poster, Society for Political Methodology Annual Meeting
2021
Synthetic controls under staggered adoption. Journal of the Royal Statistical Society (Series B). 84: 351–381. published pre-print
Challenges with evaluating education policy using panel data during and after the COVID-19 pandemic. Journal of Research on Educational Effectiveness. 2021. 14(3): 668–675. published
The Augmented Synthetic Control Method. Journal of the American Statistical Association. 116(536): 1789–1803. published pre-print
The synthetic control method (SCM) is a popular approach for estimating the impact of a treatment on a single unit in panel data settings. The “synthetic control” is a weighted average of control units that balances the treated unit’s pretreatment outcomes and other covariates as closely as possible. A critical feature of the original proposal is to use SCM only when the fit on pretreatment outcomes is excellent. We propose Augmented SCM as an extension of SCM to settings where such pretreatment fit is infeasible. Analogous to bias correction for inexact matching, augmented SCM uses an outcome model to estimate the bias due to imperfect pretreatment fit and then de-biases the original SCM estimate. Our main proposal, which uses ridge regression as the outcome model, directly controls pretreatment fit while minimizing extrapolation from the convex hull. This estimator can also be expressed as a solution to a modified synthetic controls problem that allows negative weights on some donor units. We bound the estimation error of this approach under different data-generating processes, including a linear factor model, and show how regularization helps to avoid over-fitting to noise. We demonstrate gains from Augmented SCM with extensive simulation studies and apply this framework to estimate the impact of the 2012 Kansas tax cuts on economic growth. We implement the proposed method in the new augsynth R package.
Impact Evaluation of Coronavirus Disease 2019 Policy: A Guide to Common Design Issues. American Journal of Epidemiology. 190(11): 2474-2486. published pre-print ★ American Journal of Epidemiology Article of the Year
Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- analyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.
A trial emulation approach for policy evaluations with group-level longitudinal data. Epidemiology. 32(4): 533–540. published pre-print
To limit the spread of the novel coronavirus, governments across the world implemented extraordinary physical distancing policies, such as stay-at-home orders. Numerous studies aim to estimate the effects of these policies. Many statistical and econometric methods, such as difference-in-differences, leverage repeated measurements, and variation in timing to estimate policy effects, including in the COVID-19 context. Although these methods are less common in epidemiology, epidemiologic researchers are well accustomed to handling similar complexities in studies of individual-level interventions. Target trial emulation emphasizes the need to carefully design a nonexperimental study in terms of inclusion and exclusion criteria, covariates, exposure definition, and outcome measurement—and the timing of those variables. We argue that policy evaluations using group-level longitudinal (“panel”) data need to take a similar careful approach to study design that we refer to as policy trial emulation. This approach is especially important when intervention timing varies across jurisdictions; the main idea is to construct target trials separately for each treatment cohort (states that implement the policy at the same time) and then aggregate. We present a stylized analysis of the impact of state-level stay-at-home orders on total coronavirus cases. We argue that estimates from panel methods—with the right data and careful modeling and diagnostics—can help add to our understanding of many policies, though doing so is often challenging.
Overlap in observational studies with high-dimensional covariates. Journal of Econometrics. 221: 644-654. published ★ Best early career research presentation, European Causal Inference Meeting
Estimating causal effects under exogeneity hinges on two key assumptions: unconfoundedness and overlap. Researchers often argue that unconfoundedness is more plausible when more covariates are included in the analysis. Less discussed is the fact that covariate overlap is more difficult to satisfy in this setting. In this paper, we explore the implications of overlap in observational studies with high-dimensional covariates and formalize curse-of-dimensionality argument, suggesting that these assumptions are stronger than investigators likely realize. Our key innovation is to explore how strict overlap restricts global discrepancies between the covariate distributions in the treated and control populations. Exploiting results from information theory, we derive explicit bounds on the average imbalance in covariate means under strict overlap and show that these bounds become more restrictive as the dimension grows large. We discuss how these implications interact with assumptions and procedures commonly deployed in observational causal inference, including sparsity and trimming.
2020
Flexible sensitivity analysis for observational studies without observable implications. Journal of the American Statistical Association. 115(532): 1730-1746. published pre-print
Bayesian sensitivity analysis for offline policy evaluation. AAAI/ACM Conference on AI, Ethics, and Society (AIES), 64–70. published pre-print
2019
Assessing treatment effect variation in observational studies: Results from a data challenge. Observational Studies. 5: 21-35. published pre-print
Identifying and estimating principal causal effects in a multi-site trial of Early College High Schools. Annals of Applied Statistics. 13(3): 1348-1369. published pre-print ★ Ten Have Award, Atlantic Causal Inference Conference, “for exceptionally creative or skillful research on causal inference”
Exact conditional randomization tests for causal effects under interference. Biometrika. 106(2): 487-494. published pre-print
Many causal questions involve interactions between units, also known as interference, for example between individuals in households, students in schools, or firms in markets. In this paper we formalize the concept of a conditioning mechanism, which provides a framework for constructing valid and powerful randomization tests under general forms of interference. We describe our framework in the context of two-stage randomized designs and apply our approach to a randomized evaluation of an intervention targeting student absenteeism in the school district of Philadelphia. We show improvements over existing methods in terms of computational and statistical power.
Decomposing treatment effect variation. Journal of the American Statistical Association. 114(525): 304-317. published pre-print
Understanding and characterizing treatment effect variation in randomized experiments has become essential for going beyond the “black box” of the average treatment effect. Nonetheless, traditional statistical approaches often ignore or assume away such variation. In the context of randomized experiments, this article proposes a framework for decomposing overall treatment effect variation into a systematic component explained by observed covariates and a remaining idiosyncratic component. Our framework is fully randomization-based, with estimates of treatment effect variation that are entirely justified by the randomization itself. Our framework can also account for noncompliance, which is an important practical complication. We make several contributions. First, we show that randomization-based estimates of systematic variation are very similar in form to estimates from fully interacted linear regression and two-stage least squares. Second, we use these estimators to develop an omnibus test for systematic treatment effect variation, both with and without noncompliance. Third, we propose an R2-like measure of treatment effect variation explained by covariates and, when applicable, noncompliance. Finally, we assess these methods via simulation studies and apply them to the Head Start Impact Study, a large-scale randomized experiment. Supplementary materials for this article are available online.
2018
Analyzing two-stage experiments in the presence of interference. Journal of the American Statistical Association. 113(521): 41-55. published pre-print
Reducing student absences at scale by targeting parents’ misbeliefs. Nature Human Behaviour. 2(5): 335-342. published pre-print
Student attendance is critical to educational success, and is increasingly the focus of educators, researchers and policymakers. We report the results of a randomized experiment examining interventions targeting student absenteeism. Parents of 28,080 high-risk students in grades kindergarten to 12th grade received one of three personalized information treatments repeatedly throughout the school year or received no additional information (control). The most effective versions reduced chronic absenteeism by 10% or more, partly by correcting parents’ biased beliefs about their children’s total accumulated absences. The intervention reduced student absences comparably across grade levels, and reduced absences among untreated cohabiting students in treated households. This intervention is easy to scale and is more than one order of magnitude more cost effective than current absence-reduction best practices. Educational interventions that inform and empower parents, such as the one reported here, can complement more intensive student-focused absenteeism interventions.
Information, knowledge and attitudes: An Evaluation of the taxpayer receipt. Journal of Politics. 80(2): 701-706. published pre-print Media: Washington Post op-ed
To better understand the relationship between information and political knowledge, we evaluate an ambitious government initiative: the nationwide dissemination of “taxpayer receipts,” or personalized, itemized accounts of government spending, by the UK government in fall 2014. In coordination with the British tax authorities, we embedded a survey experiment in a nationally representative panel. We find that citizens became more knowledgeable about government spending because of our encouragement to read their receipt. Although baseline levels of political knowledge are indeed low, our findings indicate that individuals are capable of learning and retaining complex political information. However, even as citizens became more knowledgeable, we uncover no evidence that their attitudes toward government and redistribution changed concomitantly. The acquisition and retention of new information does not necessarily change attitudes. Our results have implications for citizens’ capacity to learn and research on the relationship between knowledge and attitudes.
New Findings on Impact Variation from the Head Start Impact Study: Informing the Scale-up of Early Childhood Programs. AERA Open. 4(2): 1-16. published
The Millennium Villages Project: a retrospective observational end-line evaluation. Lancet Global Health. 6(5): e500–e513. published
Background. The Millennium Villages Project (MVP) was a 10 year, multisector, rural development project, initiated in 2005, operating across ten sites in ten sub-Saharan African countries to achieve the Millennium Development Goals (MDGs). In this study, we aimed to estimate the project's impact, target attainment, and on-site spending.
Methods. In this endline evaluation of the MVP, we retrospectively selected comparison villages that best matched the project villages on possible confounding variables. Cross-sectional survey data on 40 outcomes of interest were collected from both the project and the comparison villages in 2015. Using these data, as well as on-site spending data collected during the project, we estimated project impacts as differences in outcomes between the project and comparison villages; target attainment as differences between project outcomes and prespecified targets; and on-site spending as expenditures reported by communities, donors, governments, and the project. Spending data were not collected in the comparison villages.
Findings. Averaged across the ten project sites, we found that impact estimates for 30 of 40 outcomes were significant (95% uncertainty intervals [UIs] for these outcomes excluded zero) and favoured the project villages. In particular, substantial effects were seen in agriculture and health, in which some outcomes were roughly one SD better in the project villages than in the comparison villages. The project was estimated to have no significant impact on the consumption-based measures of poverty, but a significant favourable impact on an index of asset ownership. Impacts on nutrition and education outcomes were often inconclusive (95% UIs included zero). Averaging across outcomes within categories, the project had significant favourable impacts on agriculture, nutrition, education, child health, maternal health, HIV and malaria, and water and sanitation. A third of the targets were met in the project sites. Total on-site spending decreased from US$132 per person in the first half of the project (of which $66 was from the MVP) to $109 per person in the second half of the project (of which $25 was from the MVP).
Interpretation. The MVP had favourable impacts on outcomes in all MDG areas, consistent with an integrated rural development approach. The greatest effects were in agriculture and health, suggesting support for the project's emphasis on agriculture and health systems strengthening. The project conclusively met one third of its targets.
Funding. The Open Society Foundations, the Islamic Development Bank, and the governments of Japan, South Korea, Mali, Senegal, and Uganda.
2017
Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness. 11(1): 133-162. published pre-print ★ Outstanding paper award, Journal of Research on Educational Effectiveness
Estimating treatment effects for subgroups defined by posttreatment behavior (i.e., estimating causal effects in a principal stratification framework) can be technically challenging and heavily reliant on strong assumptions. We investigate an alternative path: using bounds to identify ranges of possible effects that are consistent with the data. This simple approach relies on fewer assumptions and yet can result in policy-relevant findings. As we show, even moderately predictive covariates can be used to substantially tighten bounds in a straightforward manner. Via simulation, we demonstrate which types of covariates are maximally beneficial. We conclude with an analysis of a multisite experimental study of Early College High Schools. When examining the program's impact on students completing the ninth grade “on-track” for college, we find little impact for ECHS students who would otherwise attend a high-quality high school, but substantial effects for those who would not. This suggests a potential benefit in expanding these programs in areas primarily served by lower quality schools.
Principal score methods: Assumptions, extensions, and practical considerations. Journal of Educational and Behavioral Statistics. 42(6): 726-758. published pre-print
Researchers addressing posttreatment complications in randomized trials often turn to principal stratification to define relevant assumptions and quantities of interest. One approach for the subsequent estimation of causal effects in this framework is to use methods based on the “principal score,” the conditional probability of belonging to a certain principal stratum given covariates. These methods typically assume that stratum membership is as good as randomly assigned, given these covariates. We clarify the key assumption in this context, known as principal ignorability, and argue that versions of this assumption are quite strong in practice. We describe these concepts in terms of both one- and two-sided noncompliance and propose a novel approach for researchers to “mix and match” principal ignorability assumptions with alternative assumptions, such as the exclusion restriction. Finally, we apply these ideas to randomized evaluations of a job training program and an early childhood education program. Overall, applied researchers should acknowledge that principal score methods, while useful tools, rely on assumptions that are typically hard to justify in practice.
Algorithmic decision making and the cost of fairness. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). published Media: Washington Post op-ed
Algorithms are now regularly used to decide whether defendants awaiting trial are too dangerous to be released back into the community. In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk. To mitigate such disparities, several techniques have recently been proposed to achieve algorithmic fairness. Here we reformulate algorithmic fairness as constrained optimization: the objective is to maximize public safety while satisfying formal fairness constraints designed to reduce racial disparities. We show that for several past definitions of fairness, the optimal algorithms that result require detaining defendants above race-specific risk thresholds. We further show that the optimal unconstrained algorithm requires applying a single, uniform threshold to all defendants. The unconstrained algorithm thus maximizes public safety while also satisfying one important understanding of equality: that all individuals are held to the same standard, irrespective of race. Because the optimal constrained and unconstrained algorithms generally differ, there is tension between improving public safety and satisfying prevailing notions of algorithmic fairness. By examining data from Broward County, Florida, we show that this trade-off can be large in practice. We focus on algorithms for pretrial release decisions, but the principles we discuss apply to other domains, and also to human decision makers carrying out structured decision rules.
2016
Compared to what? Variation in the impacts of early childhood education by alternative care type. Annals of Applied Statistics. 2016. 10(3): 1245-1285. published ★ AERA award for outstanding publication in “Advances in Methodology” Media: Huffington Post op-ed; IRLE brief on Head Start research at UC Berkeley
Early childhood education research often compares a group of children who receive the intervention of interest to a group of children who receive care in a range of different care settings. In this paper, we estimate differential impacts of an early childhood intervention by alternative care type, using data from the Head Start Impact Study, a large-scale randomized evaluation. To do so, we utilize a Bayesian principal stratification framework to estimate separate impacts for two types of Compliers: those children who would otherwise be in other center-based care when assigned to control and those who would otherwise be in home-based care. We find strong, positive short-term effects of Head Start on receptive vocabulary for those Compliers who would otherwise be in home-based care. By contrast, we find no meaningful impact of Head Start on vocabulary for those Compliers who would otherwise be in other center-based care. Our findings suggest that alternative care type is a potentially important source of variation in early childhood education interventions.
Randomization inference for treatment effect variation. Journal of the Royal Statistical Society (Series B). 78(3): 655-671. published pre-print ★ Winner, JSM student paper competition, ASA sections on survey research, government statistics, and social statistics
Discouraged by peer excellence: Exposure to exemplary peer performance causes quitting. Psychological Science. 27(3): 365-374. published Media: NPR Hidden Brain/Science Friday, Education Week, Science Daily
2015 and earlier
Principal Stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation. 2015. 36(4): 514-531. published
Hierarchical models for causal effects. Emerging Trends in the Social and Behavioral Sciences. 2015. published
Genome-wide profiling of chromosome interactions in Plasmodium falciparum characterizes nuclear architecture and reconfigurations associated with antigenic variation. Molecular microbiology. 2013. 90(3): 519–537. published
Red state/blue state divisions in the 2012 presidential election. The Forum. 2013. 10(4): 127–131. published ★ Companion publication in the New York Times: “Red versus blue in a new light”, Nov. 12, 2012, with Andrew Gelman
Genome wide adaptations of Plasmodium falciparum in response to lumefantrine selective drug pressure. PloS One. 2012. 7(2): e31623. published
The combination therapy of the Artemisinin-derivative Artemether (ART) with Lumefantrine (LM) (Coartem®) is an important malaria treatment regimen in many endemic countries. Resistance to Artemisinin has already been reported, and it is feared that LM resistance (LMR) could also evolve quickly. Therefore molecular markers which can be used to track Coartem® efficacy are urgently needed. Often, stable resistance arises from initial, unstable phenotypes that can be identified in vitro. Here we have used the Plasmodium falciparum multidrug resistant reference strain V1S to induce LMR in vitro by culturing the parasite under continuous drug pressure for 16 months. The initial IC50 (inhibitory concentration that kills 50% of the parasite population) was 24 nM. The resulting resistant strain V1SLM, obtained after culture for an estimated 166 cycles under LM pressure, grew steadily in 378 nM of LM, corresponding to 15 times the IC50 of the parental strain. However, after two weeks of culturing V1SLM in drug-free medium, the IC50 returned to that of the initial, parental strain V1S. This transient drug tolerance was associated with major changes in gene expression profiles: using the PFSANGER Affymetrix custom array, we identified 184 differentially expressed genes in V1SLM. Among those are 18 known and putative transporters including the multidrug resistance gene 1 (pfmdr1), the multidrug resistance associated protein and the V-type H+ pumping pyrophosphatase 2 (pfvp2) as well as genes associated with fatty acid metabolism. In addition we detected a clear selective advantage provided by two genomic loci in parasites grown under LM drug pressure, suggesting that all, or some of those genes contribute to development of LM tolerance – they may prove useful as molecular markers to monitor P. falciparum LM susceptibility.
In vivo profiles show continuous variation between 2 cellular populations. Proceedings of the National Academy of Sciences. 2009. 106 (27), E71–E72. published
Statistical estimation of cell-cycle progression and lineage commitment in Plasmodium falciparum reveals a homogeneous pattern of transcription in ex vivo culture. Proceedings of the National Academy of Sciences. 2009. 106(18): 7559–7564. published
Software
augsynth
An R package for augmented synthetic controls, including staggered treatment adoption and multiple outcomes. With Eli Ben-Michael and Jesse Rothstein.
Package and installation · Single-unit example · Staggered-adoption example
multical
An R package for multilevel calibration weighting for survey data. Developed by Eli Ben-Michael, accompanying our joint work with Erin Hartman.
hettx
An R package for testing and characterizing treatment effect variation. With Peng Ding, Ben Fifield, and Luke Miratrix.
CRAN package and documentation · Treatment effect variation tutorial
Comments
Authors’ reply to the Discussion of ‘Augmented balancing weights as linear regression’ by Bruns-Smith et al. David Bruns-Smith, Oliver Dukes, Avi Feller, and Betsy Ogburn. Journal of the Royal Statistical Society (Series B), 88(3): 748–755. published
Invited discussion of ‘Matrix completion when missing is not at random and its applications in causal panel data models’. Eli Ben-Michael and Avi Feller. Journal of the American Statistical Association, forthcoming. pre-print
Comment on ‘Causal Inference Using Invariant Prediction: Identification and Confidence Intervals’ by Peters, Buehlmann, and Meinshausen. Peng Ding and Avi Feller. Journal of the Royal Statistical Society (Series B) 2016. 78(5): 994-995. published
Comment on ‘How to find an appropriate clustering for mixed type variables with application to socio-economic stratification’ by Hennig and Liao. Avi Feller and Edo Airoldi. Journal of the Royal Statistical Society (Series C) 2013. 62(3): 347–348. published