Search arXiv⌕ Search

arXiv subjects

Jason Hartline

Publications and source records attributed to Jason Hartline.

At least 19 recordsLinked to original sources

Robust Analysis of First-Price Auctions with Large Supply

This paper performs a robust analysis of welfare (i.e., the price of anarchy) in multi-unit first-price auctions with independent private values and unit-demand bidders. We employ a smoothness framework (see \citealp{ST-13}) to analyze the price of anarchy when supply is large. The main barrier to analysis in this framework is a worst-case bid distribution that establishes a lower bound on the price of anarchy of $1- \frac{1}{e}$. We show that such worst-case distributions cannot arise in equilibrium when agents' values satisfy a mild density lower bound condition. This characterization leads to novel asymptotic lower bounds. In particular, the price of anarchy for $ε$-approximate equilibrium as the number of units goes to infinity is at least $1-Θ\left(ε\log \left(\frac{1}ε\right)\right)$. Our analysis relies on a key structural property of equilibrium bid distributions: for a welfare analysis in any multi-unit first-price auction, it is without loss to analyze (approximate) equilibria with no atoms. In addition to this, we establish continuity and monotonicity properties of Bayes-Nash equilibria. We believe these results are of independent interest.

cs.GT↗

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Large language models (LLMs) are increasingly proposed as decision assistants who must reason probabilistically from available evidence under explicit decision costs. We propose a decision-theoretic framework that decomposes LLMs' decision loss into two components: forming accurate beliefs from provided evidence and translating those beliefs into actions that optimize a provided utility function. Using a synthetic benchmark with known ground truth, we apply the decomposition to characterize probabilistic reasoning in frontier and open-sourced models. We further evaluate whether RL interventions targeting beliefs, decisions, or both improve these components across three domains, whether improvements transfer across components and elicitation formats, and whether decision performance can improve without improvement in belief formation. We find that targeting one component of probabilistic reasoning redistributes decision loss, improving the target without necessarily transferring to others, and that jointly targeting belief formation and decision-making improves both but hinges on matched formats between training and evaluation.

cs.AI↗

Selling to Non-linear Agents: A Demand-Based Auction

Auctions allocate scarce goods, from car licenses to online advertising, to buyers whose utilities are often not linear in money, as with budgets or risk aversion. We ask how a seller should sell to such buyers and what learning their preferences is worth. We propose demand-based auctions, which elicit demand curves and are dominant-strategy incentive compatible for any non-linear utilities. The revenue-maximizing one takes familiar second-price forms and is max-min optimal when the seller observes only demand curves. Under standard assumptions, learning risk attitudes beyond demand curves adds relatively little revenue, as does learning budgets unless they correlate with values.

cs.GT↗

Truthful Calibration Measures for Sequential Prediction

Calibration requires probabilistic reports to be conditionally unbiased and reliably interpretable as probabilities. A calibration measure assigns numerical error to miscalibrated reports. Haghtalab et al. (2024) proposed an approximately truthful calibration measure for online prediction, leaving open whether exact truthfulness is compatible with completeness and soundness. We resolve this question negatively for sequential binary prediction: exact truthfulness is incompatible with completeness and soundness, even for independent outcomes. We then show that this impossibility is specific to exact truthfulness. On the positive side, we give two general reductions that transform a base calibration measure satisfying a uniform completeness condition into additively and multiplicatively approximately truthful measures, respectively. Applying the multiplicative reduction, for every $0<\varepsilon<1$ we construct a sound and complete calibration measure that is $1+\exp\left(-Ω(T^{1-\varepsilon})\right)$-multiplicatively truthful. This improves the approximate truthfulness guarantee of Haghtalab et al. (2024). We also prove a general transfer theorem showing that the uniform completeness of one calibration measure transfers to another when they are quantitatively related. Using this result, we show that every consistent calibration measure (Blasiok et al., 2023) satisfies uniform completeness and can serve as the base calibration measure to achieve $1 + o(1)$ multiplicative truthfulness.

cs.DS↗

A Geometric Analysis of Gains from Trade

We provide a geometric proof that the random proposer mechanism is a $4$-approximation to the first-best gains from trade in bilateral exchange. We then refine this geometric analysis to recover the $3.147$-approximation guarantee previously established by Fei (2022).

cs.GT↗

Robustness in Mechanism Design

The article reviews the robust design and analysis of auctions, focusing on the predominant paradigm in the computer science literature, which quantifies robustness by the worst-case, over environments, ratio between the robust auction's performance and that of an optimal auction for the environment. Applications include robust analysis of canonical auctions such as the first-price auction for the welfare objective, prior-robust auction design for the revenue objective, and prior-robust design of exchange mechanisms for the gains-from-trade objective. Finally, the revelation principle is not without loss in robust mechanism design. The potential loss can be quantified by the revelation gap, the fraction of the optimal robust guarantee that is lost when restricting from all mechanisms to revelation mechanisms.

econ.TH↗

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.

cs.DL↗

Scale-robust Auctions

We study auctions that are robust at any scale, i.e., they can be applied to sell both expensive and cheap items and achieve the best multiplicative approximation of the optimal revenue in the worst case. We first show that it is without loss of optimality to restrict attention to scale-invariant mechanisms whenever the family of possible distributions is closed under every positive rescaling. This conclusion uses no regularity or other distributional shape restriction. We then solve the two-agent, single-item problem with values drawn i.i.d. from an unknown regular distribution when only a high value bidder can receive a positive allocation. The robustly optimal mechanism in this class randomizes between the second-price auction, with probability approximately 0.806, and a markup auction that offers the item to the highest-valued bidder at a price equal to 2.447 times the second-highest value. Its worst-case approximation ratio is approximately 1.907.

cs.GT↗

Sample Complexity of Peer Prediction

Peer prediction seeks to incentivize agents to truthfully report an observed signal by rewarding joint sets of reports without observing a ground truth. Following the generalization of information-theoretic mutual information introduced in Kong and Schoenebeck (2019), we call a function of a joint distribution over signals a mutual information when it is non-negative and disincentivizes garbling reports for all information structures. An unbiased estimator for a mutual information takes some number of samples from the distribution and returns rewards for both agents, such that the expected reward is equal to the mutual information. We seek to characterize the set of mutual informations with unbiased estimators for a given number of samples. We show that for three or fewer sampled report pairs, the only mutual information with an unbiased estimator is trivially zero, and for four or five samples with a binary report space, the Determinant Mutual Information (DMI) of Kong (2024) is the unique mutual information (up to a scalar multiple). We further show that DMI ceases to be unique at six samples. We provide an improved estimator of DMI for any given number of samples and characterize its convergence rate. We also examine mutual information estimators that accept a randomized number of samples. First, we show that mutual information estimators on an ex-ante bounded number of samples (termed "stop-short estimators") can achieve a lower variance than an equivalent fixed-sample estimator (for DMI). Second, we introduce the class of scoring-rule-based mutual informations and identify in this family a mutual information that can be estimated with under three samples in expectation.

cs.IT↗

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing it with a reference response. The reliability of an evaluation metric is usually judged by its statistical correlation with human ratings. However, as these metrics are increasingly used as optimization objectives, correlation alone is no longer sufficient: agents may strategically game the evaluation metric. We study this issue through two complementary notions of alignment. A metric is statistically aligned if it correlates with human ratings and strategically aligned if it resists perturbations that do not add task-relevant information. We make two contributions. First, we propose test principles for reference-based metrics consisting of human-rating correlation, degradation sensitivity, and manipulation robustness. These principles evaluate whether a metric agrees with human judgments, penalizes low-effort information loss, and resists strategic score inflation. Second, we develop a unified design framework for mutual-information-based metrics that decomposes existing and new metrics into four choices: information measure, estimation method, text representation, and prediction mechanism. Across peer review, summarization, and question answering, we find that strong human-rating correlation does not imply strategic alignment: LLM-as-a-Judge achieves high correlation but is susceptible to manipulation. In contrast, mutual-information-based metrics substantially improve manipulation robustness. Our framework also uncovers a new metric that achieves the strongest overall robustness in our experiments while remaining competitive on human-rating correlation.

cs.AI↗

Truthful Calibration Errors for Multi-Class Prediction

Calibrated predictions are useful because their numerical values can be interpreted as probabilities. Calibration errors are therefore widely used to evaluate, compare, and tune probabilistic predictors. Recently, Haghtalab et al. (2024) introduced an additional requirement for such measures: truthfulness. A calibration measure is truthful if a predictor minimizes its expected measured error by reporting the true conditional label distribution. Many standard empirical calibration errors are non-truthful: a predictor may appear better calibrated by distorting its probabilities rather than reporting them truthfully. We study the practical role of truthfulness for calibration measurement in multiclass prediction. First, we introduce perfectly truthful calibration errors for multidimensional linear properties of the label distribution, generalizing the truthful calibration error for binary predictions in Hartline et al. (2025). This framework includes full multiclass calibration and classwise calibration. We also identify a truthful correction for confidence calibration. Second, we characterize the decision-theoretic implications of these truthful errors. For calibrated predictors, truthful calibration errors preserve the Blackwell dominance: a more informative calibrated predictor receives no larger expected error. Third, we show that this decision-theoretic interpretation explains and mitigates the well-observed ranking robustness problem of binned calibration errors. Empirically, non-truthful confidence-based errors can reverse model rankings when the number of bins changes, while our truthful errors give more stable rankings across binning choices.

cs.LG↗

A Perfectly Truthful Calibration Measure

Calibration requires that predictions are conditionally unbiased and, therefore, reliably interpretable as probabilities. A calibration measure quantifies how far a predictor is from perfect calibration. As introduced by Haghtalab et al. (2024), a calibration measure is truthful if it is minimized in expectation when a predictor outputs the ground-truth probabilities. Predicting the true probabilities guarantees perfect calibration, but in reality, when calibration is evaluated on a random sample, all known calibration measures incentivize predictors to lie in order to appear more calibrated. Such lack of truthfulness motivated Haghtalab et al. (2024) and Qiao and Zhao (2025) to construct approximately truthful calibration measures in the sequential prediction setting, but no perfectly truthful calibration measure was known to exist even in the more basic batch setting. We design a simple, perfectly and strictly truthful, sound and complete calibration measure in the batch setting: averaged two-bin calibration error (ATB). ATB is quadratically related to two existing calibration measures: the smooth calibration error smCal and the lower distance to calibration distCal. The simplicity in our definition of ATB makes it efficient and straightforward to compute, allowing us to give the first linear-time calibration testing algorithm, improving a result of Hu et al. (2024). We also introduce a general recipe for constructing truthful measures based on the variance additivity of independent random variables, which proves the truthfulness of ATB as a special case and allows us to construct other truthful calibration measures such as quantile-binned l_2-ECE.

cs.LG↗

Revenue Non-monotonicity in Matching Markets

The Vickrey-Clarke-Groves (VCG) mechanism is infamously revenue non-monotone in combinatorial auctions. I.e., when a buyer increases their value for a bundle of items, the total auction revenue may decrease. Combinatorial auctions exhibit complementarities which broadly result in complexities in auction theory. This brief note shows that non-monotonicity in multi-item auctions is not a result of complementarities, and in fact, VCG is revenue non-monotone even in matching markets.

econ.TH↗

ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making

Multi-agent decision pipelines can outperform single agent workflows when complementarity holds, i.e., different agents bring unique information to the table to inform a final decision. We propose ComplLLM, a post-training framework based on decision theory that fine-tunes a decision-assistant LLM using complementary information as reward to output signals that complement existing agent decisions. We validate ComplLLM on synthetic and real-world tasks involving domain experts, demonstrating how the approach recovers known complementary information and produces plausible explanations of complementary signals to support downstream decision-makers.

cs.AI↗

Clarification of `Algorithmic Collusion without Threats'

This brief note clarifies that the scenario described in Arunachaleswaran et al. (2025) -- titled `Algorithmic Collusion without Threats' -- is not one of collusion, but one where one player is behaving non-competitively and the other is behaving competitively.

cs.GT↗

Explaining and Improving Information Complementarities in Multi-Agent Decision-making

Multiple agents are increasingly combined to make decisions with the expectation of achieving complementary performance, where the decisions they make together outperform those made individually. However, knowing how to improve the performance of collaborating agents requires knowing what information and strategies each agent employs. With a focus on human-AI pairings, we contribute a decision-theoretic framework for characterizing the value of information. By defining complementary information, our approach identifies opportunities for agents to better exploit available information in AI-assisted decision workflows. We present a novel explanation technique (ILIV-SHAP) that adapts SHAP explanations to highlight human-complementing information. We validate the effectiveness of our framework and ILIV-SHAP through a study of human-AI decision-making, and demonstrate the framework on examples from chest X-ray diagnosis and deepfake detection. We find that presenting ILIV-SHAP with AI predictions leads to reliably greater reductions in error over non-AI assisted decisions more than vanilla SHAP.

cs.AI↗

The Economics of No-regret Learning Algorithms

A fundamental challenge for modern economics is to understand what happens when actors in an economy are replaced with algorithms. Like rationality has enabled understanding of outcomes of classical economic actors, no-regret can enable the understanding of outcomes of algorithmic actors. This review article covers the classical computer science literature on no-regret algorithms to provide a foundation for an overview of the latest economics research on no-regret algorithms, focusing on the emerging topics of manipulation, statistical inference, and algorithmic collusion.

econ.TH↗

ElicitationGPT: Text Elicitation Mechanisms via Language Models

Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information. This paper develops mechanisms for scoring elicited text against ground truth text by reducing the textual information elicitation problem to a forecast elicitation problem, via domain-knowledge-free queries to a large language model (specifically ChatGPT), and empirically evaluates their alignment with human preferences. Our theoretical analysis shows that the reduction achieves provable properness via black-box language models. The empirical evaluation is conducted on peer reviews from a peer-grading dataset, in comparison to manual instructor scores for the peer reviews. Our results suggest a paradigm of algorithmic artificial intelligence that may be useful for developing artificial intelligence technologies with provable guarantees.

cs.AI↗