arXiv subjects

Nicholas Cain

Publications and source records attributed to Nicholas Cain.

5 recordsLinked to original sources

arXiv2021

Soft Calibration Objectives for Neural Networks

Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross entropy baseline on CIFAR-100. When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibration-sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross entropy loss and post-hoc recalibration methods.

cs.LG↗

arXiv2020

Mitigating Bias in Calibration Error Estimation

For an AI system to be reliable, the confidence it expresses in its decisions must match its accuracy. To assess the degree of match, examples are typically binned by confidence and the per-bin mean confidence and accuracy are compared. Most research in calibration focuses on techniques to reduce this empirical measure of calibration error, ECE_bin. We instead focus on assessing statistical bias in this empirical measure, and we identify better estimators. We propose a framework through which we can compute the bias of a particular estimator for an evaluation data set of a given size. The framework involves synthesizing model outputs that have the same statistics as common neural architectures on popular data sets. We find that binning-based estimators with bins of equal mass (number of instances) have lower bias than estimators with bins of equal width. Our results indicate two reliable calibration-error estimators: the debiased estimator (Brocker, 2012; Ferro and Fricker, 2012) and a method we propose, ECE_sweep, which uses equal-mass bins and chooses the number of bins to be as large as possible while preserving monotonicity in the calibration function. With these estimators, we observe improvements in the effectiveness of recalibration methods and in the detection of model miscalibration.

cs.LG↗

arXiv2016

Feedback through graph motifs relates structure and function in complex networks

In physics, biology and engineering, network systems abound. How does the connectivity of a network system combine with the behavior of its individual components to determine its collective function? We approach this question for networks with linear time-invariant dynamics by relating internal network feedbacks to the statistical prevalence of connectivity motifs, a set of surprisingly simple and local statistics of connectivity. This results in a reduced order model of the network input-output dynamics in terms of motifs structures. As an example, the new formulation dramatically simplifies the classic Erdos-Renyi graph, reducing the overall network behavior to one proportional feedback wrapped around the dynamics of a single node. For general networks, higher-order motifs systematically provide further layers and types of feedback to regulate the network response. Thus, the local connectivity shapes temporal and spectral processing by the network as a whole, and we show how this enables robust, yet tunable, functionality such as extending the time constant with which networks remember past signals. The theory also extends to networks composed from heterogeneous nodes with distinct dynamics and connectivity, and patterned input to (and readout from) subsets of nodes. These statistical descriptions provide a powerful theoretical framework to understand the functionality of real-world network systems, as we illustrate with examples including the mouse brain connectome.

q-bio.NC↗

arXiv2012

Impact of correlated neural activity on decision making performance

Stimulus from the environment that guides behavior and informs decisions is encoded in the firing rates of neural populations. Each neuron in the populations, however, does not spike independently: spike events are correlated from cell to cell. To what degree does this apparent redundancy impact the accuracy with which decisions can be made, and the computations that are required to optimally decide? We explore these questions for two illustrative models of correlation among cells. Each model is statistically identical at the level of pairs cells, but differs in higher-order statistics that describe the simultaneous activity of larger cell groups. We find that the presence of correlations can diminish the performance attained by an ideal decision maker to either a small or large extent, depending on the nature of the higher-order interactions. Moreover, while this optimal performance can in some cases be obtained via the standard integration-to-bound operation, in others it requires a nonlinear computation on incoming spikes. Overall, we conclude that a given level of pairwise correlations--even when restricted to identical neural populations--may not always indicate redundancies that diminish decision making performance.

q-bio.NC↗

arXiv2011

Neural integrators for decision making: A favorable tradeoff between robustness and sensitivity

A key step in many perceptual decision tasks is the integration of sensory inputs over time, but fundamental questions remain about how this is accomplished in neural circuits. One possibility is to balance decay modes of membranes and synapses with recurrent excitation. To allow integration over long timescales, however, this balance must be precise; this is known as the fine tuning problem. The need for fine tuning can be overcome via a ratchet-like mechanism, in which momentary inputs must be above a preset limit to be registered by the circuit. The degree of this ratcheting embodies a tradeoff between sensitivity to the input stream and robustness against parameter mistuning. The goal of our study is to analyze the consequences of this tradeoff for decision making performance. For concreteness, we focus on the well-studied random dot motion discrimination task. For stimulus parameters constrained by experimental data, we find that loss of sensitivity to inputs has surprisingly little cost for decision performance. This leads robust integrators to performance gains when feedback becomes mistuned. Moreover, we find that substantially robust and mistuned integrator models remain consistent with chronometric and accuracy functions found in experiments. We explain our findings via sequential analysis of the momentary and integrated signals, and discuss their implication: robust integrators may be surprisingly well-suited to subserve the basic function of evidence integration in many cognitive tasks.

q-bio.NC↗