Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2009.00729

Evaluating Catchment Models as Multiple Working Hypotheses: on the Role of Error Metrics, Parameter Sampling, Model Structure, and Data Information Content

Sina Khatami·Timothy John Peterson·Murray C Peel·Andrew Western
View original source ↗Read source document ↗Export metadata ↓

Abstract

To evaluate models as hypotheses, we developed the method of Flux Mapping to construct a hypothesis space based on dominant runoff generating mechanisms. Acceptable model runs, defined as total simulated flow with similar (and minimal) model error, are mapped to the hypothesis space given their simulated runoff components. In each modeling case, the hypothesis space is the result of an interplay of factors: model structure and parameterization, chosen error metric, and data information content. The aim of this study is to disentangle the role of each factor in model evaluation. We used two model structures (SACRAMENTO and SIMHYD), two parameter sampling approaches (Latin Hypercube Sampling of the parameter space and guided-search of the solution space), three widely used error metrics (Nash-Sutcliffe Efficiency - NSE, Kling-Gupta Efficiency skill score - KGEss, and Willmott refined Index of Agreement - WIA), and hydrological data from a large sample of Australian catchments. First, we characterized how the three error metrics behave under different error types and magnitudes independent of any modeling. We then conducted a series of controlled experiments to unpack the role of each factor in runoff generation hypotheses. We show that KGEss is a more reliable metric compared to NSE and WIA for model evaluation. We further demonstrate that only changing the error metric -- while other factors remain constant -- can change the model solution space and hence vary model performance, parameter sampling sufficiency, and or the flux map. We show how unreliable error metrics and insufficient parameter sampling impair model-based inferences, particularly runoff generation hypotheses.

Explore related subjects

stat.MEphysics.data-anphysics.geo-phstat.AP

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sina Khatami, Timothy John Peterson, Murray C Peel, Andrew Western. 2020-09-01. Evaluating Catchment Models as Multiple Working Hypotheses: on the Role of Error Metrics, Parameter Sampling, Model Structure, and Data Information Content. https://doi.org/10.1002/essoar.10504066.1

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Modeling cyclostationarity in time series using ASCA

Modern data analysis across diverse disciplines increasingly relies on time series. Many of these datasets exhibit cyclostationarity, where patterns approximately repeat in a regular manner, often across multiple time scales, such as daily, weekly or yearly cycles. In this context, statistical inference is essential to distinguish genuine underlying effects from random variability. While tools like Analysis of Variance (ANOVA) provide such inference, they often lack interpretability and struggle with the complexities of multivariate data. To address these limitations, we propose a unified pipeline for the exploratory analysis of cyclostationary times series using ANOVA Simultaneous Component Analysis (ASCA). ASCA is an extension of ANOVA that is able to work in both univariate and multivariate cases. Combining inference with the visualization capabilities of Principal Component Analysis (PCA), ASCA provides powerful options for interpretability. ASCA's capabilities have been well-established in the analysis of experimental data, but they remain largely unexplored for observational data like time series. Our workflow introduces an algorithmic approach to modeling time-dependent data using ASCA, enabling control over multiple cyclostationary time scales while also accounting for the specific challenges of this type of data, such as autocorrelation. Furthermore, we observed that ASCA provides a better separation of variability across factors than ANOVA in unbalanced designs due to its multivariate nature. We demonstrate the efficacy of this methodology through two real-world case studies: water temperature trends in mountain lakes in Sierra Nevada, Spain, and airborne pollen trends over 30 years recorded in the city of Granada, Spain.

stat.ME↗
arXiv2026

Distributionally balanced sampling designs via tactical configurations

Distributionally balanced sampling designs are low-discrepancy probability designs obtained by minimizing the expected discrepancy between the auxiliary-variable distribution of a random sample and the target population distribution. Existing constructions rely on circular population sequences, which restrict the design space by forcing samples to be contiguous blocks of a sequence. We propose a new construction based on tactical configurations that removes this topological constraint. The resulting designs are fixed-size and have equal inclusion probabilities. We develop both a simple initialization valid for arbitrary population and sample sizes and a spatial initialization that yields a lower initial expected discrepancy, together with a simulated annealing algorithm for optimization within this class. In simulations and empirical examples, the proposed method outperforms state-of-the-art alternatives in terms of distributional fit and balance.

stat.ME↗
arXiv2026

Bayesian Quantile Regression for Misclassified Binary Data with an Application to Spousal Violence Reporting

Survey responses on socially undesirable behaviors, such as self-reported spousal violence, are often subject to underreporting due to social stigma, fear of retaliation, and other reporting pressures. When such data are analyzed using standard econometric models that focus on conditional means, such as probit and logit models, the resulting estimates are likely to be biased and can obscure heterogeneity in covariate effects. To address these challenges, we propose a Bayesian binary quantile regression framework that accounts for misclassification and provides quantile-specific effects for the latent true response. The framework incorporates false-negative and false-positive probabilities to capture reporting errors and employs a novel partially collapsed Gibbs sampler for estimation. We also discuss the computation of covariate effects and marginal likelihood for Bayesian model comparison. Simulation studies under various settings (prior effective sample size, misclassification rates, and prior distribution) show that accounting for misclassification improves inference across quantiles relative to models that ignore reporting errors. We apply the framework to women's self-reported spousal violence and find that underreporting of spousal violence exceeds overreporting across quantiles, while model comparisons using marginal likelihood generally favor the quantile model with misclassification and yield different conclusions about the determinants of reported violence.

stat.ME↗