An Explainable Approach to Document-level Translation Evaluation with Topic Modeling

Hyeokmin Lee; Youngkyu Kim; Byounghyun Yoo

Home/Archive/arXiv

arXiv · 2604.20334

An Explainable Approach to Document-level Translation Evaluation with Topic Modeling

Hyeokmin Lee·Youngkyu Kim·Byounghyun Yoo

View original source ↗Read source document ↗Export metadata ↓

Abstract

The advent of NMT has expanded the scope of translation beyond isolated sentences, enabling context to be preserved across paragraphs and documents. However, current evaluation metrics largely remain restricted to the sentence level and typically depend on reference translations. Without references, existing metrics cannot provide a clear basis for their quality assessments. To address these limitations, we propose an evaluation framework that independently extracts and compares latent topic structures within source and translated texts. This framework utilises various topic modelling techniques, including LSA, LDA and BERTopic, to achieve this. Our methodology captures statistical frequency information and semantic context, providing a comprehensive evaluation of the entire document. It aligns key topic tokens across languages using a bilingual dictionary and quantifies thematic consistency via cosine similarity. This allows us to evaluate how faithfully the translation maintains the thematic integrity of the source text, even in the absence of reference translations. To this end, we used a large scale dataset of 9.38 million Korean to English sentence pairs from AI Hub, which includes pre evaluated BLEU scores. We also calculated CometKiwi, a state of the art, reference free metric for this dataset, in order to conduct a comparative analysis with our proposed, topic based framework. Through this analysis, we confirmed that, unlike existing metrics, our framework evaluates the differentiated attribute of document level thematic units. Furthermore, visualising the key tokens that underpin the quantitative evaluation score provides clear insight into translation quality. Consequently, this study contributes to effectively complementing the existing translation evaluation system by proposing a new metric that intuitively identifies whether the document's theme has been preserved.

Explore related subjects

cs.CE

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Hyeokmin Lee, Youngkyu Kim, Byounghyun Yoo. 2026-04-22. An Explainable Approach to Document-level Translation Evaluation with Topic Modeling. https://arxiv.org/abs/2604.20334

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Dimension Bridging for 3D RANS with Neural Network Accelerated Gaussian Functional Regression

In many computational science and engineering problems, repeatedly solving fully resolved physics-based models to design for a quantity of interest (QoI) can quickly become intractable, requiring the use of low-fidelity models to predict the same QoI but introduce errors where some features are neglected or are otherwise inaccurately resolved. We use Gaussian Functional Regression (GFR) to learn a correction to a 2D Reynolds-Averaged Navier-Stokes (RANS) model to predict the aerodynamic coefficients from a 3D RANS model. This model pair has a disparity in the governing physics from the reduced dimensionality, a previously unexplored application for GFR. Empirically, our results show that with a proper choice of low-dimensional (LD) model, the proposed kernel allows for the use of fewer high-dimensional (HD) evaluations to regress a response surface to the same level of accuracy as standard stationary kernels. Moreover, the new kernel provides more informative uncertainty quantification, which we show is advantageous when used to drive an adaptive sampling algorithm. Finally, we propose a novel neural network accelerated kernel, which we show offers predictions in good agreement while speeding up evaluations by millions of times in wall clock measurements, bringing the computational budget within the real-time regime.

cs.CE↗

arXiv2026

SabreAgent: Language Models at Design Time for Lost-Sales Inventory Control

SabreAgent uses a language model at design time to construct two components for lost-sales inventory control: a product-specific seasonal prior and a validation-selected capped base-stock policy family. During operation, statistical forecasting and inventory optimization use these frozen artifacts to determine orders, with zero language-model calls. We evaluate the approach on the $1{,}320$ instances of InventoryBench. Under the benchmark's cost assumptions, the operations-research core draws on a zero-lead-time optimality result and a projected-inventory rule for positive deterministic lead times. The latter computes replenishment shortfalls by propagating inventory using sales along simulated demand paths. The seasonal prior adds forecast variants alongside the original forecaster, and the selected policy family handles stochastic lead times with order destruction. SabreAgent scores $0.6311$, compared with $0.5380$ for the strongest published baseline, and ranks first in all six benchmark cells. Ablations attribute most of the gain to the OR core. In the paired analysis, the seasonal component adds $1.79\%$ across the three real-data cells, and the search component adds $2.3\%$ across the two stochastic-lead-time cells. These results demonstrate how model-generated priors and policy structure can improve an OR controller through design-time use.

cs.CE↗

arXiv2026

Hierarchical Multi-Task Learning with Liquidity-Aware Signals for Stock Forecasting

Stock price forecasting is a long-standing challenge in computational finance, driven by the inherent randomness of markets and complex temporal patterns. While recent deep-learning models have raised forecasting accuracy by jointly modeling inter-stock and temporal price dynamics, they conflate inter-stock relationships with intra-stock temporal dependencies and focus solely on the univariate objective of price movement. To address these limitations, we propose LiMT, a Hierarchical Multi-Task Learning framework that integrates liquidity-aware signals for stock price forecasting. LiMT employs a Market Regime Encoder (MRE) module that first extracts contemporaneous cross-stock dependencies, then models each stock's temporal dynamics, yielding a unified latent state. Building on this latent state, we introduce a Liquidity-Driven Learning (LDL) module, a mixture-of-experts architecture that features cross-task gating mechanisms to jointly predict price movement, volatility, and trading volume. We further design an Adaptive Portfolio Optimization (APO) mechanism that converts multi-task forecasts into executable portfolio weights under transaction-cost and liquidity constraints. Extensive experiments on the CSI300 and CSI500 benchmarks show that LiMT performs best among strong neural and tree-based baselines across the reported metrics. In realistic CSI300 backtests, APO improves annualized return from 3.99% to 10.01% and Sharpe ratio from 1.22 to 1.86 over equal weighting, showing that the multi-task forecasts translate into deployable portfolio gains.

cs.CE↗