Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.05099

Improving Language Identification for Code-Switched Utterances with Integer Linear Programming

Joanna Radoła·Josep Maria Crego·François Yvon
View original source ↗Read source document ↗Export metadata ↓

Abstract

Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.

Explore related subjects

cs.CL

Keep this discovery

BibTeXRIS

Joanna Radoła, Josep Maria Crego, François Yvon. 2026-09-04. Improving Language Identification for Code-Switched Utterances with Integer Linear Programming. https://arxiv.org/abs/2609.05099

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.CL subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

CyrillicQA: The Influence of Phonetically Encoded Secret Language on LLM Performance

Due to the selection of their training data, large language models (LLMs) perform best on standard-language inputs from languages using the Latin alphabet with large speaker populations, while disadvantaging other language varieties. Nevertheless, they can also be a versatile tool for preserving precisely such endangered languages. But do they also possess the necessary creativity and capacity for abstraction to decode phonetically encoded language the same way humans do?

cs.CL↗
arXiv2024

Density Matrices for Metaphor Understanding

In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept has previously been used to model lexical ambiguity. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.

cs.CL↗
arXiv2026

Informational Antilocality and the Locality Bias in LLMs

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of $k$ contiguous symbols. We construct such languages with increasing $k$, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

cs.CL↗