Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2509.14233

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

Project Apertus·Alejandro Hernández-Cano·Alexander Hägele·Allen Hao Huang·Angelika Romanou·Antoni-Joan Solergibert·Barna Pasztor·Bettina Messmer·Dhia Garbaya·Eduard Frank Ďurech·Ido Hakimi·Juan García Giraldo·Mete Ismayilzada·Negar Foroutan·Skander Moalla·Tiancheng Chen·Vinko Sabolčec·Yixuan Xu·Michael Aerni·Badr AlKhamissi·Inés Altemir Mariñas·Mohammad Hossein Amani·Matin Ansaripour·Ilia Badanin·Harold Benoit·Emanuela Boros·Nicholas Browning·Fabian Bösch·Maximilian Böther·Niklas Canova·Camille Challier·Clement Charmillot·Jonathan Coles·Jan Deriu·Arnout Devos·Lukas Drescher·Daniil Dzenhaliou·Maud Ehrmann·Dongyang Fan·Simin Fan·Silin Gao·Miguel Gila·María Grandury·Diba Hashemi·Alexander Hoyle·Jiaming Jiang·Mark Klein·Andrei Kucharavy·Anastasiia Kucherenko·Frederike Lübeck·Roman Machacek·Theofilos Manitaras·Andreas Marfurt·Kyle Matoba·Simon Matrenok·Henrique Mendonça·Fawzi Roberto Mohamed·Syrielle Montariol·Luca Mouchel·Sven Najem-Meyer·Jingwei Ni·Gennaro Oliva·Matteo Pagliardini·Elia Palme·Andrei Panferov·Léo Paoletti·Marco Passerini·Ivan Pavlov·Auguste Poiroux·Kaustubh Ponkshe·Nathan Ranchin·Javi Rando·Mathieu Sauser·Jakhongir Saydaliev·Muhammad Ali Sayfiddinov·Marian Schneider·Stefano Schuppli·Marco Scialanga·Andrei Semenov·Kumar Shridhar·Raghav Singhal·Anna Sotnikova·Alexander Sternfeld·Ayush Kumar Tarun·Paul Teiletche·Jannis Vamvas·Xiaozhe Yao·Hao Zhao·Alexander Ilic·Ana Klimovic·Andreas Krause·Caglar Gulcehre·David Rosenthal·Elliott Ash·Florian Tramèr·Joost VandeVondele·Livio Veraldi·Martin Rajman·Thomas Schulthess·Torsten Hoefler
View original source ↗Read source document ↗Export metadata ↓

Abstract

We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.

Explore related subjects

cs.CLcs.AIcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Mariñas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clement Charmillot, Jonathan Coles, Jan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javi Rando, Mathieu Sauser, Jakhongir Saydaliev, Muhammad Ali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas Schulthess, Torsten Hoefler. 2025-12-01. Apertus: Democratizing Open and Compliant LLMs for Global Language Environments. https://arxiv.org/abs/2509.14233

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LiSeCo: Linear Semantic Control for Language Generation

The prevalence of Large Language Models (LLMs) in critical applications highlights the need for controlled language generation methods that are both computationally efficient and enjoy performance guarantees. To address this need, we use a common model of concept semantics as linearly represented in an LLM's latent space. In particular, we take the view that natural language generation traces a trajectory in this continuous semantic space, realized by the language model's hidden activations. This view permits a control-theoretic treatment of text generation in latent space, in which we propose Linear Semantic Control (LiSeCo), a lightweight, gradient-free intervention that dynamically steers trajectories away from regions corresponding to undesired meanings. In particular, we propose to directly intervene, in an online fashion, the activations of the token that is being generated in embedding space. Crucially, LiSeCo does not simply steer activations towards a desirable region. Instead, it relies on classical techniques from control theory to precisely control activations in a context-dependent way, and guarantees that they are brought into a specific pre-defined region of embedding space that corresponds to allowed semantics. The intervention is computed in closed form according to an optimal controller formulation, minimally impacting generation time. This control of the activations in embedding space allows for fine-grained steering of attributes of the generated sequence. We demonstrate that our approach is effective on different tasks -- toxicity, sentiment, and language (English/Spanish) steering -- while maintaining text quality.

cs.CL↗
arXiv2026

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) interpret and reason over visual and textual information beyond English. VMMU consists of 2.5k multimodal questions across 7 tasks, covering a diverse range of problem contexts, including STEM problem solving, data interpretation, rule-governed visual reasoning, and abstract visual reasoning. All questions require genuine multimodal integration, rather than reliance on text-only cues or OCR-based shortcuts. We evaluate a diverse set of state-of-the-art proprietary and open-source VLMs on VMMU. Despite strong Vietnamese OCR performance, proprietary models achieve only 66% mean accuracy. Further analysis shows that the primary source of failure is not OCR, but instead multimodal grounding and reasoning over text and visual evidence. Code and data are available at https://vmmu-bench.github.io/

cs.CL↗
arXiv2026

RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

Machine unlearning for large language models (LLMs) remains challenging because full retraining is costly, while approximate methods often struggle to remove targeted behaviors without degrading retained utility, especially under limited post-deployment supervision. We consider a practical PEFT setting for targeted behavioral contamination removal with a small forget set, a limited retain buffer, and LoRA-only updates, and propose RapidUn, an influence-guided framework that converts cross-sample influence estimates into fixed sample-specific weights for weighted LoRA unlearning. Across Llama-3-8B on Dolly-15k and Alpaca-57k, with cross-model validation on Mistral-7B + Dolly-15k, RapidUn achieves lower seen-trigger and OOD-trigger-family ASR than Fisher, GA, and LoReUn while maintaining competitive clean utility. On Llama-3-8B + Alpaca-57k, it achieves a 77x wall-clock speedup over the clean-corpus LoRA retraining reference. Complementary TOFU, semantic LLM-judge, and IFEval evaluations further support the effectiveness of influence-guided sample reweighting beyond the controlled trigger benchmark.

cs.CL↗