Stability and Political Orientation of International LLMs: An Exploratory Multi-Run Study Conducted in French

Abstract

This study explores the stability and ideological orientation of political responses produced by various large language models (LLMs) in French. We designed a standardised experimental protocol based on a questionnaire inspired by the Political Compass, aimed at measuring the economic and socio-cultural positions of each model across 62 political statements. Eleven models from various organisations and countries were tested, each subjected to twenty independent runs to assess intra-model variability. The analysis focuses on response consistency, inter-model differences, and the presence of implicit political orientations. The results show that, despite a general degree of stability, significant variations appear from one run to the next and between models, reflecting the impact of architectures, training data, and moderation mechanisms. This study proposes a comparative evaluation protocol for LLMs in the context of political information and underscores the importance of accounting for implicit biases in the use of these systems.