Parameters efficient language model with latent cross-attention for IoT intrusion detection

Frederic Adjewa; Moez Esseghir; Leila Merghem-Boulahia; Cheikh Kacfah

Home/Archive/arXiv

arXiv · 2610.30222

Parameters efficient language model with latent cross-attention for IoT intrusion detection

Frederic Adjewa·Moez Esseghir·Leila Merghem-Boulahia·Cheikh Kacfah

View original source ↗Read source document ↗Export metadata ↓

Abstract

Language models have shown strong potential for intrusion detection, but their high memory and computational demands make them difficult to deploy in resource-constrained IoT environments. In this work, we propose a compact Transformer-based architecture that compresses the input into a small latent space before processing it with a minimal Transformer core. We analyze nine compression configurations, compare parameter sharing with layer stacking, and evaluate our approach against SecurityBERT on two IoT intrusion detection datasets, Edge-IIoTset and CICIoT2023. With only 306k non-embedding parameters and 1.6M parameters in total (6MB), our model is about 7x smaller than SecurityBERT (11.2M parameters, 45MB). With the same privacy-preserving encoding, it outperforms SecurityBERT by about 2 points of macro F1-score on Edge-IIoTset and is within 0.7 points on CICIoT2023. It is also about 89x faster on a simulated edge device. We further show that a single Transformer layer is sufficient, and that parameter sharing matches layer stacking with up to 66% fewer parameters. These findings show that lightweight Transformer-based architectures can provide a practical and accurate solution for intrusion detection in resource-constrained IoT settings

Explore related subjects

cs.CR

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Frederic Adjewa, Moez Esseghir, Leila Merghem-Boulahia, Cheikh Kacfah. 2026-10-21. Parameters efficient language model with latent cross-attention for IoT intrusion detection. https://arxiv.org/abs/2610.30222

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

How to Backdoor Image Knowledge Distillation

Knowledge distillation is widely used to transfer behavior from a large teacher model to a smaller student. It is often assumed to be safe when the teacher is clean, because classic backdoor attacks rely on poisoned labels and triggers in supervised training, whereas distillation trains the student to match a teacher's outputs. We show that this assumption can fail when the distillation dataset itself is poisoned. Our attack injects triggered and manipulated images that a clean teacher already predicts as an attacker chosen target label, which causes the student to learn a backdoor even though the teacher remains unaffected. We evaluate this threat across multiple manipulation strategies, including targeted adversarial perturbations and targeted GAN based class transitions, and study how distillation settings influence both accuracy and attack success. We show that the attack remains effective even at a 10% poisoning rates. The results demonstrate that a clean teacher alone is not a sufficient safeguard: poisoned distillation data can produce a strongly backdoored student while maintaining competitive performance on clean images. These findings show that the integrity and provenance of distillation data are part of the security boundary of data intensive KD pipelines, even when the teacher itself is trusted.

cs.CR↗

arXiv2026

Computer Science Conferences Should Require Nonrepudiable Experimental Results

This position paper argues that computer science conferences should require tamper-evident, nonrepudiable attestations of experimental results. We name the underlying problem experiment nonrepudiation: a compliant protocol must bind the numbers in a paper to an actual executed computation in a way the author cannot later alter or deny. The current system relies on self-reported checklists, optional code sharing, and author-controlled logging. None of these mechanisms answer the question a reviewer cannot check: did the code the paper describes produce the numbers the paper reports? We define the problem formally, state the security properties any compliant protocol must satisfy, and describe a threat model that includes attacks current approaches do not prevent. We frame the mechanism that provides this property as a proof-of-compute layer, a general requirement that computed results be admitted as evidence only when accompanied by a verifiable record of the computation that produced them. To show that the problem is solvable, we built K-Veritas, the official reference implementation and public infrastructure for experiment nonrepudiation, written in Go, which produces signed reports without accessing training data. K-Veritas is in use today, with a public registry of sealed records; it is not final, and we intend to improve it through community collaboration. We call on conferences and the community to treat nonrepudiation as a major requirement and to help build an open, independent standard for it.

cs.CR↗

arXiv2026

Giskard : Byzantine Robust and Private Aggregation for Large-Scale Decentralized Learning

In decentralized learning, clients train a machine learning model while keeping their data locally and share their model parameters or gradients with a set of neighbors, making it vulnerable to both privacy attacks and model poisoning attacks. Jointly addressing privacy and Byzantine behavior in decentralized learning is challenging. Indeed, Differential Privacy (DP) applied on the client side is known to increase the variance of honest model updates, which weakens Byzantine-robust aggregation against poisoning attacks. Distributed DP can mitigate this issue by adding noise after aggregation, but implementing it without a trusted curator requires both confidential and Byzantine-resilient computation. Recent work addresses this challenge using secure multi-party computation (MPC) to implement robust aggregation while preserving input confidentiality. However, these solutions scale badly: they either require all-to-all communication between participants or delegate the entire computation to a small subset of parties whose computational and communication cost grows with network size. In this paper, we present Giskard, a scalable protocol for private and Byzantine-robust decentralized aggregation. Giskard organizes $n$ parties into a tree of committees of size $O(\log n)$ and evaluates a differentially private coordinate-wise approximate median via a committee-adapted secure quantile search over the value domain, using MPC within each committee. We assess Giskard both theoretically by proving its security and privacy properties and empirically through extensive experiments involving up to one million participants. Compared to its closest competitors, Giskard reduces per-party communication complexity by close to three orders of magnitude over the closest competitor while exhibiting comparable robustness under up to $n/4$ Byzantine parties.

cs.CR↗