Towards Foundation Models on Hardware Accelerators for Particle Physics

Abstract

Bandwidth constraints require many particle physics experiments to make real-time decisions on custom hardware or firmware running simplified algorithms, unlike offline analysis, where latency is usually not a limiting factor. For example, for particle jet tagging at colliders, state-of-the-art performance is achieved by foundation models with hundreds of millions of parameters pre-trained with billions of jets. We use knowledge distillation to transfer what such models have learned into efficient networks towards deployment in hardware accelerators. The teacher is the OmniLearned foundation model fine-tuned on top quark jet tagging; the student is an attention-free Deep Sets network. We demonstrate three ways the student's performance improves: adding a message-passing layer to the Deep Sets architecture, training on the teacher's soft labels rather than on ground-truth labels alone, and distilling from a pretrained teacher rather than from the same architecture trained from scratch. In each case the gain is largest in the background rejection at low signal efficiency, the regime that is most relevant for a trigger.