Federated Topic Model and Model Pruning Based on Variational Autoencoder

Abstract

Topic modeling has emerged as a valuable tool for discovering patterns and topics within large collections of documents. However, when cross-analysis involves multiple parties, data privacy becomes a critical concern. Federated topic modeling has been developed to address this issue, allowing multiple parties to jointly train models while protecting privacy. However, there are communication and performance challenges in the federated scenario. To solve these problems, this paper proposes a method to establish a federated topic model while ensuring the privacy of each node and uses neural network model pruning to accelerate the model. The client periodically sends cumulative neuron gradients and model weights to the server, and the server prunes the model. To address different requirements, two methods are proposed to determine the pruning rate. The first slowly prunes throughout training, which has limited acceleration during training but can ensure higher accuracy and significantly reduce inference time. The second quickly reaches the target pruning rate early in training and then continues training with a smaller model. This approach may lose more useful information but can complete training faster. Experimental results show that the proposed variational-autoencoder-based federated topic model pruning can greatly accelerate training while maintaining model performance.