LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter

Abstract

Blind face restoration from low-quality images is a challenging task that requires not only high-fidelity image reconstruction, but also preservation of facial identity. Although diffusion models like Stable Diffusion have shown promise in generating high-quality images, their VAE modules are typically trained on broad natural-image data dominated by HQ content; severely degraded LQ inputs therefore yield latents that fall in low-density regions of the diffusion prior, weakening the effectiveness of LQ conditions during the denoising process. Existing approaches often tackle this issue by retraining the VAE encoder, which is computationally expensive and memory intensive. To address this limitation efficiently, we propose LAFR, a codebook-based latent space adapter that aligns LQ latents to the in-distribution region the diffusion prior was trained to denoise, enabling sampling from a better matched latent condition without altering the original VAE. To further improve identity and structural consistency relative to OSEDiff-style baselines, we introduce a multilevel restoration loss combining constraints from semantic identity embeddings and facial structural priors. Furthermore, by leveraging the inherent structural regularity of facial images, we show that lightweight fine-tuning of a diffusion prior on just 0.9% of FFHQ can achieve competitive results while reducing training time by 70%. We also give a short analysis of why a codebook correction closes a mode-dependent latent gap that no global normalisation or affine map can. In summary, LAFR is a data and compute efficient one-step face-restoration method that improves perceptual quality, FID, landmark accuracy, and identity/structure consistency over OSEDiff-style baselines, without claiming uniform state of the art; it reaches this operating point while remaining competitive with far more expensive pipelines.