Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Andrey Ignatov; Cheng-Ming Chiang; Hsien-Kai Kuo; Anastasia Sycheva; Radu Timofte; Min-Hung Chen; Man-Yu Lee; Yu-Syuan Xu; Yu Tseng; Shusong Xu; Jin Guo; Chao-Hung Chen; Ming-Chun Hsyu; Wen-Chia Tsai; Chao-Wei Chen; Grigory Malivenko; Minsu Kwon; Myungje Lee; Jaeyoon Yoo; Changbeom Kang; Shinjo Wang; Zheng Shaolong; Hao Dejun; Xie Fen; Feng Zhuang; Yipeng Ma; Jingyang Peng; Tao Wang; Fenglong Song; Chih-Chung Hsu; Kwan-Lin Chen; Mei-Hsuang Wu; Vishal Chudasama; Kalpesh Prajapati; Heena Patel; Anjali Sarvaiya; Kishor Upla; Kiran Raja; Raghavendra Ramachandra; Christoph Busch; Etienne de Stoutz

arXiv · 2105.07809

Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

View original source ↗Read source document ↗Export metadata ↓

Abstract

As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

Explore related subjects

eess.IV cs.CV cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Andrey Ignatov, Cheng-Ming Chiang, Hsien-Kai Kuo, Anastasia Sycheva, Radu Timofte, Min-Hung Chen, Man-Yu Lee, Yu-Syuan Xu, Yu Tseng, Shusong Xu, Jin Guo, Chao-Hung Chen, Ming-Chun Hsyu, Wen-Chia Tsai, Chao-Wei Chen, Grigory Malivenko, Minsu Kwon, Myungje Lee, Jaeyoon Yoo, Changbeom Kang, Shinjo Wang, Zheng Shaolong, Hao Dejun, Xie Fen, Feng Zhuang, Yipeng Ma, Jingyang Peng, Tao Wang, Fenglong Song, Chih-Chung Hsu, Kwan-Lin Chen, Mei-Hsuang Wu, Vishal Chudasama, Kalpesh Prajapati, Heena Patel, Anjali Sarvaiya, Kishor Upla, Kiran Raja, Raghavendra Ramachandra, Christoph Busch, Etienne de Stoutz. 2021-05-17. Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report. https://arxiv.org/abs/2105.07809

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Hyperspectral Image Restoration and Super-resolution with Physics-Aware Deep Learning for Biomedical Applications

Hyperspectral imaging is a powerful bioimaging tool which can uncover novel insights, thanks to its sensitivity to the intrinsic properties of materials. However, this enhanced contrast comes at the cost of system complexity, constrained by an inherent trade-off between spatial, spectral, and temporal resolution. To overcome this limitation, we present a self-supervised deep learning-based approach that restores and enhances pixel resolution post-acquisition without requiring external training data beyond the images to be restored. Fine-tuned using metrics aligned with the imaging model, our physics-aware method achieves a 16$\times$ pixel super-resolution enhancement and a 12$\times$ imaging speedup without the need of additional training data for transfer learning. Applied to both synthetic and experimental data from five different sample types, including healthy and diseased tissues, we demonstrate that the model preserves biological integrity, as we did not detect systematic loss of biological features or biologically consequential hallucinations in tested datasets. We also concretely demonstrate the model's ability to reveal disease-associated metabolic changes that would otherwise remain undetectable. Furthermore, we provide physical insights into the model's inner workings, paving the way for future refinements that could potentially reveal novel high resolution features in an explainable manner. All methods are available as open-source software on GitHub.

eess.IV↗

arXiv2026

OASIS: Online Adaptive Video Compression via Closed-loop Feedback Control

Computer vision systems are a key building block in an autonomous vehicle, responsible for a range of perception tasks. However, they incur massive data transmission over long communication links from multiple cameras, creating a critical bandwidth and energy bottleneck. Although conventional codecs such as H.264 can reduce data rates, they are ill-suited for real-time vision systems due to high processing latency and energy consumption, as well as their reliance on static user-defined compression settings. In light of these challenges, we propose OASIS, an adaptive video compression framework that integrates lightweight in-sensor compression with task-aware compression ratio control. Based on the real-time task performance, it dynamically updates the optimal compression ratio. Experimental results demonstrate that OASIS generalizes across multiple vision tasks, achieving on average a 6x data compression, 5.8x reduction in link power consumption, and 2.5x reduction in link latency, with at most 1.5% performance degradation.

eess.IV↗

arXiv2026

Echo-E$^3$Net: Efficient Endocardial Spatio-Temporal Network for Ejection Fraction Estimation

Left ventricular ejection fraction (LVEF) is a primary marker of cardiac function. However, routine estimation from endocardial measurements requires manual delineation at end-diastole (ED) and end-systole (ES), a process that is time-consuming and subject to inter-observer variability. Reliable automation is especially valuable for point-of-care ultrasound (POCUS), where computational resources are limited and acquisition quality varies. We propose Echo-E$^3$Net, an anatomy-guided spatio-temporal network that explicitly embeds cardiac anatomy into LVEF prediction. A dual-phase Endocardial Border Detector (E$^2$CBD) uses phase-specific cross-attention to localize ED/ES endocardial landmarks and produce phase-aware landmark embeddings, while an Endocardial Feature Aggregator (E$^2$FA) fuses these embeddings with global statistical descriptors of deep feature maps to refine EF regression. Training is guided by a lightweight geometric loss that uses ED and ES endocardial landmarks to regularize EF prediction. On EchoNet-Dynamic and a PSAX subset of EchoNet-Pediatric, Echo-E$^3$Net attains competitive performance using only 1.55M parameters and 8.05 GFLOPs, an order-of-magnitude compute reduction versus recent baselines, supporting real-time deployment. Our code is publicly available at https://github.com/moeinheidari7829/Echo-E3Net.

eess.IV↗