Search arXiv⌕ Search

arXiv subjects

Li Zhang

Publications and source records attributed to Li Zhang.

At least 523 records · Page 29Linked to original sources
arXiv2024

Probabilistic computation and uncertainty quantification with emerging covariance

Building robust, interpretable, and secure AI system requires quantifying and representing uncertainty under a probabilistic perspective to mimic human cognitive abilities. However, probabilistic computation presents significant challenges for most conventional artificial neural network, as they are essentially implemented in a deterministic manner. In this paper, we develop an efficient probabilistic computation framework by truncating the probabilistic representation of neural activation up to its mean and covariance and construct a moment neural network that encapsulates the nonlinear coupling between the mean and covariance of the underlying stochastic network. We reveal that when only the mean but not the covariance is supervised during gradient-based learning, the unsupervised covariance spontaneously emerges from its nonlinear coupling with the mean and faithfully captures the uncertainty associated with model predictions. Our findings highlight the inherent simplicity of probabilistic computation by seamlessly incorporating uncertainty into model prediction, paving the way for integrating it into large-scale AI systems.

cs.LG↗
arXiv2024

UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer

Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet by directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.

cs.CV↗
arXiv2024

Optimal Transcoding Resolution Prediction for Efficient Per-Title Bitrate Ladder Estimation

Adaptive video streaming requires efficient bitrate ladder construction to meet heterogeneous network conditions and end-user demands. Per-title optimized encoding typically traverses numerous encoding parameters to search the Pareto-optimal operating points for each video. Recently, researchers have attempted to predict the content-optimized bitrate ladder for pre-encoding overhead reduction. However, existing methods commonly estimate the encoding parameters on the Pareto front and still require subsequent pre-encodings. In this paper, we propose to directly predict the optimal transcoding resolution at each preset bitrate for efficient bitrate ladder construction. We adopt a Temporal Attentive Gated Recurrent Network to capture spatial-temporal features and predict transcoding resolutions as a multi-task classification problem. We demonstrate that content-optimized bitrate ladders can thus be efficiently determined without any pre-encoding. Our method well approximates the ground-truth bitrate-resolution pairs with a slight Bjøntegaard Delta rate loss of 1.21% and significantly outperforms the state-of-the-art fixed ladder.

cs.MM↗
arXiv2024

DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction

Forecasting future stock trends remains challenging for academia and industry due to stochastic inter-stock dynamics and hierarchical intra-stock dynamics influencing stock prices. In recent years, graph neural networks have achieved remarkable performance in this problem by formulating multiple stocks as graph-structured data. However, most of these approaches rely on artificially defined factors to construct static stock graphs, which fail to capture the intrinsic interdependencies between stocks that rapidly evolve. In addition, these methods often ignore the hierarchical features of the stocks and lose distinctive information within. In this work, we propose a novel graph learning approach implemented without expert knowledge to address these issues. First, our approach automatically constructs dynamic stock graphs by entropy-driven edge generation from a signal processing perspective. Then, we further learn task-optimal dependencies between stocks via a generalized graph diffusion process on constructed stock graphs. Last, a decoupled representation learning scheme is adopted to capture distinctive hierarchical intra-stock features. Experimental results demonstrate substantial improvements over state-of-the-art baselines on real-world datasets. Moreover, the ablation study and sensitivity study further illustrate the effectiveness of the proposed method in modeling the time-evolving inter-stock and intra-stock dynamics.

cs.LG↗
arXiv2024

Fast KV-Switching and Dual-Layer Flat-Panel Detector Enabled Cone-Beam CT Joint Spectral Imaging

Purpose: Fast kV-switching (FKS) and dual-layer flat-panel detector (DL-FPD) technologies have been actively studied as promising dual-energy solutions for FPD-based cone-beam computed tomography (CBCT). However, CBCT spectral imaging is known to face challenges in obtaining accurate and robust material discrimination performance due to the limited energy separation. To further improve CBCT spectral imaging capability, this work aims to promote a source-detector joint spectral imaging solution which takes advantages of both FKS and DL-FPD, and to conduct a feasibility study on the first tabletop CBCT system with the joint spectral imaging capability developed. Methods: In this work, the first FKS and DL-FPD jointly enabled multi-energy tabletop CBCT system has been developed in our laboratory. To evaluate its spectral imaging performance, a set of physics experiments are conducted, where the multi-energy and head phantoms are scanned using the 80/105/130kVp switching pairs and projection data are collected using a prototype DL-FPD. To compensate for the slightly angular mismatch between the low- and high-energy projections in FKS, a dual-domain projection completion scheme is implemented. Afterwards material decomposition is carried out by using the maximum-likelihood method, followed by reconstruction of basis material and virtual monochromatic images. Results: The physics experiments confirmed the feasibility and superiority of the joint spectral imaging, whose CNR of the multi-energy phantom were boosted by an average improvement of 21.9%, 20.4% for water and 32.8%, 62.8% for iodine when compared with that of the FKS and DL-FPD in fan-beam and cone-beam experiments, respectively. Conclusions: A feasibility study of the joint spectral imaging for CBCT by utilizing both the FKS and DL-FPD was conducted, with the first tabletop CBCT system having such a capability being developed.

physics.med-ph↗
arXiv2024

FGENet: Fine-Grained Extraction Network for Congested Crowd Counting

Crowd counting has gained significant popularity due to its practical applications. However, mainstream counting methods ignore precise individual localization and suffer from annotation noise because of counting from estimating density maps. Additionally, they also struggle with high-density images.To address these issues, we propose an end-to-end model called Fine-Grained Extraction Network (FGENet). Different from methods estimating density maps, FGENet directly learns the original coordinate points that represent the precise localization of individuals.This study designs a fusion module, named Fine-Grained Feature Pyramid(FGFP), that is used to fuse feature maps extracted by the backbone of FGENet. The fused features are then passed to both regression and classification heads, where the former provides predicted point coordinates for a given image, and the latter determines the confidence level for each predicted point being an individual. At the end, FGENet establishes correspondences between prediction points and ground truth points by employing the Hungarian algorithm. For training FGENet, we design a robust loss function, named Three-Task Combination (TTC), to mitigate the impact of annotation noise. Extensive experiments are conducted on four widely used crowd counting datasets. Experimental results demonstrate the effectiveness of FGENet. Notably, our method achieves a remarkable improvement of 3.14 points in Mean Absolute Error (MAE) on the ShanghaiTech Part A dataset, showcasing its superiority over the existing state-of-the-art methods. Even more impressively, FGENet surpasses previous benchmarks on the UCF\_CC\_50 dataset with an astounding enhancement of 30.16 points in MAE.

cs.CV↗
arXiv2023

Harnessing Diffusion Models for Visual Perception with Meta Prompts

The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiments across various benchmarks validate the effectiveness of our approach. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes. Concurrently, the proposed method attains results comparable to the current state-of-the-art in semantic segmentation on ADE20K and pose estimation on COCO datasets, further exemplifying its robustness and versatility.

cs.CV↗
arXiv2023

Second-harmonic generation with a 440,000% W-1 conversion efficiency in a lithium niobate microcavity without periodic poling

Thin-film lithium niobate (TFLN) enables extremely high-efficiency second-order nonlinear optical effects due to large nonlinear coefficient d33 and strong optical field localization. Here, we first designed and fabricated a pulley-waveguide-coupled microring resonator with an intrinsic quality factor above 9.4 x10^5 on the reverse-polarized double-layer X-cut TFLN. In such a TFLN resonator without fine domain structures, second harmonic generation with an absolute (normalized) conversion efficiency of 30% (440,000% W-1), comparable to that in periodically poled lithium niobate (PPLN) microring resonators, was realized with a sub-microwatt continuous pump. This work reduces the dependence of high-efficiency nonlinear frequency conversion on PPLN microcavities that are difficult to prepare.

physics.optics↗
arXiv2023

Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization

The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained supervised and self-supervised speech models for audio-visual speaker diarization. Specifically, we adopt supervised~(ResNet and ECAPA-TDNN) and self-supervised pre-trained models~(WavLM and HuBERT) as the speaker and audio embedding extractors in an end-to-end audio-visual speaker diarization~(AVSD) system. Then we explore the effectiveness of different frameworks, including Transformer, Conformer, and cross-attention mechanism, in the audio-visual decoder. To mitigate the degradation of performance caused by separate training, we jointly train the audio encoder, speaker encoder, and audio-visual decoder in the AVSD system. Experiments on the MISP dataset demonstrate that the proposed method achieves superior performance and obtained third place in MISP Challenge 2022.

eess.AS↗
arXiv2023

PARTNER: Level up the Polar Representation for LiDAR 3D Object Detection

Recently, polar-based representation has shown promising properties in perceptual tasks. In addition to Cartesian-based approaches, which separate point clouds unevenly, representing point clouds as polar grids has been recognized as an alternative due to (1) its advantage in robust performance under different resolutions and (2) its superiority in streaming-based approaches. However, state-of-the-art polar-based detection methods inevitably suffer from the feature distortion problem because of the non-uniform division of polar representation, resulting in a non-negligible performance gap compared to Cartesian-based approaches. To tackle this issue, we present PARTNER, a novel 3D object detector in the polar coordinate. PARTNER alleviates the dilemma of feature distortion with global representation re-alignment and facilitates the regression by introducing instance-level geometric information into the detection head. Extensive experiments show overwhelming advantages in streaming-based detection and different resolutions. Furthermore, our method outperforms the previous polar-based works with remarkable margins of 3.68% and 9.15% on Waymo and ONCE validation set, thus achieving competitive results over the state-of-the-art methods.

cs.CV↗
arXiv2023

Demonstration of Programmable Brain-Inspired Optoelectronic Neuron in Photonic Spiking Neural Network with Neural Heterogeneity

Photonic Spiking Neural Networks (PSNN) composed of the co-integrated CMOS and photonic elements can offer low loss, low power, highly-parallel, and high-throughput computing for brain-inspired neuromorphic systems. In addition, heterogeneity of neuron dynamics can also bring greater diversity and expressivity to brain-inspired networks, potentially allowing for the implementation of complex functions with fewer neurons. In this paper, we design, fabricate, and experimentally demonstrate an optoelectronic spiking neuron that can simultaneously achieve high programmability for heterogeneous biological neural networks and maintain high-speed computing. We demonstrate that our neuron can be programmed to tune four essential parameters of neuron dynamics under 1GSpike/s input spiking pattern signals. A single neuron circuit can be tuned to output three spiking patterns, including chattering behaviors. The PSNN consisting of the optoelectronic spiking neuron and a Mach-Zehnder interferometer (MZI) mesh synaptic network achieves 89.3% accuracy on the Iris dataset. Our neuron power consumption is 1.18 pJ/spike output, mainly limited by the power efficiency of the vertical-cavity-lasers, optical coupling efficiency, and the 45 nm CMOS platform used in this experiment, and is predicted to achieve 36.84 fJ/spike output with a 7 nm CMOS platform (e.g. ASAP7) integrated with silicon photonics containing on-chip micron-scale lasers.

eess.SY↗
arXiv2023

Liquid-shaped microlens for scalable production of ultrahigh-resolution OCT microendoscope

Endoscopic optical coherence tomography (OCT) is a valuable tool for providing diagnostic images of internal organs and guiding interventions in real time. Miniaturized OCT endoscopes are essential for imaging small and convoluted luminal organs while minimizing invasiveness. However, current methods for fabricating miniature fiber probes have limited ability to correct optical aberrations, leading to suboptimal imaging performance. In this study, we introduce a new paradigm of liquid shaping technique for the rapid and scalable fabrication of ultrathin and high-performance OCT microendoscopes suitable for minimally invasive clinical applications. This technique enables the flexible customization of freeform microlenses with sub-nanometer optical surface roughness by regulating the minimum energy state of curable optical liquid on a wettability-modified substrate and precisely controlling the liquid volume and physical boundary on a substrate. Using this technique, we simultaneously fabricated 800-nm OCT microendoscopes with a diameter of approximately 0.6 mm and evaluated their ultrahigh-resolution imaging performance in the esophagus of rats and the aorta and brain of mice.

physics.ins-det↗
arXiv2023

Applying hybrid clustering in pulsar candidate sifting with multi-modality for FAST survey

Pulsar search is always the basis of pulsar navigation, gravitational wave detection and other research topics. Currently, the volume of pulsar candidates collected by Five-hundred-meter Aperture Spherical radio Telescope (FAST) shows an explosive growth rate that has brought challenges for its pulsar candidate filtering System. Particularly, the multi-view heterogeneous data and class imbalance between true pulsars and non-pulsar candidates have negative effects on traditional single-modal supervised classification methods. In this study, a multi-modal and semi-supervised learning based pulsar candidate sifting algorithm is presented, which adopts a hybrid ensemble clustering scheme of density-based and partition-based methods combined with a feature-level fusion strategy for input data and a data partition strategy for parallelization. Experiments on both HTRU (The High Time Resolution Universe Survey) 2 and FAST actual observation data demonstrate that the proposed algorithm could excellently identify the pulsars: On HTRU2, the precision and recall rates of its parallel mode reach 0.981 and 0.988. On FAST data, those of its parallel mode reach 0.891 and 0.961, meanwhile, the running time also significantly decrease with the increment of parallel nodes within limits. So, we can get the conclusion that our algorithm could be a feasible idea for large scale pulsar candidate sifting of FAST drift scan observation.

astro-ph.IM↗
arXiv2023

Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

In this paper, we present Consistent4D, a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. Uniquely, we cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model as the primary supervision signal for training Dynamic Neural Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to facilitate stable convergence and temporal continuity under the supervision signal which is discrete along the time axis. To achieve spatial and temporal consistency, we further introduce an Interpolation-driven Consistency Loss. It is optimized by minimizing the discrepancy between rendered frames from DyNeRF and interpolated frames from a pre-trained video interpolation model. Extensive experiments show that our Consistent4D can perform competitively to prior art alternatives, opening up new possibilities for 4D dynamic object generation from monocular videos, whilst also demonstrating advantage for conventional text-to-3D generation tasks. Our project page is https://consistent4d.github.io/.

cs.CV↗
arXiv2023

Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review

With the rapid development of artificial intelligence, large language models (LLMs) have shown promising capabilities in mimicking human-level language comprehension and reasoning. This has sparked significant interest in applying LLMs to enhance various aspects of healthcare, ranging from medical education to clinical decision support. However, medicine involves multifaceted data modalities and nuanced reasoning skills, presenting challenges for integrating LLMs. This paper provides a comprehensive review on the applications and implications of LLMs in medicine. It begins by examining the fundamental applications of general-purpose and specialized LLMs, demonstrating their utilities in knowledge retrieval, research support, clinical workflow automation, and diagnostic assistance. Recognizing the inherent multimodality of medicine, the review then focuses on multimodal LLMs, investigating their ability to process diverse data types like medical imaging and EHRs to augment diagnostic accuracy. To address LLMs' limitations regarding personalization and complex clinical reasoning, the paper explores the emerging development of LLM-powered autonomous agents for healthcare. Furthermore, it summarizes the evaluation methodologies for assessing LLMs' reliability and safety in medical contexts. Overall, this review offers an extensive analysis on the transformative potential of LLMs in modern medicine. It also highlights the pivotal need for continuous optimizations and ethical oversight before these models can be effectively integrated into clinical practice. Visit https://github.com/mingze-yuan/Awesome-LLM-Healthcare for an accompanying GitHub repository containing latest papers.

cs.CL↗
arXiv2023

Federated Learning of Shareable Bases for Personalization-Friendly Image Classification

Personalized federated learning (PFL) aims to harness the collective wisdom of clients' data while building personalized models tailored to individual clients' data distributions. Existing works offer personalization primarily to clients who participate in the FL process, making it hard to encompass new clients who were absent or newly show up. In this paper, we propose FedBasis, a novel PFL framework to tackle such a deficiency. FedBasis learns a set of few shareable ``basis'' models, which can be linearly combined to form personalized models for clients. Specifically for a new client, only a small set of combination coefficients, not the model weights, needs to be learned. This notion makes FedBasis more parameter-efficient, robust, and accurate than competitive PFL baselines, especially in the low data regime, without increasing the inference cost. To demonstrate the effectiveness and applicability of FedBasis, we also present a more practical PFL testbed for image classification, featuring larger data discrepancies across clients in both the image and label spaces as well as more faithful training and test splits.

cs.CV↗
arXiv2023

Private Learning with Public Features

We study a class of private learning problems in which the data is a join of private and public features. This is often the case in private personalization tasks such as recommendation or ad prediction, in which features related to individuals are sensitive, while features related to items (the movies or songs to be recommended, or the ads to be shown to users) are publicly available and do not require protection. A natural question is whether private algorithms can achieve higher utility in the presence of public features. We give a positive answer for multi-encoder models where one of the encoders operates on public features. We develop new algorithms that take advantage of this separation by only protecting certain sufficient statistics (instead of adding noise to the gradient). This method has a guaranteed utility improvement for linear regression, and importantly, achieves the state of the art on two standard private recommendation benchmarks, demonstrating the importance of methods that adapt to the private-public feature separation.

cs.LG↗
arXiv2023

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

cs.LG↗