Search arXiv⌕ Search

arXiv subjects

Kai Wang

Publications and source records attributed to Kai Wang.

At least 541 records · Page 30Linked to original sources
arXiv2023

DREAM: Efficient Dataset Distillation by Representative Matching

Dataset distillation aims to synthesize small datasets with little information loss from original large-scale ones for reducing storage and training costs. Recent state-of-the-art methods mainly constrain the sample synthesis process by matching synthetic images and the original ones regarding gradients, embedding distributions, or training trajectories. Although there are various matching objectives, currently the strategy for selecting original images is limited to naive random sampling. We argue that random sampling overlooks the evenness of the selected sample distribution, which may result in noisy or biased matching targets. Besides, the sample diversity is also not constrained by random sampling. These factors together lead to optimization instability in the distilling process and degrade the training efficiency. Accordingly, we propose a novel matching strategy named as \textbf{D}ataset distillation by \textbf{RE}present\textbf{A}tive \textbf{M}atching (DREAM), where only representative original images are selected for matching. DREAM is able to be easily plugged into popular dataset distillation frameworks and reduce the distilling iterations by more than 8 times without performance drop. Given sufficient training time, DREAM further provides significant improvements and achieves state-of-the-art performances.

cs.CV↗
arXiv2023

Spatio-temporal Joint Modelling on Moderate and Extreme Air Pollution in Spain

Very unhealthy air quality is consistently connected with numerous diseases. Appropriate extreme analysis and accurate predictions are in rising demand for exploring potential linked causes and for providing suggestions for the environmental agency in public policy strategy. This paper aims to model the spatial and temporal pattern of both moderate and extremely poor PM10 concentrations (of daily mean) collected from 342 representative monitors distributed throughout mainland Spain from 2017 to 2021. We firstly propose and compare a series of Bayesian hierarchical generalized extreme models of annual maxima PM10 concentrations, including both the fixed effect of altitude, temperature, precipitation, vapour pressure and population density, as well as the spatio-temporal random effect with the Stochastic Partial Differential Equation (SPDE) approach and a lag-one dynamic auto-regressive component (AR(1)). Under WAIC, DIC and other criteria, the best model is selected with good predictive ability based on the first four-year data (2017--2020) for training and the last-year data (2021) for testing. We bring the structure of the best model to establish the joint Bayesian model of annual mean and annual maxima PM10 concentrations and provide evidence that certain predictors (precipitation, vapour pressure and population density) influence comparably while the other predictors (altitude and temperature) impact reversely in the different scaled PM10 concentrations. The findings are applied to identify the hot-spot regions with poor air quality using excursion functions specified at the grid level. It suggests that the community of Madrid and some sites in northwestern and southern Spain are likely to be exposed to severe air pollution, simultaneously exceeding the warning risk threshold.

stat.AP↗
arXiv2023

Dataset Quantization

State-of-the-art deep neural networks are trained with large amounts (millions or even billions) of data. The expensive computation and memory costs make it difficult to train them on limited hardware resources, especially for recent popular large language models (LLM) and computer vision models (CV). Recent popular dataset distillation methods are thus developed, aiming to reduce the number of training samples via synthesizing small-scale datasets via gradient matching. However, as the gradient calculation is coupled with the specific network architecture, the synthesized dataset is biased and performs poorly when used for training unseen architectures. To address these limitations, we present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets which can be used for training any neural network architectures. Extensive experiments demonstrate that DQ is able to generate condensed small datasets for training unseen network architectures with state-of-the-art compression ratios for lossless model training. To the best of our knowledge, DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio. Notably, with 60% data from ImageNet and 20% data from Alpaca's instruction tuning data, the models can be trained with negligible or no performance drop for both vision tasks (including classification, semantic segmentation, and object detection) as well as language tasks (including instruction tuning tasks such as BBH and DROP).

cs.CV↗
arXiv2023

Color Prompting for Data-Free Continual Unsupervised Domain Adaptive Person Re-Identification

Unsupervised domain adaptive person re-identification (Re-ID) methods alleviate the burden of data annotation through generating pseudo supervision messages. However, real-world Re-ID systems, with continuously accumulating data streams, simultaneously demand more robust adaptation and anti-forgetting capabilities. Methods based on image rehearsal addresses the forgetting issue with limited extra storage but carry the risk of privacy leakage. In this work, we propose a Color Prompting (CoP) method for data-free continual unsupervised domain adaptive person Re-ID. Specifically, we employ a light-weighted prompter network to fit the color distribution of the current task together with Re-ID training. Then for the incoming new tasks, the learned color distribution serves as color style transfer guidance to transfer the images into past styles. CoP achieves accurate color style recovery for past tasks with adequate data diversity, leading to superior anti-forgetting effects compared with image rehearsal methods. Moreover, CoP demonstrates strong generalization performance for fast adaptation into new domains, given only a small amount of unlabeled images. Extensive experiments demonstrate that after the continual training pipeline the proposed CoP achieves 6.7% and 8.1% average rank-1 improvements over the replay method on seen and unseen domains, respectively. The source code for this work is publicly available in https://github.com/vimar-gu/ColorPromptReID.

cs.CV↗
arXiv2023

The Snowflake Hypothesis: Training Deep GNN with One Node One Receptive field

Despite Graph Neural Networks demonstrating considerable promise in graph representation learning tasks, GNNs predominantly face significant issues with over-fitting and over-smoothing as they go deeper as models of computer vision realm. In this work, we conduct a systematic study of deeper GNN research trajectories. Our findings indicate that the current success of deep GNNs primarily stems from (I) the adoption of innovations from CNNs, such as residual/skip connections, or (II) the tailor-made aggregation algorithms like DropEdge. However, these algorithms often lack intrinsic interpretability and indiscriminately treat all nodes within a given layer in a similar manner, thereby failing to capture the nuanced differences among various nodes. To this end, we introduce the Snowflake Hypothesis -- a novel paradigm underpinning the concept of ``one node, one receptive field''. The hypothesis draws inspiration from the unique and individualistic patterns of each snowflake, proposing a corresponding uniqueness in the receptive fields of nodes in the GNNs. We employ the simplest gradient and node-level cosine distance as guiding principles to regulate the aggregation depth for each node, and conduct comprehensive experiments including: (1) different training schemes; (2) various shallow and deep GNN backbones, and (3) various numbers of layers (8, 16, 32, 64) on multiple benchmarks (six graphs including dense graphs with millions of nodes); (4) compare with different aggregation strategies. The observational results demonstrate that our hypothesis can serve as a universal operator for a range of tasks, and it displays tremendous potential on deep GNNs. It can be applied to various GNN frameworks, enhancing its effectiveness when operating in-depth, and guiding the selection of the optimal network depth in an explainable and generalizable way.

cs.LG↗
arXiv2023

Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health

This paper studies restless multi-armed bandit (RMAB) problems with unknown arm transition dynamics but with known correlated arm features. The goal is to learn a model to predict transition dynamics given features, where the Whittle index policy solves the RMAB problems using predicted transitions. However, prior works often learn the model by maximizing the predictive accuracy instead of final RMAB solution quality, causing a mismatch between training and evaluation objectives. To address this shortcoming, we propose a novel approach for decision-focused learning in RMAB that directly trains the predictive model to maximize the Whittle index solution quality. We present three key contributions: (i) we establish differentiability of the Whittle index policy to support decision-focused learning; (ii) we significantly improve the scalability of decision-focused learning approaches in sequential problems, specifically RMAB problems; (iii) we apply our algorithm to a previously collected dataset of maternal and child health to demonstrate its performance. Indeed, our algorithm is the first for decision-focused learning in RMAB that scales to real-world problem sizes.

cs.LG↗
arXiv2023

Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training. Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting. Existing CL methods can mitigate forgetting by replaying previous data. However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset. In addition, replaying data of previously learned downstream tasks can enhance their performance but comes at the cost of sacrificing zero-shot performance. To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space. In the feature space, a reference dataset is introduced for distillation between the current and initial models. The reference dataset should have semantic diversity but no need to be labeled, seen in pre-training, or matched image-text pairs. In parameter space, we prevent a large parameter shift by averaging weights during the training. We propose a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different methods, where tasks are from various domains instead of class-separated in a single dataset. Our method outperforms other methods in the traditional class-incremental learning setting and the MTIL by 9.7% average score. Our code locates at https://github.com/Thunderbeee/ZSCL.

cs.CV↗
arXiv2023

Experimentally Realizing Convolution Processing in the Photonic Synthetic Frequency Dimension

Convolution is an essential operation in signal and image processing and consumes most of the computing power in convolutional neural networks. Photonic convolution has the promise of addressing computational bottlenecks and outperforming electronic implementations. Performing photonic convolution in the synthetic frequency dimension, which harnesses the dynamics of light in the spectral degrees of freedom for photons, can lead to highly compact devices. Here we experimentally realize convolution operations in the synthetic frequency dimension. Using a modulated ring resonator, we synthesize arbitrary convolution kernels using a pre-determined modulation waveform with high accuracy. We demonstrate the convolution computation between input frequency combs and synthesized kernels. We also introduce the idea of an additive offset to broaden the kinds of kernels that can be implemented experimentally when the modulation strength is limited. Our work demonstrate the use of synthetic frequency dimension to efficiently encode data and implement computation tasks, leading to a compact and scalable photonic computation architecture.

physics.optics↗
arXiv2023

Dissect two-halo galactic conformity effect for central galaxies: The dependence of star formation activities on the large-scale environment

We investigate the two-halo galactic conformity effect for central galaxies, which is the spatial correlation of the star formation activities for central galaxies to several Mpcs, by studying the dependence of the star formation activities of central galaxies on their large-scale structure in our local Universe using the SDSS data. Here we adopt a novel environment metric using only central galaxies quantified by the distance to the $n$-th nearest central galaxy. This metric measures the environment within an aperture from $\sim$ 1 Mpc to $\gtrsim$ 10 Mpc, with a median value of $\sim$ 4 Mpc. We found that two kinds of conformity effects in our local Universe. The first one is that low-mass central galaxies are more quenched in high-density regions, and we found that this effect mainly comes from low-mass centrals that are close to a more massive halo. A similar trend is also found in the IllustrisTNG simulation, which can be entirely explained by backsplash galaxies. The second conformity effect is that massive central galaxies in low-density regions are more star-forming. This population of galaxies also possesses a higher fraction of spiral morphology and lower central stellar velocity dispersion, suggesting that their low quiescent fraction is due to less-frequent major merger events experienced in the low-density regions, and as a consequence, less-massive bulges and central black holes.

astro-ph.GA↗
arXiv2023

Environmental dependence of the mass-metallicity relation in cosmological hydrodynamical simulations

We investigate the environmental dependence of the gas-phase metallicity for galaxies at $z=0$ to $z\gtrsim 2$ and the underlying physical mechanisms driving this dependence using state-of-the-art cosmological hydrodynamical simulations. We find that, at fixed stellar mass, central galaxies in massive halos have lower gas-phase metallicity than those in low-mass halos. On the contrary, satellite galaxies residing in more massive halos are more metal-rich. The combined effect is that massive galaxies are more metal-poor in massive halos, and low-mass galaxies are more metal-rich in massive halos. By inspecting the environmental dependence of other galaxy properties, we identify that the accretion of low-metallicity gas is responsible for the environmental dependence of central galaxies at high $z$, whereas the AGN feedback processes play a crucial role at low $z$. For satellite galaxies, we find that both the suppression of gas accretion and the stripping of existing gas are responsible for their environmental dependence, with negligible effect from the AGN feedback. Finally, we show that the difference of gas-phase metallicity as a function of stellar mass between protocluster and field galaxies agrees with recent observational results, for example from the MAMMOTH-Grism survey.

astro-ph.GA↗
arXiv2023

DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for Medical Image Analysis

Limited labeled data makes it hard to train models from scratch in medical domain, and an important paradigm is pre-training and then fine-tuning. Large pre-trained models contain rich representations, which can be adapted to downstream medical tasks. However, existing methods either tune all the parameters or the task-specific layers of the pre-trained models, ignoring the input variations of medical images, and thus they are not efficient or effective. In this work, we aim to study parameter-efficient fine-tuning (PEFT) for medical image analysis, and propose a dynamic visual prompt tuning method, named DVPT. It can extract knowledge beneficial to downstream tasks from large models with a few trainable parameters. Firstly, the frozen features are transformed by an lightweight bottleneck layer to learn the domain-specific distribution of downstream medical tasks, and then a few learnable visual prompts are used as dynamic queries and then conduct cross-attention with the transformed features, attempting to acquire sample-specific knowledge that are suitable for each sample. Finally, the features are projected to original feature dimension and aggregated with the frozen features. This DVPT module can be shared between different Transformer layers, further reducing the trainable parameters. To validate DVPT, we conduct extensive experiments with different pre-trained models on medical classification and segmentation tasks. We find such PEFT method can not only efficiently adapt the pre-trained models to the medical domain, but also brings data efficiency with partial labeled data. For example, with 0.5\% extra trainable parameters, our method not only outperforms state-of-the-art PEFT methods, even surpasses the full fine-tuning by more than 2.20\% Kappa score on medical classification task. It can saves up to 60\% labeled data and 99\% storage cost of ViT-B/16.

cs.CV↗
arXiv2023

Implementation and Evaluation of Networked Model Predictive Control System on Universal Robot

Networked control systems are closed-loop feedback control systems containing system components that may be distributed geographically in different locations and interconnected via a communication network such as the Internet. The quality of network communication is a crucial factor that significantly affects the performance of remote control. This is due to the fact that network uncertainties can occur in the transmission of packets in the forward and backward channels of the system. The two most significant among these uncertainties are network time delay and packet loss. To overcome these challenges, the networked predictive control system has been proposed to provide improved performance and robustness using predictive controllers and compensation strategies. In particular, the model predictive control method is well-suited as an advanced approach compared to conventional methods. In this paper, a networked model predictive control system consisting of a model predictive control method and compensation strategies is implemented to control and stabilize a robot arm as a physical system. In particular, this work aims to analyze the performance of the system under the influence of network time delay and packet loss. Using appropriate performance and robustness metrics, an in-depth investigation of the impacts of these network uncertainties is performed. Furthermore, the forward and backward channels of the network are examined in detail in this study.

cs.RO↗
arXiv2023

The 3rd Anti-UAV Workshop & Challenge: Methods and Results

The 3rd Anti-UAV Workshop & Challenge aims to encourage research in developing novel and accurate methods for multi-scale object tracking. The Anti-UAV dataset used for the Anti-UAV Challenge has been publicly released. There are two main differences between this year's competition and the previous two. First, we have expanded the existing dataset, and for the first time, released a training set so that participants can focus on improving their models. Second, we set up two tracks for the first time, i.e., Anti-UAV Tracking and Anti-UAV Detection & Tracking. Around 76 participating teams from the globe competed in the 3rd Anti-UAV Challenge. In this paper, we provide a brief summary of the 3rd Anti-UAV Workshop & Challenge including brief introductions to the top three methods in each track. The submission leaderboard will be reopened for researchers that are interested in the Anti-UAV challenge. The benchmark dataset and other information can be found at: https://anti-uav.github.io/.

cs.CV↗
arXiv2023

Evidential Detection and Tracking Collaboration: New Problem, Benchmark and Algorithm for Robust Anti-UAV System

Unmanned Aerial Vehicles (UAVs) have been widely used in many areas, including transportation, surveillance, and military. However, their potential for safety and privacy violations is an increasing issue and highly limits their broader applications, underscoring the critical importance of UAV perception and defense (anti-UAV). Still, previous works have simplified such an anti-UAV task as a tracking problem, where the prior information of UAVs is always provided; such a scheme fails in real-world anti-UAV tasks (i.e. complex scenes, indeterminate-appear and -reappear UAVs, and real-time UAV surveillance). In this paper, we first formulate a new and practical anti-UAV problem featuring the UAVs perception in complex scenes without prior UAVs information. To benchmark such a challenging task, we propose the largest UAV dataset dubbed AntiUAV600 and a new evaluation metric. The AntiUAV600 comprises 600 video sequences of challenging scenes with random, fast, and small-scale UAVs, with over 723K thermal infrared frames densely annotated with bounding boxes. Finally, we develop a novel anti-UAV approach via an evidential collaboration of global UAVs detection and local UAVs tracking, which effectively tackles the proposed problem and can serve as a strong baseline for future research. Extensive experiments show our method outperforms SOTA approaches and validate the ability of AntiUAV600 to enhance UAV perception performance due to its large scale and complexity. Our dataset, pretrained models, and source codes will be released publically.

cs.CV↗
arXiv2023

Analytical Constraints on the Radius and Bulk Lorentz Factor in the Lepto-Hadronic One-Zone Model of BL Lacs

In this work, we study the parameter space of neutrino-emitting BL Lacs under the framework of the one-zone lepto-hadronic model. We show that constraints on the model come from various aspects of observations such as the variability timescale of blazar flares, gamma-ray opacity and the spectral energy distribution of electromagnetic emission, as well as the inferred neutrino emissivity of the blazar. We apply our method to two potential neutrino sources, i.e., TXS 0506+056 and PKS 0735+178, which are BL Lacs. Then, we explore and summarize the allowed range of parameters such as the bulk Lorentz factor and the blob radius under different distributions of injected protons. We find that the parameter space that is available to explain the BL Lac--neutrino association is sensitive to the proton distribution, and usually, an injected proton luminosity significantly exceeding the Eddington luminosity is required for both sources. Our results suggest that the simple lepto-hadronic one-zone model may not be a reasonable interpretation for BL Lac--neutrino associations.

astro-ph.HE↗
arXiv2023

Gd-Based Solvated Shells for Defect Passivation of CsPbBr$_3$ Nanoplatelets Enabling Efficient Color-Saturated Blue Electroluminescence

Reduced-dimensional CsPbBr$_3$ nanoplatelets (NPLs) are promising candidates for color-saturated blue emitters, yet their electroluminescence performance is hampered by non-radiative recombination, which is associated with bromine vacancies. Here, we show that a post-synthetic treatment of CsPbBr$_3$ NPLs with GdBr$_3$-dimethylformamide (DMF) can effectively eliminate defects while preserving the color. According to a combined experimental and theoretical study, Gd$^{3+}$ ions are less reactive with NPLs as a result of compact interaction between them and DMF, and this stable Gd$^{3+}$-DMF solvation structure makes Brions more available and allows them to move more freely. Consequently, defects are rapidly passivated and photoluminescence quantum yield increases dramatically (from 35 to ~100%), while the surface ligand density and emission color remain unchanged. The result is a remarkable electroluminescence efficiency of 2.4% (at 464 nm), one of the highest in pure blue perovskite NPL light-emitting diodes. It is noteworthy that the conductive NPL film shows a high photoluminescence quantum yield of 80%, demonstrating NPLs' significant electroluminescence potential with further device structure design.

physics.app-ph↗
arXiv2023

Watt-class CMOS-compatible power amplifier

Power amplifier is becoming a critical component for integrated photonics as the integrated devices try to carve out a niche in the world of real-world applications of photonics. That is because the signal generated from an integrated device severely lacks in power which is due mainly to the small size which, although gives size and weight advantage, limits the energy storage capacity of an integrated device due to the small volume, causing it to rely on its bench-top counterpart for signal amplification downstream. Therefore, an integrated high-power signal booster can play a major role by replacing these large solid-state and fiber-based benchtop systems. For decades, large mode area (LMA) technology has played a disruptive role by increasing the signal power and energy by orders of magnitude in the fiber-based lasers and amplifiers. Thanks to the capability of LMA fiber to support significantly larger optical modes the energy storage and handling capability has significantly increased. Such an LMA device on an integrated platform can play an important role for high power applications. In this work, we demonstrate LMA waveguide based CMOS compatible watt-class power amplifier with an on-chip output power reaching ~ 1W within a footprint of ~4mm2.The power achieved is comparable and even surpasses many fiber-based amplifiers. We believe this work opens up opportunities for integrated photonics to find real world application on-par with its benchtop counterpart.

physics.optics↗
arXiv2023

BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency

As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to project various sensory modalities into a shared feature space. To achieve this, massive and correctly aligned data pairs are required for model training. However, unlike unimodal datasets, multimodal datasets are extremely harder to collect and annotate precisely. As an alternative, the co-occurred data pairs (e.g., image-text pairs) collected from the Internet have been widely exploited in the area. Unfortunately, the cheaply collected dataset unavoidably contains many mismatched data pairs, which have been proven to be harmful to the model's performance. To address this, we propose a general framework called BiCro (Bidirectional Cross-modal similarity consistency), which can be easily integrated into existing cross-modal matching models and improve their robustness against noisy data. Specifically, BiCro aims to estimate soft labels for noisy data pairs to reflect their true correspondence degree. The basic idea of BiCro is motivated by that -- taking image-text matching as an example -- similar images should have similar textual descriptions and vice versa. Then the consistency of these two similarities can be recast as the estimated soft labels to train the matching model. The experiments on three popular cross-modal matching datasets demonstrate that our method significantly improves the noise-robustness of various matching models, and surpass the state-of-the-art by a clear margin.

cs.CV↗