arXiv subjects

Ke Li

Publications and source records attributed to Ke Li.

At least 325 records · Page 18Linked to original sources

arXiv2022

Efficient Decoder-free Object Detection with Transformers

Vision transformers (ViTs) are changing the landscape of object detection approaches. A natural usage of ViTs in detection is to replace the CNN-based backbone with a transformer-based backbone, which is straightforward and effective, with the price of bringing considerable computation burden for inference. More subtle usage is the DETR family, which eliminates the need for many hand-designed components in object detection but introduces a decoder demanding an extra-long time to converge. As a result, transformer-based object detection can not prevail in large-scale applications. To overcome these issues, we propose a novel decoder-free fully transformer-based (DFFT) object detector, achieving high efficiency in both training and inference stages, for the first time. We simplify objection detection into an encoder-only single-level anchor-based dense prediction problem by centering around two entry points: 1) Eliminate the training-inefficient decoder and leverage two strong encoders to preserve the accuracy of single-level feature map prediction; 2) Explore low-level semantic features for the detection task with limited computational resources. In particular, we design a novel lightweight detection-oriented transformer backbone that efficiently captures low-level features with rich semantics based on a well-conceived ablation study. Extensive experiments on the MS COCO benchmark demonstrate that DFFT_SMALL outperforms DETR by 2.5% AP with 28% computation cost reduction and more than $10$x fewer training epochs. Compared with the cutting-edge anchor-based detector RetinaNet, DFFT_SMALL obtains over 5.5% AP gain while cutting down 70% computation cost.

cs.CV↗

arXiv2022

Private Language Model Adaptation for Speech Recognition

Speech model adaptation is crucial to handle the discrepancy between server-side proxy training data and actual data received on local devices of users. With the use of federated learning (FL), we introduce an efficient approach on continuously adapting neural network language models (NNLMs) on private devices with applications on automatic speech recognition (ASR). To address the potential speech transcription errors in the on-device training corpus, we perform empirical studies on comparing various strategies of leveraging token-level confidence scores to improve the NNLM quality in the FL settings. Experiments show that compared with no model adaptation, the proposed method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two speech evaluation datasets, respectively. We also provide analysis in evaluating privacy guarantees of our presented procedure.

eess.AS↗

arXiv2022

Evolutionary Multi-Task Injection Testing on Web Application Firewalls

Web application firewall (WAF) plays an integral role nowadays to protect web applications from various malicious injection attacks such as SQL injection, XML injection, and PHP injection, to name a few. However, given the evolving sophistication of injection attacks and the increasing complexity of tuning a WAF, it is challenging to ensure that the WAF is free of injection vulnerabilities such that it will block all malicious injection attacks without wrongly affecting the legitimate message. Automatically testing the WAF is, therefore, a timely and essential task. In this paper, we propose DaNuoYi, an automatic injection testing tool that simultaneously generates test inputs for multiple types of injection attacks on a WAF. Our basic idea derives from the cross-lingual translation in the natural language processing domain. In particular, test inputs for different types of injection attacks are syntactically different but may be semantically similar. Sharing semantic knowledge across multiple programming languages can thus stimulate the generation of more sophisticated test inputs and discovering injection vulnerabilities of the WAF that are otherwise difficult to find. To this end, in DaNuoYi, we train several injection translation models by using multi-task learning that translates the test inputs between any pair of injection attacks. The model is then used by a novel multi-task evolutionary algorithm to co-evolve test inputs for different types of injection attacks facilitated by a shared mating pool and domain-specific mutation operators at each generation. We conduct experiments on three real-world open-source WAFs and six types of injection attacks, the results reveal that DaNuoYi generates up to 3.8x and 5.78x more valid test inputs (i.e., bypassing the underlying WAF) than its state-of-the-art single-task counterparts and the context-free grammar-based injection construction.

cs.NE↗

arXiv2022

MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

U-Nets have achieved tremendous success in medical image segmentation. Nevertheless, it may suffer limitations in global (long-range) contextual interactions and edge-detail preservation. In contrast, Transformer has an excellent ability to capture long-range dependencies by leveraging the self-attention mechanism into the encoder. Although Transformer was born to model the long-range dependency on the extracted feature maps, it still suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. This motivates us to design the efficiently Transformer-based UNet model and study the feasibility of Transformer-based network architectures for medical image segmentation tasks. To this end, we propose to self-distill a Transformer-based UNet for medical image segmentation, which simultaneously learns global semantic information and local spatial-detailed features. Meanwhile, a local multi-scale fusion block is first proposed to refine fine-grained details from the skipped connections in the encoder by the main CNN stem through self-distillation, only computed during training and removed at inference with minimal overhead. Extensive experiments on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best performance over previous state-of-the-art methods. Code and models are available at \url{https://github.com/wangn123/MISSU.git}

cs.CV↗

arXiv2022

The tracking detector of the FASER experiment

FASER is a new experiment designed to search for new light weakly-interacting long-lived particles (LLPs) and study high-energy neutrino interactions in the very forward region of the LHC collisions at CERN. The experimental apparatus is situated 480 m downstream of the ATLAS interaction-point aligned with the beam collision axis. The FASER detector includes four identical tracker stations constructed from silicon microstrip detectors. Three of the tracker stations form a tracking spectrometer, and enable FASER to detect the decay products of LLPs decaying inside the apparatus, whereas the fourth station is used for the neutrino analysis. The spectrometer has been installed in the LHC complex since March 2021, while the fourth station is not yet installed. FASER will start physics data taking when the LHC resumes operation in early 2022. This paper describes the design, construction and testing of the tracking spectrometer, including the associated components such as the mechanics, readout electronics, power supplies and cooling system.

physics.ins-det↗

arXiv2022

PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment. Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs. Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP. In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification top-1 accuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks. In particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.

cs.CV↗

arXiv2022

Data-Driven Evolutionary Multi-Objective Optimization Based on Multiple-Gradient Descent for Disconnected Pareto Fronts

Data-driven evolutionary multi-objective optimization (EMO) has been recognized as an effective approach for multi-objective optimization problems with expensive objective functions. The current research is mainly developed for problems with a 'regular' triangle-like Pareto-optimal front (PF), whereas the performance can significantly deteriorate when the PF consists of disconnected segments. Furthermore, the offspring reproduction in the current data-driven EMO does not fully leverage the latent information of the surrogate model. Bearing these considerations in mind, this paper proposes a data-driven EMO algorithm based on multiple-gradient descent. By leveraging the regularity information provided by the up-to-date surrogate model, it is able to progressively probe a set of well distributed candidate solutions with a convergence guarantee. In addition, its infill criterion recommends a batch of promising candidate solutions to conduct expensive objective function evaluations. Experiments on $33$ benchmark test problem instances with disconnected PFs fully demonstrate the effectiveness of our proposed method against four selected peer algorithms.

cs.NE↗

arXiv2022

Do We Really Need to Use Constraint Violation in Constrained Evolutionary Multi-Objective Optimization?

Constraint violation has been a building block to design evolutionary multi-objective optimization algorithms for solving constrained multi-objective optimization problems. However, it is not uncommon that the constraint violation is hardly approachable in real-world black-box optimization scenarios. It is unclear that whether the existing constrained evolutionary multi-objective optimization algorithms, whose environmental selection mechanism are built upon the constraint violation, can still work or not when the formulations of the constraint functions are unknown. Bearing this consideration in mind, this paper picks up four widely used constrained evolutionary multi-objective optimization algorithms as the baseline and develop the corresponding variants that replace the constraint violation by a crisp value. From our experiments on both synthetic and real-world benchmark test problems, we find that the performance of the selected algorithms have not been significantly influenced when the constraint violation is not used to guide the environmental selection.

cs.NE↗

arXiv2022

Shifts in BCFW method for QED

We study the application of BCFW recursion relations to the QED processes $0\to e^- e^+ n γ$. Based on 6-point amplitudes (both MHVA and NMHVA) computed from Feynman diagrams in the Berends-Giele gauge, we conduct a comprehensive study on all different shifts. Then we propose a new shift (LLYZ shift) which can lead to the full amplitudes for these processes and can have some realistic computation advantages. We compare the number of terms and the independent amplitudes of this novel shift with a few typical shifts.

hep-th↗

arXiv2022

Scattering-assisted and logic-controllable WGM laser in liquid crystal micropillar

Whispering gallery mode (WGM) microcavities can efficiently store and manipulate light with strong light confinement and long photon lifetime, while coupling light into and from WGMs is intrinsically hindered by their unique feature of rotational symmetry. Here, a scattering-assisted liquid crystal (LC) micropillar WGM laser is proposed. WGM lasing at the surface of the micropillar is obviously enhanced by fluorescence scattering in the core of the micropillar. Besides, weak scattering of LC molecules also builds efficient coupling channels between the laser modes and the axial transmission modes of the micropillar-based waveguide, providing an all-in-one liquid WGM laser with functions of self-seeding and self-guiding. Furthermore, based on the hysteresis characteristics of the electrically anchored LC molecules under the interaction of thermal force, an erasable read-write liquid memory device is proposed, paving the way for the application of logic-controllable WGM lasers in optical storage and optical control.

physics.optics↗

arXiv2022

Anchor-free Oriented Proposal Generator for Object Detection

Oriented object detection is a practical and challenging task in remote sensing image interpretation. Nowadays, oriented detectors mostly use horizontal boxes as intermedium to derive oriented boxes from them. However, the horizontal boxes are inclined to get small Intersection-over-Unions (IoUs) with ground truths, which may have some undesirable effects, such as introducing redundant noise, mismatching with ground truths, detracting from the robustness of detectors, etc. In this paper, we propose a novel Anchor-free Oriented Proposal Generator (AOPG) that abandons horizontal box-related operations from the network architecture. AOPG first produces coarse oriented boxes by a Coarse Location Module (CLM) in an anchor-free manner and then refines them into high-quality oriented proposals. After AOPG, we apply a Fast R-CNN head to produce the final detection results. Furthermore, the shortage of large-scale datasets is also a hindrance to the development of oriented object detection. To alleviate the data insufficiency, we release a new dataset on the basis of our DIOR dataset and name it DIOR-R. Massive experiments demonstrate the effectiveness of AOPG. Particularly, without bells and whistles, we achieve the accuracy of 64.41%, 75.24% and 96.22% mAP on the DIOR-R, DOTA and HRSC2016 datasets respectively. Code and models are available at https://github.com/jbwang1997/AOPG.

cs.CV↗

arXiv2022

C3KG: A Chinese Commonsense Conversation Knowledge Graph

Existing commonsense knowledge bases often organize tuples in an isolated manner, which is deficient for commonsense conversational models to plan the next steps. To fill the gap, we curate a large-scale multi-turn human-written conversation corpus, and create the first Chinese commonsense conversation knowledge graph which incorporates both social commonsense knowledge and dialog flow information. To show the potential of our graph, we develop a graph-conversation matching approach, and benchmark two graph-grounded conversational tasks.

cs.CL↗

arXiv2022

Interactive Evolutionary Multi-Objective Optimization via Learning-to-Rank

In practical multi-criterion decision-making, it is cumbersome if a decision maker (DM) is asked to choose among a set of trade-off alternatives covering the whole Pareto-optimal front. This is a paradox in conventional evolutionary multi-objective optimization (EMO) that always aim to achieve a well balance between convergence and diversity. In essence, the ultimate goal of multi-objective optimization is to help a decision maker (DM) identify solution(s) of interest (SOI) achieving satisfactory trade-offs among multiple conflicting criteria. Bearing this in mind, this paper develops a framework for designing preference-based EMO algorithms to find SOI in an interactive manner. Its core idea is to involve human in the loop of EMO. After every several iterations, the DM is invited to elicit her feedback with regard to a couple of incumbent candidates. By collecting such information, her preference is progressively learned by a learning-to-rank neural network and then applied to guide the baseline EMO algorithm. Note that this framework is so general that any existing EMO algorithm can be applied in a plug-in manner. Experiments on $48$ benchmark test problems with up to 10 objectives fully demonstrate the effectiveness of our proposed algorithms for finding SOI.

cs.NE↗

arXiv2022

Probing highly collimated photon-jets with deep learning

Many extensions of the standard model (SM) predict the existence of axion-like particles and/or dark Higgs in the sub-GeV scale. Two new sub-GeV particles, a scalar and a pseudoscalar, produced through the Higgs boson exotic decays, are investigated. The decay signatures of these two new particles with highly collimated photons in the final states are discriminated from the ones of SM backgrounds using the Convolutional Neural Networks and Boosted Decision Trees techniques. The sensitivities of searching for such new physics signatures at the Large Hadron Collider are obtained.

hep-ph↗

arXiv2022

Streaming parallel transducer beam search with fast-slow cascaded encoders

Streaming ASR with strict latency constraints is required in many speech recognition applications. In order to achieve the required latency, streaming ASR models sacrifice accuracy compared to non-streaming ASR models due to lack of future input context. Previous research has shown that streaming and non-streaming ASR for RNN Transducers can be unified by cascading causal and non-causal encoders. This work improves upon this cascaded encoders framework by leveraging two streaming non-causal encoders with variable input context sizes that can produce outputs at different audio intervals (e.g. fast and slow). We propose a novel parallel time-synchronous beam search algorithm for transducers that decodes from fast-slow encoders, where the slow encoder corrects the mistakes generated from the fast encoder. The proposed algorithm, achieves up to 20% WER reduction with a slight increase in token emission delays on the public Librispeech dataset and in-house datasets. We also explore techniques to reduce the computation by distributing processing between the fast and slow encoders. Lastly, we explore sharing the parameters in the fast encoder to reduce the memory footprint. This enables low latency processing on edge devices with low computation cost and a low memory footprint.

cs.CL↗

arXiv2022

Training-free Transformer Architecture Search

Recently, Vision Transformer (ViT) has achieved remarkable success in several computer vision tasks. The progresses are highly relevant to the architecture design, then it is worthwhile to propose Transformer Architecture Search (TAS) to search for better ViTs automatically. However, current TAS methods are time-consuming and existing zero-cost proxies in CNN do not generalize well to the ViT search space according to our experimental observations. In this paper, for the first time, we investigate how to conduct TAS in a training-free manner and devise an effective training-free TAS (TF-TAS) scheme. Firstly, we observe that the properties of multi-head self-attention (MSA) and multi-layer perceptron (MLP) in ViTs are quite different and that the synaptic diversity of MSA affects the performance notably. Secondly, based on the observation, we devise a modular strategy in TF-TAS that evaluates and ranks ViT architectures from two theoretical perspectives: synaptic diversity and synaptic saliency, termed as DSS-indicator. With DSS-indicator, evaluation results are strongly correlated with the test accuracies of ViT models. Experimental results demonstrate that our TF-TAS achieves a competitive performance against the state-of-the-art manually or automatically design ViT architectures, and it promotes the searching efficiency in ViT search space greatly: from about $24$ GPU days to less than $0.5$ GPU days. Moreover, the proposed DSS-indicator outperforms the existing cutting-edge zero-cost approaches (e.g., TE-score and NASWOT).

cs.CV↗

arXiv2022

Study of Electroweak Phase Transition in Exotic Higgs Decays at the CEPC

A strong first-order electroweak phase transition (EWPT) can be induced by light new physics weakly coupled to the Higgs. This study focuses on a scenario in which the first-order EWPT is driven by a light scalar $s$ with a mass between 15-60 GeV. A search for exotic decays of the Higgs boson into a pair of spin-zero particles, $h \to ss$, where the $s$-boson decays into $b$-quarks promptly is presented. The search is performed in events where the Higgs boson is produced in association with a $Z$ boson, giving rise to a signature of two charged leptons (electrons or muons) and multiple jets from $b$-quark decays. The analysis is considering a scenario of analysing 5000 fb$^{-1}$ $e^+ e^-$ collision data at $\sqrt{s} = 240 $ GeV from the Circular Electron Positron Collider (CEPC). This study with $4b$ final state conclusively tests the expected sensitivity of probing the light scalars in the CEPC experiment. The sensitivity reach is significantly larger than that can be achieved at the LHC.

hep-ex↗

arXiv2022

Automated Few-Shot Time Series Forecasting based on Bi-level Programming

New micro-grid design with renewable energy sources and battery storage systems can help improve greenhouse gas emissions and reduce the operational cost. To provide an effective short-/long-term forecasting of both energy generation and load demand, time series predictive modeling has been one of the key tools to guide the optimal decision-making for planning and operation. One of the critical challenges of time series renewable energy forecasting is the lack of historical data to train an adequate predictive model. Moreover, the performance of a machine learning model is sensitive to the choice of its corresponding hyperparameters. Bearing these considerations in mind, this paper develops a BiLO-Auto-TSF/ML framework that automates the optimal design of a few-shot learning pipeline from a bi-level programming perspective. Specifically, the lower-level meta-learning helps boost the base-learner to mitigate the small data challenge while the hyperparameter optimization at the upper level proactively searches for the optimal hyperparameter configurations for both base- and meta-learners. Note that the proposed framework is so general that any off-the-shelf machine learning method can be used in a plug-in manner. Comprehensive experiments fully demonstrate the effectiveness of our proposed BiLO-Auto-TSF/ML framework to search for a high-performance few-shot learning pipeline for various energy sources.

cs.LG↗