arXiv subjects

Hang Liu

Publications and source records attributed to Hang Liu.

At least 145 records · Page 8Linked to original sources

arXiv2021

Conversational Query Rewriting with Self-supervised Learning

Context modeling plays a critical role in building multi-turn dialogue systems. Conversational Query Rewriting (CQR) aims to simplify the multi-turn dialogue modeling into a single-turn problem by explicitly rewriting the conversational query into a self-contained utterance. However, existing approaches rely on massive supervised training data, which is labor-intensive to annotate. And the detection of the omitted important information from context can be further improved. Besides, intent consistency constraint between contextual query and rewritten query is also ignored. To tackle these issues, we first propose to construct a large-scale CQR dataset automatically via self-supervised learning, which does not need human annotation. Then we introduce a novel CQR model Teresa based on Transformer, which is enhanced by self-attentive keywords detection and intent consistency constraint. Finally, we conduct extensive experiments on two public datasets. Experimental results demonstrate that our proposed model outperforms existing CQR baselines significantly, and also prove the effectiveness of self-supervised learning on improving the CQR performance.

cs.CL↗

arXiv2021

Semi-Blind Cascaded Channel Estimation for Reconfigurable Intelligent Surface Aided Massive MIMO

Reconfigurable intelligent surface (RIS) is envisioned to be a promising green technology to reduce the energy consumption and improve the coverage and spectral efficiency of massive multiple-input multiple-output (MIMO) wireless networks. In a RIS-aided MIMO system, the acquisition of channel state information (CSI) is important for achieving passive beamforming gains of the RIS, but is also challenging due to the cascaded property of the transmitter-RIS-receiver channel and the lack of signal processing capability of the passive RIS elements. The state-of-the-art approach for CSI acquisition in such a system is a pure training-based strategy that depends on a long sequence of pilot symbols. In this paper, we investigate semi-blind cascaded channel estimation for RIS-aided massive MIMO systems, in which the receiver simultaneously estimates the channel coefficients and the partially unknown transmit signal with a small number of pilot sequences. Specifically, we formulate the semi-blind cascaded channel estimation as a trilinear matrix factorization task. Under the Bayesian inference framework, we develop a computationally efficient iterative algorithm using the approximate message passing principle to resolve the trilinear inference problem. Meanwhile, we present an analytical framework to characterize the theoretical performance bound of the proposed approach in the large-system limit via the replica method developed in statistical physics. Extensive simulation results demonstrate the effectiveness of the proposed semi-blind cascaded channel estimation algorithm.

cs.IT↗

arXiv2021

SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks

The performance and efficiency of distributed training of Deep Neural Networks highly depend on the performance of gradient averaging among all participating nodes, which is bounded by the communication between nodes. There are two major strategies to reduce communication overhead: one is to hide communication by overlapping it with computation, and the other is to reduce message sizes. The first solution works well for linear neural architectures, but latest networks such as ResNet and Inception offer limited opportunity for this overlapping. Therefore, researchers have paid more attention to minimizing communication. In this paper, we present a novel gradient compression framework derived from insights of real gradient distributions, and which strikes a balance between compression ratio, accuracy, and computational overhead. Our framework has two major novel components: sparsification of gradients in the frequency domain, and a range-based floating point representation to quantize and further compress gradients frequencies. Both components are dynamic, with tunable parameters that achieve different compression ratio based on the accuracy requirement and systems' platforms, and achieve very high throughput on GPUs. We prove that our techniques guarantee the convergence with a diminishing compression ratio. Our experiments show that the proposed compression framework effectively improves the scalability of most popular neural networks on a 32 GPU cluster to the baseline of no compression, without compromising the accuracy and convergence speed.

cs.DC↗

arXiv2020

Acoustic black hole in Schwarzschild spacetime: quasi-normal modes, analogous Hawking radiation and shadows

Various properties of acoustic black holes constructed in Minkowski spacetime have been widely studied in the past decades. Recently the acoustic black holes in general spacetime were proposed . In this paper, we first investigate the basic characteristics of `curved' acoustic black hole in Schwarzschild spacetime, including the quasi-normal modes, grey-body factor and analogous Hawking radiation. We find that the signal of quasi-normal mode is weaker than that of Schwarzschild black hole. Moreover, as the tuning parameter increases, both the positive real part and negative imaginal part of the quasi-normal frequency approach to the horizonal axis, but they will not change sign. This means that all the perturbations could die off and the system is stable under those perturbations. Since the larger tuning parameter suppresses the effective potential barrier, so it enhances the grey-body factor. The energy emission rate of Hawking radiation does not monotonically increase of the tuning parameter because of the non-monotonicity of the Hawking temperature. Finally, as a first attempt, we study the acoustic black hole shadow. The radius of acoustic shadow becomes larger as the tuning parameter increases, because both the related acoustic horizon and the acoustic sphere become larger. Our studies could help us to further understand the near horizon geometrical features of the black hole. We also expect that our observations could be detected experimentally in the near future.

gr-qc↗

arXiv2020

Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0x with zero or minor accuracy degradation on certain task(s). Our proposed method is also orthogonal to existing compact pre-trained language models such as DistilBERT using knowledge distillation, since a further 1.79x average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suitable to deploy the final compressed model on resource-constrained edge devices.

cs.CL↗

arXiv2020

Rank-Based Testing for Semiparametric VAR Models: a measure transportation approach

We develop a class of tests for semiparametric vector autoregressive (VAR) models with unspecified innovation densities, based on the recent measure-transportation-based concepts of multivariate {\it center-outward ranks} and {\it signs}. We show that these concepts, combined with Le Cam's asymptotic theory of statistical experiments, yield novel testing procedures, which (a)~are valid under a broad class of innovation densities (possibly non-elliptical, skewed, and/or with infinite moments), (b)~are optimal (locally asymptotically maximin or most stringent) at selected ones, and (c) are robust against additive outliers. In order to do so, we establish a H\' ajek asymptotic representation result, of independent interest, for a general class of center-outward rank-based serial statistics. As an illustration, we consider the problems of testing the absence of serial correlation in multiple-output and possibly non-linear regression (an extension of the classical Durbin-Watson problem) and the sequential identification of the order $p$ of a vector autoregressive (VAR($p$)) model. A Monte Carlo comparative study of our tests and their routinely-applied Gaussian competitors demonstrates the benefits (in terms of size, power, and robustness) of our methodology; these benefits are particularly significant in the presence of asymmetric and leptokurtic innovation densities. A real data application concludes the paper.

math.ST↗

arXiv2020

Exploring non-singular black holes in gravitational perturbations

We calculate gravitational perturbation quasinormal modes (QNMs) of non-singular Bardeen black holes (BHs) and singularity-free BHs in conformal gravity and examine their spectra in wave dynamics comparing with standard BHs in general relativity. After testing the validity of the approximate signal-to-noise ratio (SNR) calculation for different space based interferometers, we discuss the SNR of non-singular BHs in single-mode gravitational waveform detections in LISA, TianQin and TaiJi. We explore the impact of the Bardeen parameter and the conformal factor on the behavior of the SNR and find that in comparison with the standard Schwarzschild BHs, the increase of non-singular parameters leads to higher SNR for more massive non-singular BHs. We also examine the effect of the galactic confusion noise on the SNR and find that a dip appears in SNR due to such noise. For non-singular BHs, with the increase of the non-singular parameters the dip emerges for more massive BHs. It suggests a wider range of mass of non-singular black holes whose SNR will not be lowered by galactic noise, which implies the SNR for the super massive black holes centered at our galaxy will less likely to be influenced by galactic noise in non-singular black holes models than singular case in general relativity. We conduct Fisher analysis which suggests that the non-singular black holes parameters can be detected accurately with measurement errors as small as $\sim10^{-4}-10^{-5}$. The detections of non-singular BHs are expected to be realized more likely by LISA or TaiJi.

gr-qc↗

arXiv2020

Echoes from phantom wormholes

We study the time evolution of the test scalar and electromagnetic fields perturbations in configurations of phantom wormholes surrounded by dark energy with state parameter $ω< -1$. We observe obvious signals of echoes reflecting wormholes properties and disclose the physical reasons behind such phenomena. In particular, we find that the dark energy equation of state has a clear imprint in echoes in wave perturbations. When $ω$ approaches the phantom divide $ω=-1$ from below, the delay time of echoes becomes longer. The echo of gravitational wave is likely to be detected in the near future, the signature of the dark energy equation of state in the echo spectrum can serve as a local measurement of the dark energy.

gr-qc↗

arXiv2020

C-SAW: A Framework for Graph Sampling and Random Walk on GPUs

Many applications require to learn, mine, analyze and visualize large-scale graphs. These graphs are often too large to be addressed efficiently using conventional graph processing technologies. Many applications have requirements to analyze, transform, visualize and learn large scale graphs. These graphs are often too large to be addressed efficiently using conventional graph processing technologies. Recent literatures convey that graph sampling/random walk could be an efficient solution. In this paper, we propose, to the best of our knowledge, the first GPU-based framework for graph sampling/random walk. First, our framework provides a generic API which allows users to implement a wide range of sampling and random walk algorithms with ease. Second, offloading this framework on GPU, we introduce warp-centric parallel selection, and two optimizations for collision migration. Third, towards supporting graphs that exceed GPU memory capacity, we introduce efficient data transfer optimizations for out-of-memory sampling, such as workload-aware scheduling and batched multi-instance sampling. In its entirety, our framework constantly outperforms the state-of-the-art projects. First, our framework provides a generic API which allows users to implement a wide range of sampling and random walk algorithms with ease. Second, offloading this framework on GPU, we introduce warp-centric parallel selection, and two novel optimizations for collision migration. Third, towards supporting graphs that exceed the GPU memory capacity, we introduce efficient data transfer optimizations for out-of-memory and multi-GPU sampling, such as workload-aware scheduling and batched multi-instance sampling. Taken together, our framework constantly outperforms the state of the art projects in addition to the capability of supporting a wide range of sampling and random walk algorithms.

cs.DC↗

arXiv2020

SAPAG: A Self-Adaptive Privacy Attack From Gradients

Distributed learning such as federated learning or collaborative learning enables model training on decentralized data from users and only collects local gradients, where data is processed close to its sources for data privacy. The nature of not centralizing the training data addresses the privacy issue of privacy-sensitive data. Recent studies show that a third party can reconstruct the true training data in the distributed machine learning system through the publicly-shared gradients. However, existing reconstruction attack frameworks lack generalizability on different Deep Neural Network (DNN) architectures and different weight distribution initialization, and can only succeed in the early training phase. To address these limitations, in this paper, we propose a more general privacy attack from gradient, SAPAG, which uses a Gaussian kernel based of gradient difference as a distance measure. Our experiments demonstrate that SAPAG can construct the training data on different DNNs with different weight initializations and on DNNs in any training phases.

cs.LG↗

arXiv2020

R-estimators in GARCH models; asymptotics, applications and bootstrapping

The quasi-maximum likelihood estimation is a commonly-used method for estimating GARCH parameters. However, such estimators are sensitive to outliers and their asymptotic normality is proved under the finite fourth moment assumption on the underlying error distribution. In this paper, we propose a novel class of estimators of the GARCH parameters based on ranks, called R-estimators, with the property that they are asymptotic normal under the existence of a more than second moment of the errors and are highly efficient. We also consider the weighted bootstrap approximation of the finite sample distributions of the R-estimators. We propose fast algorithms for computing the R-estimators and their bootstrap replicates. Both real data analysis and simulations show the superior performance of the proposed estimators under the normal and heavy-tailed distributions. Our extensive simulations also reveal excellent coverage rates of the weighted bootstrap approximations. In addition, we discuss empirical and simulation results of the R-estimators for the higher order GARCH models such as the GARCH~($2, 1$) and asymmetric models such as the GJR model.

math.ST↗

arXiv2020

Reconfigurable-Intelligent-Surface Empowered Wireless Communications: Challenges and Opportunities

Reconfigurable intelligent surfaces (RISs) are regarded as a promising emerging hardware technology to improve the spectrum and energy efficiency of wireless networks by artificially reconfiguring the propagation environment of electromagnetic waves. Due to the unique advantages in enhancing wireless channel capacity, RISs have recently become a hot research topic. In this article, we focus on three fundamental physical-layer challenges for the incorporation of RISs into wireless networks, namely, channel state information acquisition, passive information transfer, and low-complexity robust system design. We summarize the state-of-the-art solutions and explore potential research directions. Furthermore, we discuss other promising research directions of RISs, including edge intelligence and physical-layer security.

cs.IT↗

arXiv2020

EZLDA: Efficient and Scalable LDA on GPUs

LDA is a statistical approach for topic modeling with a wide range of applications. However, there exist very few attempts to accelerate LDA on GPUs which come with exceptional computing and memory throughput capabilities. To this end, we introduce EZLDA which achieves efficient and scalable LDA training on GPUs with the following three contributions: First, EZLDA introduces three-branch sampling method which takes advantage of the convergence heterogeneity of various tokens to reduce the redundant sampling task. Second, to enable sparsity-aware format for both D and W on GPUs with fast sampling and updating, we introduce hybrid format for W along with corresponding token partition to T and inverted index designs. Third, we design a hierarchical workload balancing solution to address the extremely skewed workload imbalance problem on GPU and scaleEZLDA across multiple GPUs. Taken together, EZLDA achieves superior performance over the state-of-the-art attempts with lower memory consumption.

cs.DC↗

arXiv2020

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory-constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduces the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07x and 81x improvement in performance and energy efficiency compared to CPU, and up to 8.80x improvement in energy efficiency compared to GPU.

cs.DC↗

arXiv2020

Information Freshness-Aware Task Offloading in Air-Ground Integrated Edge Computing Systems

This paper studies the problem of information freshness-aware task offloading in an air-ground integrated multi-access edge computing system, which is deployed by an infrastructure provider (InP). A third-party real-time application service provider provides computing services to the subscribed mobile users (MUs) with the limited communication and computation resources from the InP based on a long-term business agreement. Due to the dynamic characteristics, the interactions among the MUs are modelled by a non-cooperative stochastic game, in which the control policies are coupled and each MU aims to selfishly maximize its own expected long-term payoff. To address the Nash equilibrium solutions, we propose that each MU behaves in accordance with the local system states and conjectures, based on which the stochastic game is transformed into a single-agent Markov decision process. Moreover, we derive a novel online deep reinforcement learning (RL) scheme that adopts two separate double deep Q-networks for each MU to approximate the Q-factor and the post-decision Q-factor. Using the proposed deep RL scheme, each MU in the system is able to make decisions without a priori statistical knowledge of dynamics. Numerical experiments examine the potentials of the proposed scheme in balancing the age of information and the energy consumption.

eess.SP↗

arXiv2020

Center-Outward R-Estimation for Semiparametric VARMA Models

We propose a new class of R-estimators for semiparametric VARMA models in which the innovation density plays the role of the nuisance parameter. Our estimators are based on the novel concepts of multivariate center-outward ranks and signs. We show that these concepts, combined with Le Cam's asymptotic theory of statistical experiments, yield a class of semiparametric estimation procedures, which are efficient (at a given reference density), root-$n$ consistent, and asymptotically normal under a broad class of (possibly non elliptical) actual innovation densities. No kernel density estimation is required to implement our procedures. A Monte Carlo comparative study of our R-estimators and other routinely-applied competitors demonstrates the benefits of the novel methodology, in large and small sample. Proofs, computational aspects, and further numerical results are available in the supplementary material.

math.ST↗

arXiv2020

Matrix-Calibration-Based Cascaded Channel Estimation for Reconfigurable Intelligent Surface Assisted Multiuser MIMO

Reconfigurable intelligent surface (RIS) is envisioned to be an essential component of the paradigm for beyond 5G networks as it can potentially provide similar or higher array gains with much lower hardware cost and energy consumption compared with the massive multiple-input multiple-output (MIMO) technology. In this paper, we focus on one of the fundamental challenges, namely the channel acquisition, in an RIS-assisted multiuser MIMO system. The state-of-the-art channel acquisition approach in such a system with fully passive RIS elements estimates the cascaded transmitter-to-RIS and RIS-to-receiver channels by adopting excessively long training sequences. To estimate the cascaded channels with an affordable training overhead, we formulate the channel estimation problem in the RIS-assisted multiuser MIMO system as a matrix-calibration based matrix factorization task. By exploiting the information on the slow-varying channel components and the hidden channel sparsity, we propose a novel message-passing based algorithm to factorize the cascaded channels. Furthermore, we present an analytical framework to characterize the theoretical performance bound of the proposed estimator in the large-system limit. Finally, we conduct simulations to verify the high accuracy and efficiency of the proposed algorithm.

cs.IT↗

arXiv2020

Efficient decoy-states for the reference-frame-independent measurement-device-independent quantum key distribution

Reference-frame-independent measurement-device-independent quantum key distribution (RFI-MDI-QKD) is a novel protocol which eliminates all possible attacks on detector side and necessity of reference-frame alignment in source sides. However, its performance may degrade notably due to statistical fluctuations, since more parameters, e.g. yields and error rates for mismatched-basis events, must be accumulated to monitor the security. In this work, we find that the original decoy-states method estimates these yields over pessimistically since it ignores the potential relations between different bases. Through processing parameters of different bases jointly, the performance of RFI-MDI-QKD is greatly improved in terms of secret key rate and achievable distance when statistical fluctuations are considered. Our results pave an avenue towards practical RFI-MDI-QKD.

quant-ph↗