arXiv subjects

Yi Yang

Publications and source records attributed to Yi Yang.

At least 901 records · Page 50Linked to original sources

arXiv2020

ActBERT: Learning Global-Local Video-Text Representations

In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.

cs.CV↗

arXiv2020

Large-scale multilingual audio visual dubbing

We describe a system for large-scale audiovisual translation and dubbing, which translates videos from one language to another. The source language's speech content is transcribed to text, translated, and automatically synthesized into target language speech using the original speaker's voice. The visual content is translated by synthesizing lip movements for the speaker to match the translated audio, creating a seamless audiovisual experience in the target language. The audio and visual translation subsystems each contain a large-scale generic synthesis model trained on thousands of hours of data in the corresponding domain. These generic models are fine-tuned to a specific speaker before translation, either using an auxiliary corpus of data from the target speaker, or using the video to be translated itself as the input to the fine-tuning process. This report gives an architectural overview of the full system, as well as an in-depth discussion of the video dubbing component. The role of the audio and text components in relation to the full system is outlined, but their design is not discussed in detail. Translated and dubbed demo videos generated using our system can be viewed at https://www.youtube.com/playlist?list=PLSi232j2ZA6_1Exhof5vndzyfbxAhhEs5

cs.CV↗

arXiv2020

Entanglement Entropy Inequalities in BCFT by Holography

We study entanglement entropy inequalities in boundary conformal field theory (BCFT) by holographic correspondence. By carefully classifying all the configurations for different phases, we prove the strong subadditiviy and the monogamy of mutual information for holographic entanglement entropy in BCFT at both zero and finite temperatures

hep-th↗

arXiv2020

Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Domain adaptive semantic segmentation aims to train a model performing satisfactory pixel-level predictions on the target with only out-of-domain (source) annotations. The conventional solution to this task is to minimize the discrepancy between source and target to enable effective knowledge transfer. Previous domain discrepancy minimization methods are mainly based on the adversarial training. They tend to consider the domain discrepancy globally, which ignore the pixel-wise relationships and are less discriminative. In this paper, we propose to build the pixel-level cycle association between source and target pixel pairs and contrastively strengthen their connections to diminish the domain gap and make the features more discriminative. To the best of our knowledge, this is a new perspective for tackling such a challenging task. Experiment results on two representative domain adaptation benchmarks, i.e. GTAV $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes, verify the effectiveness of our proposed method and demonstrate that our method performs favorably against previous state-of-the-arts. Our method can be trained end-to-end in one stage and introduces no additional parameters, which is expected to serve as a general framework and help ease future research in domain adaptive semantic segmentation. Code is available at https://github.com/kgl-prml/Pixel- Level-Cycle-Association.

cs.CV↗

arXiv2020

Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classification

Corporate mergers and acquisitions (M&A) account for billions of dollars of investment globally every year, and offer an interesting and challenging domain for artificial intelligence. However, in these highly sensitive domains, it is crucial to not only have a highly robust and accurate model, but be able to generate useful explanations to garner a user's trust in the automated system. Regrettably, the recent research regarding eXplainable AI (XAI) in financial text classification has received little to no attention, and many current methods for generating textual-based explanations result in highly implausible explanations, which damage a user's trust in the system. To address these issues, this paper proposes a novel methodology for producing plausible counterfactual explanations, whilst exploring the regularization benefits of adversarial training on language models in the domain of FinTech. Exhaustive quantitative experiments demonstrate that not only does this approach improve the model accuracy when compared to the current state-of-the-art and human performance, but it also generates counterfactual explanations which are significantly more plausible based on human trials.

cs.CL↗

arXiv2020

Query Attack via Opposite-Direction Feature:Towards Robust Image Retrieval

Most existing works of adversarial samples focus on attacking image recognition models, while little attention is paid to the image retrieval task. In this paper, we identify two inherent challenges in applying prevailing image recognition attack methods to image retrieval. First, image retrieval demands discriminative visual features, which is significantly different from the one-hot class prediction in image recognition. Second, due to the disjoint and potentially unrelated classes between the training and test set in image retrieval, predicting the query category from predefined training classes is not accurate and leads to a sub-optimal adversarial gradient. To address these limitations, we propose a new white-box attack approach, Opposite-Direction Feature Attack (ODFA), to generate adversarial queries. Opposite-Direction Feature Attack (ODFA) effectively exploits feature-level adversarial gradients and takes advantage of feature distance in the representation space. To our knowledge, we are among the early attempts to design an attack method specifically for image retrieval. When we deploy an attacked image as the query, the true matches are prone to receive low ranks. We demonstrate through extensive experiments that (1) only crafting adversarial queries is sufficient to fool the state-of-the-art retrieval systems; (2) the proposed attack method, ODFA, leads to a higher attack success rate than classification attack methods, validating the necessity of leveraging characteristics of image retrieval; (3) the adversarial queries generated by our method have good transferability to other retrieval models without accessing their parameters, i.e.,the black-box setting.

cs.CV↗

arXiv2020

SN 2018fif: The Explosion of a Large Red Supergiant Discovered in Its Infancy by the Zwicky Transient Facility

High cadence transient surveys are able to capture supernovae closer to their first light than before. Applying analytical models to such early emission, we can constrain the progenitor stars properties. In this paper, we present observations of SN2018fif (ZTF18abokyfk). The supernova was discovered close to first light and monitored by the Zwicky Transient Facility (ZTF) and the Neil Gehrels Swift Observatory. Early spectroscopic observations suggest that the progenitor of SN2018fif was surrounded by relatively small amounts of circumstellar material (CSM) compared to all previous cases. This particularity, coupled with the high cadence multiple-band coverage, makes it a good candidate to investigate using shock-cooling models. We employ the SOPRANOS code, an implementation of the model by Sapir & Waxman and its extension to early times by Morag, Sapir & Waxman. Compared with previous implementations, SOPRANOS has the advantage of including a careful account of the limited temporal validity domain of the shock-cooling model as well as allowing usage of the entirety of the early UV data. We find that the progenitor of SN2018fif was a large red supergiant, with a radius of R=744.0_{-128.0}^{+183.0} solar radii and an ejected mass of Mej=9.3_{-5.8}^{+0.4} solar masses. Our model also gives information on the explosion epoch, the progenitor inner structure, the shock velocity and the extinction. The distribution of radii is double-peaked, with lower radii corresponding to lower values of the extinction, earlier recombination times and better match to the early UV data. If these correlations persist in future objects, denser spectroscopic monitoring constraining the time of recombination, as well as accurate UV observations (e.g. with ULTRASAT), will help break the radius-extinction degeneracy and independently determine both.

astro-ph.HE↗

arXiv2020

LID 2020: The Learning from Imperfect Data Challenge Results

Learning from imperfect data becomes an issue in many industrial applications after the research community has made profound progress in supervised learning from perfectly annotated datasets. The purpose of the Learning from Imperfect Data (LID) workshop is to inspire and facilitate the research in developing novel approaches that would harness the imperfect data and improve the data-efficiency during training. A massive amount of user-generated data nowadays available on multiple internet services. How to leverage those and improve the machine learning models is a high impact problem. We organize the challenges in conjunction with the workshop. The goal of these challenges is to find the state-of-the-art approaches in the weakly supervised learning setting for object detection, semantic segmentation, and scene parsing. There are three tracks in the challenge, i.e., weakly supervised semantic segmentation (Track 1), weakly supervised scene parsing (Track 2), and weakly supervised object localization (Track 3). In Track 1, based on ILSVRC DET, we provide pixel-level annotations of 15K images from 200 categories for evaluation. In Track 2, we provide point-based annotations for the training set of ADE20K. In Track 3, based on ILSVRC CLS-LOC, we provide pixel-level annotations of 44,271 images for evaluation. Besides, we further introduce a new evaluation metric proposed by \cite{zhang2020rethinking}, i.e., IoU curve, to measure the quality of the generated object localization maps. This technical report summarizes the highlights from the challenge. The challenge submission server and the leaderboard will continue to open for the researchers who are interested in it. More details regarding the challenge and the benchmarks are available at https://lidchallenge.github.io

cs.CV↗

arXiv2020

Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

This paper focuses on the unsupervised domain adaptation of transferring the knowledge from the source domain to the target domain in the context of semantic segmentation. Existing approaches usually regard the pseudo label as the ground truth to fully exploit the unlabeled target-domain data. Yet the pseudo labels of the target-domain data are usually predicted by the model trained on the source domain. Thus, the generated labels inevitably contain the incorrect prediction due to the discrepancy between the training domain and the test domain, which could be transferred to the final adapted model and largely compromises the training process. To overcome the problem, this paper proposes to explicitly estimate the prediction uncertainty during training to rectify the pseudo label learning for unsupervised semantic segmentation adaptation. Given the input image, the model outputs the semantic segmentation prediction as well as the uncertainty of the prediction. Specifically, we model the uncertainty via the prediction variance and involve the uncertainty into the optimization objective. To verify the effectiveness of the proposed method, we evaluate the proposed method on two prevalent synthetic-to-real semantic segmentation benchmarks, i.e., GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, as well as one cross-city benchmark, i.e., Cityscapes -> Oxford RobotCar. We demonstrate through extensive experiments that the proposed approach (1) dynamically sets different confidence thresholds according to the prediction variance, (2) rectifies the learning from noisy pseudo labels, and (3) achieves significant improvements over the conventional pseudo label learning and yields competitive performance on all three benchmarks.

cs.CV↗

arXiv2020

Symmetry governed valley-pseudospin textures of the full-zone excitonic bands of transition-metal dichalcogenide monolayers

Preserving a high degree of valley polarization of excitons in photo-excited transition-metal dichalcogenide monolayers (TMD-MLs) is desirable for the valley-based photonic applications, but widely recognized as a hard task hindered by the intrinsic electron-hole exchange interaction. In this study, we present a comprehensive investigation of valley-polarized finite-momentum excitons in WSe$_2$-MLs over the entire Brillouin zone by solving the density-functional-theory(DFT)-based Bethe-Salpeter equation (BSE) under the guidance of symmetry analysis. We reveal that finite-momentum excitons are actually in general well immune from the exchange-induced valley depolarization, except for those with specific exciton momenta directionally coincide with the axes associated with the $3σ_v$ and $3C_2'$ symmetries in TMD-MLs. Governed by the symmetries, the valley pseudo-spin texture of the full-zone exciton band in the momentum space is locally featured by individual skyrmion-like structures where highly valley-polarized finite-momentum exciton states are centred. Remarkably, we show that the high degrees of valley polarizations of the finite-momentum exciton states are excellently well transferable to the optical polarizations in the resulting phonon-assisted photo-luminescences, suggesting the prospective usefulness of those inter-valley excitons in valley-based photonics.

cond-mat.mes-hall↗

arXiv2020

Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior

Deep neural networks are known to be susceptible to adversarial noise, which are tiny and imperceptible perturbations. Most of previous work on adversarial attack mainly focus on image models, while the vulnerability of video models is less explored. In this paper, we aim to attack video models by utilizing intrinsic movement pattern and regional relative motion among video frames. We propose an effective motion-excited sampler to obtain motion-aware noise prior, which we term as sparked prior. Our sparked prior underlines frame correlations and utilizes video dynamics via relative motion. By using the sparked prior in gradient estimation, we can successfully attack a variety of video classification models with fewer number of queries. Extensive experimental results on four benchmark datasets validate the efficacy of our proposed method.

cs.CV↗

arXiv2020

Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

We present a simple few-shot named entity recognition (NER) system based on nearest neighbor learning and structured inference. Our system uses a supervised NER model trained on the source domain, as a feature extractor. Across several test domains, we show that a nearest neighbor classifier in this feature-space is far more effective than the standard meta-learning approaches. We further propose a cheap but effective method to capture the label dependencies between entity tags without expensive CRF training. We show that our method of combining structured decoding with nearest neighbor learning achieves state-of-the-art performance on standard few-shot NER evaluation tasks, improving F1 scores by $6\%$ to $16\%$ absolute points over prior meta-learning based systems.

cs.CL↗

arXiv2020

Feature Robust Optimal Transport for High-dimensional Data

Optimal transport is a machine learning problem with applications including distribution comparison, feature selection, and generative adversarial networks. In this paper, we propose feature-robust optimal transport (FROT) for high-dimensional data, which solves high-dimensional OT problems using feature selection to avoid the curse of dimensionality. Specifically, we find a transport plan with discriminative features. To this end, we formulate the FROT problem as a min--max optimization problem. We then propose a convex formulation of the FROT problem and solve it using a Frank--Wolfe-based optimization algorithm, whereby the subproblem can be efficiently solved using the Sinkhorn algorithm. Since FROT finds the transport plan from selected features, it is robust to noise features. To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence. By conducting synthetic and benchmark experiments, we demonstrate that the proposed method can find a strong correspondence by determining important layers. We show that the FROT algorithm achieves state-of-the-art performance in real-world semantic correspondence datasets.

stat.ML↗

arXiv2020

Dialog Intent Induction with Deep Multi-View Clustering

We introduce the dialog intent induction task and present a novel deep multi-view clustering approach to tackle the problem. Dialog intent induction aims at discovering user intents from user query utterances in human-human conversations such as dialogs between customer support agents and customers. Motivated by the intuition that a dialog intent is not only expressed in the user query utterance but also captured in the rest of the dialog, we split a conversation into two independent views and exploit multi-view clustering techniques for inducing the dialog intent. In particular, we propose alternating-view k-means (AV-KMEANS) for joint multi-view representation learning and clustering analysis. The key innovation is that the instance-view representations are updated iteratively by predicting the cluster assignment obtained from the alternative view, so that the multi-view representations of the instances lead to similar cluster assignments. Experiments on two public datasets show that AV-KMEANS can induce better dialog intent clusters than state-of-the-art unsupervised representation learning methods and standard multi-view clustering approaches.

cs.CL↗

arXiv2020

Non-Abelian Generalizations of the Hofstadter model: Spin-orbit-coupled Butterfly Pairs

The Hofstadter model, well-known for its fractal butterfly spectrum, describes two-dimensional electrons under a perpendicular magnetic field, which gives rise to the integer quantum hall effect. Inspired by the real-space building blocks of non-Abelian gauge fields from a recent experiment [Science, 365, 1021 (2019)], we introduce and theoretically study two non-Abelian generalizations of the Hofstadter model. Each model describes two pairs of Hofstadter butterflies that are spin-orbit coupled. In contrast to the original Hofstadter model that can be equivalently studied in the Landau and symmetric gauges, the corresponding non-Abelian generalizations exhibit distinct spectra due to the non-commutativity of the gauge fields. We derive the genuine (necessary and sufficient) non-Abelian condition for the two models from the commutativity of their arbitrary loop operators. At zero energy, the models are gapless and host Weyl and Dirac points protected by internal and crystalline symmetries. Double (8-fold), triple (12-fold), and quadrupole (16-fold) Dirac points also emerge, especially under equal hopping phases of the non-Abelian potentials. At other fillings, the gapped phases of the models give rise to $\mathbb{Z}_2$ topological insulators. We conclude by discussing possible schemes for the experimental realizations of the models in photonic platforms.

cond-mat.mes-hall↗

arXiv2020

Revisiting EmbodiedQA: A Simple Baseline and Beyond

In Embodied Question Answering (EmbodiedQA), an agent interacts with an environment to gather necessary information for answering user questions. Existing works have laid a solid foundation towards solving this interesting problem. But the current performance, especially in navigation, suggests that EmbodiedQA might be too challenging for the contemporary approaches. In this paper, we empirically study this problem and introduce 1) a simple yet effective baseline that achieves promising performance; 2) an easier and practical setting for EmbodiedQA where an agent has a chance to adapt the trained model to a new environment before it actually answers users questions. In this new setting, we randomly place a few objects in new environments, and upgrade the agent policy by a distillation network to retain the generalization ability from the trained model. On the EmbodiedQA v1 benchmark, under the standard setting, our simple baseline achieves very competitive results to the-state-of-the-art; in the new setting, we found the introduced small change in settings yields a notable gain in navigation.

cs.CV↗

arXiv2020

Tasks Integrated Networks: Joint Detection and Retrieval for Image Search

The traditional object retrieval task aims to learn a discriminative feature representation with intra-similarity and inter-dissimilarity, which supposes that the objects in an image are manually or automatically pre-cropped exactly. However, in many real-world searching scenarios (e.g., video surveillance), the objects (e.g., persons, vehicles, etc.) are seldom accurately detected or annotated. Therefore, object-level retrieval becomes intractable without bounding-box annotation, which leads to a new but challenging topic, i.e. image-level search. In this paper, to address the image search issue, we first introduce an end-to-end Integrated Net (I-Net), which has three merits: 1) A Siamese architecture and an on-line pairing strategy for similar and dissimilar objects in the given images are designed. 2) A novel on-line pairing (OLP) loss is introduced with a dynamic feature dictionary, which alleviates the multi-task training stagnation problem, by automatically generating a number of negative pairs to restrict the positives. 3) A hard example priority (HEP) based softmax loss is proposed to improve the robustness of classification task by selecting hard categories. With the philosophy of divide and conquer, we further propose an improved I-Net, called DC-I-Net, which makes two new contributions: 1) two modules are tailored to handle different tasks separately in the integrated framework, such that the task specification is guaranteed. 2) A class-center guided HEP loss (C2HEP) by exploiting the stored class centers is proposed, such that the intra-similarity and inter-dissimilarity can be captured for ultimate retrieval. Extensive experiments on famous image-level search oriented benchmark datasets demonstrate that the proposed DC-I-Net outperforms the state-of-the-art tasks-integrated and tasks-separated image search models.

cs.CV↗

arXiv2020

A large fraction of hydrogen-rich supernova progenitors experience elevated mass loss shortly prior to explosion

Spectroscopic detection of narrow emission lines traces the presence of circumstellar mass distributions around massive stars exploding as core-collapse supernovae. Transient emission lines disappearing shortly after the supernova explosion suggest that the spatial extent of such material is compact, and hence imply an increased mass loss shortly prior to explosion. Here, we present a systematic survey for such transient emission lines (Flash Spectroscopy) among Type II supernovae detected in the first year of the Zwicky Transient Facility (ZTF) survey. We find that at least six out of ten events for which a spectrum was obtained within two days of estimated explosion time show evidence for such transient flash lines. Our measured flash event fraction ($>30\%$ at $95\%$ confidence level) indicates that elevated mass loss is a common process occurring in massive stars that are about to explode as supernovae.

astro-ph.HE↗