arXiv subjects

Yi Yang

Publications and source records attributed to Yi Yang.

At least 847 records · Page 47Linked to original sources

arXiv2021

Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference

Video-and-Language Inference is a recently proposed task for joint video-and-language understanding. This new task requires a model to draw inference on whether a natural language statement entails or contradicts a given video clip. In this paper, we study how to address three critical challenges for this task: judging the global correctness of the statement involved multiple semantic meanings, joint reasoning over video and subtitles, and modeling long-range relationships and complex social interactions. First, we propose an adaptive hierarchical graph network that achieves in-depth understanding of the video over complex interactions. Specifically, it performs joint reasoning over video and subtitles in three hierarchies, where the graph structure is adaptively adjusted according to the semantic structures of the statement. Secondly, we introduce semantic coherence learning to explicitly encourage the semantic coherence of the adaptive hierarchical graph network from three hierarchies. The semantic coherence learning can further improve the alignment between vision and linguistics, and the coherence across a sequence of video segments. Experimental results show that our method significantly outperforms the baseline by a large margin.

cs.CV↗

arXiv2021

Improving Weakly-supervised Object Localization via Causal Intervention

The recent emerged weakly supervised object localization (WSOL) methods can learn to localize an object in the image only using image-level labels. Previous works endeavor to perceive the interval objects from the small and sparse discriminative attention map, yet ignoring the co-occurrence confounder (e.g., bird and sky), which makes the model inspection (e.g., CAM) hard to distinguish between the object and context. In this paper, we make an early attempt to tackle this challenge via causal intervention (CI). Our proposed method, dubbed CI-CAM, explores the causalities among images, contexts, and categories to eliminate the biased co-occurrence in the class activation maps thus improving the accuracy of object localization. Extensive experiments on several benchmarks demonstrate the effectiveness of CI-CAM in learning the clear object boundaries from confounding contexts. Particularly, in CUB-200-2011 which severely suffers from the co-occurrence confounder, CI-CAM significantly outperforms the traditional CAM-based baseline (58.39% vs 52.4% in top-1 localization accuracy). While in more general scenarios such as ImageNet, CI-CAM can also perform on par with the state of the arts.

cs.CV↗

arXiv2021

Parameter-Efficient Person Re-identification in the 3D Space

People live in a 3D world. However, existing works on person re-identification (re-id) mostly consider the semantic representation learning in a 2D space, intrinsically limiting the understanding of people. In this work, we address this limitation by exploring the prior knowledge of the 3D body structure. Specifically, we project 2D images to a 3D space and introduce a novel parameter-efficient Omni-scale Graph Network (OG-Net) to learn the pedestrian representation directly from 3D point clouds. OG-Net effectively exploits the local information provided by sparse 3D points and takes advantage of the structure and appearance information in a coherent manner. With the help of 3D geometry information, we can learn a new type of deep re-id feature free from noisy variants, such as scale and viewpoint. To our knowledge, we are among the first attempts to conduct person re-identification in the 3D space. We demonstrate through extensive experiments that the proposed method (1) eases the matching difficulty in the traditional 2D space, (2) exploits the complementary information of 2D appearance and 3D structure, (3) achieves competitive results with limited parameters on four large-scale person re-id datasets, and (4) has good scalability to unseen datasets. Our code, models and generated 3D human data are publicly available at https://github.com/layumi/person-reid-3d .

cs.CV↗

arXiv2021

Resisting Out-of-Distribution Data Problem in Perturbation of XAI

With the rapid development of eXplainable Artificial Intelligence (XAI), perturbation-based XAI algorithms have become quite popular due to their effectiveness and ease of implementation. The vast majority of perturbation-based XAI techniques face the challenge of Out-of-Distribution (OoD) data -- an artifact of randomly perturbed data becoming inconsistent with the original dataset. OoD data leads to the over-confidence problem in model predictions, making the existing XAI approaches unreliable. To our best knowledge, the OoD data problem in perturbation-based XAI algorithms has not been adequately addressed in the literature. In this work, we address this OoD data problem by designing an additional module quantifying the affinity between the perturbed data and the original dataset distribution, which is integrated into the process of aggregation. Our solution is shown to be compatible with the most popular perturbation-based XAI algorithms, such as RISE, OCCLUSION, and LIME. Experiments have confirmed that our methods demonstrate a significant improvement in general cases using both computational and cognitive metrics. Especially in the case of degradation, our proposed approach demonstrates outstanding performance comparing to baselines. Besides, our solution also resolves a fundamental problem with the faithfulness indicator, a commonly used evaluation metric of XAI algorithms that appears to be sensitive to the OoD issue.

cs.AI↗

arXiv2021

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

Matching images and sentences demands a fine understanding of both modalities. In this paper, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image / text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss is hard for network learning, since it starts from the two heterogeneous features to build inter-modal relationship. To address this problem, we propose the instance loss which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image / text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this paper constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.

cs.CV↗

arXiv2021

A Brewster route to Cherenkov detectors

The Cherenkov effect enables a valuable tool, known as the Cherenkov detector, to identify high-energy particles via the measurement of the Cherenkov cone. However, the sensitivity and momentum coverage of such detectors are intrinsically limited by the refractive index of the host material. Especially, identifying particles with energy above multiple gigaelectronvolts requires host materials with a near-unity refractive index, which are often limited to large and bulky gas chambers. Overcoming this fundamental material limit is important for future particle detectors yet remains a long-standing scientific challenge. Here, we propose a different paradigm for Cherenkov detectors that utilizes the broadband angular filter made from stacks of variable one-dimensional photonic crystals. Owing to the Brewster effect, the angular filter is transparent only to Cherenkov photons from a precise incident angle, and particle identification is achieved by mapping each Cherenkov angle to the peak-intensity position of transmitted photons in the detection plane. This unique property of the angular filter is exceptionally beneficial to Cherenkov detection as it enables the realization of a non-dispersive pseudo refractive index over the entire visible spectrum. Moreover, such a pseudo refractive index can be flexibly tuned to arbitrary values, including those close to unity. Our angular-selective Brewster paradigm offers a feasible solution to implement compact and highly sensitive Cherenkov detectors especially in beam lines and it can cover a wide momentum range using readily available dielectric materials.

physics.optics↗

arXiv2021

Roadmap of Designing Cognitive Metrics for Explainable Artificial Intelligence (XAI)

More recently, Explainable Artificial Intelligence (XAI) research has shifted to focus on a more pragmatic or naturalistic account of understanding, that is, whether the stakeholders understand the explanation. This point is especially important for research on evaluation methods for XAI systems. Thus, another direction where XAI research can benefit significantly from cognitive science and psychology research is ways to measure understanding of users, responses and attitudes. These measures can be used to quantify explanation quality and as feedback to the XAI system to improve the explanations. The current report aims to propose suitable metrics for evaluating XAI systems from the perspective of the cognitive states and processes of stakeholders. We elaborate on 7 dimensions, i.e., goodness, satisfaction, user understanding, curiosity & engagement, trust & reliance, controllability & interactivity, and learning curve & productivity, together with the recommended subjective and objective psychological measures. We then provide more details about how we can use the recommended measures to evaluate a visual classification XAI system according to the recommended cognitive metrics.

cs.HC↗

arXiv2021

LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport

Estimating mutual information is an important statistics and machine learning problem. To estimate the mutual information from data, a common practice is preparing a set of paired samples $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^n \stackrel{\mathrm{i.i.d.}}{\sim} p(\mathbf{x},\mathbf{y})$. However, in many situations, it is difficult to obtain a large number of data pairs. To address this problem, we propose the semi-supervised Squared-loss Mutual Information (SMI) estimation method using a small number of paired samples and the available unpaired ones. We first represent SMI through the density ratio function, where the expectation is approximated by the samples from marginals and its assignment parameters. The objective is formulated using the optimal transport problem and quadratic programming. Then, we introduce the Least-Squares Mutual Information with Sinkhorn (LSMI-Sinkhorn) algorithm for efficient optimization. Through experiments, we first demonstrate that the proposed method can estimate the SMI without a large number of paired samples. Then, we show the effectiveness of the proposed LSMI-Sinkhorn algorithm on various types of machine learning problems such as image matching and photo album summarization. Code can be found at https://github.com/csyanbin/LSMI-Sinkhorn.

stat.ML↗

arXiv2021

Saying the Unseen: Video Descriptions via Dialog Agents

Current vision and language tasks usually take complete visual data (e.g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e.g., restricted view with fixed camera or intentional vision block for security concerns. As a step towards the more practical application scenarios, we introduce a novel task that aims to describe a video using the natural language dialog between two agents as a supplementary information source given incomplete visual data. Different from most existing vision-language tasks where AI systems have full access to images or video clips, which may reveal sensitive information such as recognizable human faces or voices, we intentionally limit the visual input for AI systems and seek a more secure and transparent information medium, i.e., the natural language dialog, to supplement the missing visual information. Specifically, one of the intelligent agents - Q-BOT - is given two semantic segmented frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has access to the entire video, assists Q-BOT to accomplish the goal by answering the asked questions. We introduce two different experimental settings with either a generative (i.e., agents generate questions and answers freely) or a discriminative (i.e., agents select the questions and answers from candidates) internal dialog generation process. With the proposed unified QA-Cooperative networks, we experimentally demonstrate the knowledge transfer process between the two dialog agents and the effectiveness of using the natural language dialog as a supplement for incomplete implicit visions.

cs.CV↗

arXiv2021

The generalized Klein-Gordon oscillator with position-dependent mass in a particular Gödel-type space-time

The relativistic quantum dynamics of the generalized Klein-Gordon (KG) oscillator having position-dependent mass in the Gödel-type space-time is investigated. We have presented the generalized KG oscillator in this space-time, and discussed the effect of Cornell potential and linear potential for our considered system. The modification from the parameters of position-dependent mass and characterizing the space-time for the energy spectrums are presented.

quant-ph↗

arXiv2021

Aharonov-Bohm effect on the generalized Duffin-Kemmer-Petiau oscillator in the Som-Raychaudhuri space-time

The generalized Duffin-Kemmer-Petiau (DKP) oscillator with electromagnetic interactions in the curved space-times are investigated. We introduce firstly the generalized DKP oscillator in Som-Raychaudhuri space-time with Cornell potential. Then, we consider the electromagnetic interactions into the generalized DKP oscillator. The energy eigenvalues and eigenfunction of our problem are obtained. The effect from the parameters of space-time, the frequency of oscillator, the Cornell potential and the magnetic flux on the energy eigenvalues have been analyzed. We find a analogs effect for the bound states from the Aharonov-Bohm effect in our considered system.

quant-ph↗

arXiv2021

Graph-based Joint Pandemic Concern and Relation Extraction on Twitter

Public concern detection provides potential guidance to the authorities for crisis management before or during a pandemic outbreak. Detecting people's concerns and attention from online social media platforms has been widely acknowledged as an effective approach to relieve public panic and prevent a social crisis. However, detecting concerns in time from massive information in social media turns out to be a big challenge, especially when sufficient manually labeled data is in the absence of public health emergencies, e.g., COVID-19. In this paper, we propose a novel end-to-end deep learning model to identify people's concerns and the corresponding relations based on Graph Convolutional Network and Bi-directional Long Short Term Memory integrated with Concern Graph. Except for the sequential features from BERT embeddings, the regional features of tweets can be extracted by the Concern Graph module, which not only benefits the concern detection but also enables our model to be high noise-tolerant. Thus, our model can address the issue of insufficient manually labeled data. We conduct extensive experiments to evaluate the proposed model by using both manually labeled tweets and automatically labeled tweets. The experimental results show that our model can outperform the state-of-art models on real-world datasets.

cs.CL↗

arXiv2021

DKP oscillator in the presence of a spinning cosmic string

We investigate the modification of gravitational fields generated by topological defects on a generalized Duffin-Kemmer-Petiau (DKP) oscillator for spin-0 particle under spinning cosmic string background. The generalized DKP oscillator equation under spinning cosmic string background is established, and the impact of the Cornell potential on the generalized DKP oscillator is presented. We give the influence of space-time and potential parameters on energy levels.

quant-ph↗

arXiv2021

Three-dimensional non-Abelian generalizations of the Hofstadter model: spin-orbit-coupled butterfly trios

We theoretically introduce and study a three-dimensional Hofstadter model with linearly varying non-Abelian gauge potentials along all three dimensions. The model can be interpreted as spin-orbit coupling among a trio of Hofstadter butterfly pairs since each Cartesian surface ($xy$, $yz$, or $zx$) of the model reduces to a two-dimensional non-Abelian Hofstadter problem. By evaluating the commutativity among arbitrary loop operators around all axes, we derive its genuine (necessary and sufficient) non-Abelian condition, namely, at least two out of the three hopping phases should be neither 0 nor $π$. Under different choices of gauge fields in either the Abelian or the non-Abelian regime, both weak and strong topological insulating phases are identified in the model.

cond-mat.mes-hall↗

arXiv2021

Inherently high valley polarizations of momentum-forbidden dark excitons in transition-metal dichalcogenide monolayers

High degree of valley polarization of optically active excitons in transition-metal dichalcogenide monolayers (TMD-MLs) is vital in valley-based photonic applications but known to be likely spoiled by the intrinsic electron-hole exchange interactions. In this study, we present a theoretical investigation of the valley and optical properties of finite-momentum dark excitons in WSe$_2$-MLs by solving the density-functional-theory(DFT)-based Bethe-Salpeter equation (BSE) under the guidance of symmetry analysis. %We reveal that, in general, finite-momentum excitons are actually well immune from the exchange-induced valley depolarization, except for those with specific exciton momenta coincident with the $3σ_v$ and $3C_2'$ symmetries in the $D_{3h}$ point group of TMD-MLs. We reveal that, unlike the bright exciton inevitably subjected to electron-hole exchange interaction, inter-valley finite-momentum dark excitons in WSe$_2$-MLs are well immune from the exchange-induced valley depolarization and inherently highly valley-polarized under the enforcement of the crystal symmetries. More importantly, the superior valley polarizations of the inter-valley dark excitons in WSe$_2$-MLs are shown almost fully transferable to the optical polarization in the phonon-assisted photo-luminescences because of the native suppression of exchange-induced depolarization in the second-order optical processes. The analysis of phonon-assisted photo-luminescences accounts for the recently observed brightness, high degree of optical polarization and long lifetime of the inter-valley dark exciton states in tungsten-based TMD-MLs.

cond-mat.mes-hall↗

arXiv2021

Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization

Cross-view geo-localization is to spot images of the same geographic target from different platforms, e.g., drone-view cameras and satellites. It is challenging in the large visual appearance changes caused by extreme viewpoint variations. Existing methods usually concentrate on mining the fine-grained feature of the geographic target in the image center, but underestimate the contextual information in neighbor areas. In this work, we argue that neighbor areas can be leveraged as auxiliary information, enriching discriminative clues for geolocalization. Specifically, we introduce a simple and effective deep neural network, called Local Pattern Network (LPN), to take advantage of contextual information in an end-to-end manner. Without using extra part estimators, LPN adopts a square-ring feature partition strategy, which provides the attention according to the distance to the image center. It eases the part matching and enables the part-wise representation learning. Owing to the square-ring partition design, the proposed LPN has good scalability to rotation variations and achieves competitive results on three prevailing benchmarks, i.e., University-1652, CVUSA and CVACT. Besides, we also show the proposed LPN can be easily embedded into other frameworks to further boost performance.

cs.CV↗

arXiv2021

CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes

Continuity of care is crucial to ensuring positive health outcomes for patients discharged from an inpatient hospital setting, and improved information sharing can help. To share information, caregivers write discharge notes containing action items to share with patients and their future caregivers, but these action items are easily lost due to the lengthiness of the documents. In this work, we describe our creation of a dataset of clinical action items annotated over MIMIC-III, the largest publicly available dataset of real clinical notes. This dataset, which we call CLIP, is annotated by physicians and covers 718 documents representing 100K sentences. We describe the task of extracting the action items from these documents as multi-aspect extractive summarization, with each aspect representing a type of action to be taken. We evaluate several machine learning models on this task, and show that the best models exploit in-domain language model pre-training on 59K unannotated documents, and incorporate context from neighboring sentences. We also propose an approach to pre-training data selection that allows us to explore the trade-off between size and domain-specificity of pre-training datasets for this task.

cs.CL↗

arXiv2021

Less is More: Sparse Sampling for Dense Reaction Predictions

Obtaining viewer responses from videos can be useful for creators and streaming platforms to analyze the video performance and improve the future user experience. In this report, we present our method for 2021 Evoked Expression from Videos Challenge. In particular, our model utilizes both audio and image modalities as inputs to predict emotion changes of viewers. To model long-range emotion changes, we use a GRU-based model to predict one sparse signal with 1Hz. We observe that the emotion changes are smooth. Therefore, the final dense prediction is obtained via linear interpolating the signal, which is robust to the prediction fluctuation. Albeit simple, the proposed method has achieved pearson's correlation score of 0.04430 on the final private test set.

cs.CV↗