arXiv subjects

Yi Yang

Publications and source records attributed to Yi Yang.

At least 685 records · Page 38Linked to original sources

arXiv2023

Gloss-Free End-to-End Sign Language Translation

In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations. Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain coverage of translation datasets, thus handicapping real-world applications. To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign. The code and model will be available at https://github.com/HenryLittle/GloFE.

cs.CV↗

arXiv2023

PointGPT: Auto-regressively Generative Pre-training from Point Clouds

Large language models (LLMs) based on the generative pre-training transformer (GPT) have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task is proposed to pre-train transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, an extractor-generator based transformer decoder, with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming to predict the next one in an auto-regressive manner. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance on various downstream tasks. In particular, our approach achieves classification accuracies of 94.9% on the ModelNet40 dataset and 93.4% on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-art accuracies on all four few-shot learning benchmarks.

cs.CV↗

arXiv2023

VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending

Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. However, there is limited research on learning video-text representations for general video multimodal tasks based on these powerful features. Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks. Specifically, VLAB is founded on two key strategies: feature adapting and feature blending. In the former, we introduce a new video adapter module to address CLIP's deficiency in modeling temporal information and extend the model's capability to encompass both contrastive and generative tasks. In the latter, we propose an end-to-end training method that further enhances the model's performance by exploiting the complementarity of image and video features. We validate the effectiveness and versatility of VLAB through extensive experiments on highly competitive video multimodal tasks, including video text retrieval, video captioning, and video question answering. Remarkably, VLAB outperforms competing methods significantly and sets new records in video question answering on MSRVTT, MSVD, and TGIF datasets. It achieves an accuracy of 49.6, 61.0, and 79.0, respectively. Codes and models will be released.

cs.CV↗

arXiv2023

Non-Abelian physics in light and sound

There has been a recent surge of interest in using light and sound as platforms for studying non-Abelian physics. Through a kaleidoscope of physical effects, light and sound provide diverse ways to manipulate their degrees of freedom to constitute the Hilbert space for demonstrating non-Abelian phenomena. The review aims to provide a timely and comprehensive account of this emerging topic. Starting from the foundation of matrix-valued geometric phases, we cover non-Abelian topological charges, non-Abelian gauge fields, non-Abelian braiding, non-Hermitian non-Abelian phenomena, and their realizations with photonics and acoustics. This topic is fast evolving at the intersection of atomic, molecular, optical physics, condensed matter physics, and mathematical physics, with fascinating prospects ahead.

physics.optics↗

arXiv2023

PTGB: Pre-Train Graph Neural Networks for Brain Network Analysis

The human brain is the central hub of the neurobiological system, controlling behavior and cognition in complex ways. Recent advances in neuroscience and neuroimaging analysis have shown a growing interest in the interactions between brain regions of interest (ROIs) and their impact on neural development and disorder diagnosis. As a powerful deep model for analyzing graph-structured data, Graph Neural Networks (GNNs) have been applied for brain network analysis. However, training deep models requires large amounts of labeled data, which is often scarce in brain network datasets due to the complexities of data acquisition and sharing restrictions. To make the most out of available training data, we propose PTGB, a GNN pre-training framework that captures intrinsic brain network structures, regardless of clinical outcomes, and is easily adaptable to various downstream tasks. PTGB comprises two key components: (1) an unsupervised pre-training technique designed specifically for brain networks, which enables learning from large-scale datasets without task-specific labels; (2) a data-driven parcellation atlas mapping pipeline that facilitates knowledge transfer across datasets with different ROI systems. Extensive evaluations using various GNN models have demonstrated the robust and superior performance of PTGB compared to baseline methods.

q-bio.NC↗

arXiv2023

Pyramid Diffusion Models For Low-light Image Enhancement

Recovering noise-covered details from low-light images is challenging, and the results given by previous methods leave room for improvement. Recent diffusion models show realistic and detailed image generation through a sequence of denoising refinements and motivate us to introduce them to low-light image enhancement for recovering realistic details. However, we found two problems when doing this, i.e., 1) diffusion models keep constant resolution in one reverse process, which limits the speed; 2) diffusion models sometimes result in global degradation (e.g., RGB shift). To address the above problems, this paper proposes a Pyramid Diffusion model (PyDiff) for low-light image enhancement. PyDiff uses a novel pyramid diffusion method to perform sampling in a pyramid resolution style (i.e., progressively increasing resolution in one reverse process). Pyramid diffusion makes PyDiff much faster than vanilla diffusion models and introduces no performance degradation. Furthermore, PyDiff uses a global corrector to alleviate the global degradation that may occur in the reverse process, significantly improving the performance and making the training of diffusion models easier with little additional computational consumption. Extensive experiments on popular benchmarks show that PyDiff achieves superior performance and efficiency. Moreover, PyDiff can generalize well to unseen noise and illumination distributions.

cs.CV↗

arXiv2023

Efficient Multimodal Fusion via Interactive Prompting

Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. Following this trend, the size of multi-modal learning models constantly increases, leading to an urgent need to reduce the massive computational cost of finetuning these models for downstream tasks. In this paper, we propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pre-trained transformers. Specifically, we first present a modular multimodal fusion framework that exhibits high flexibility and facilitates mutual interactions among different modalities. In addition, we disentangle vanilla prompts into three types in order to learn different optimizing objectives for multimodal learning. It is also worth noting that we propose to add prompt vectors only on the deep layers of the unimodal transformers, thus significantly reducing the training memory usage. Experiment results show that our proposed method achieves comparable performance to several other multimodal finetuning methods with less than 3% trainable parameters and up to 66% saving of training memory usage.

cs.CV↗

arXiv2023

Feature-compatible Progressive Learning for Video Copy Detection

Video Copy Detection (VCD) has been developed to identify instances of unauthorized or duplicated video content. This paper presents our second place solutions to the Meta AI Video Similarity Challenge (VSC22), CVPR 2023. In order to compete in this challenge, we propose Feature-Compatible Progressive Learning (FCPL) for VCD. FCPL trains various models that produce mutually-compatible features, meaning that the features derived from multiple distinct models can be directly compared with one another. We find this mutual compatibility enables feature ensemble. By implementing progressive learning and utilizing labeled ground truth pairs, we effectively gradually enhance performance. Experimental results demonstrate the superiority of the proposed FCPL over other competitors. Our code is available at https://github.com/WangWenhao0716/VSC-DescriptorTrack-Submission and https://github.com/WangWenhao0716/VSC-MatchingTrack-Submission.

cs.CV↗

arXiv2023

Segment and Track Anything

This report presents a framework called Segment And Track Anything (SAMTrack) that allows users to precisely and effectively segment and track any object in a video. Additionally, SAM-Track employs multimodal interaction methods that enable users to select multiple objects in videos for tracking, corresponding to their specific requirements. These interaction methods comprise click, stroke, and text, each possessing unique benefits and capable of being employed in combination. As a result, SAM-Track can be used across an array of fields, ranging from drone technology, autonomous driving, medical imaging, augmented reality, to biological analysis. SAM-Track amalgamates Segment Anything Model (SAM), an interactive key-frame segmentation model, with our proposed AOT-based tracking model (DeAOT), which secured 1st place in four tracks of the VOT 2022 challenge, to facilitate object tracking in video. In addition, SAM-Track incorporates Grounding-DINO, which enables the framework to support text-based interaction. We have demonstrated the remarkable capabilities of SAM-Track on DAVIS-2016 Val (92.0%), DAVIS-2017 Test (79.2%)and its practicability in diverse applications. The project page is available at: https://github.com/z-x-yang/Segment-and-Track-Anything.

cs.CV↗

arXiv2023

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks, which is very practical in the medical domain. It can significantly reduce the requirement of large amounts of task-specific data by sufficiently sharing medical knowledge among different tasks. However, due to the challenges of designing strongly generalizable models with limited and complex medical data, most existing approaches tend to develop task-specific models. To take a step towards MAGI, we propose a new paradigm called Medical-knOwledge-enhanced mulTimOdal pretRaining (MOTOR). In MOTOR, we combine two kinds of basic medical knowledge, i.e., general and specific knowledge, in a complementary manner to boost the general pretraining process. As a result, the foundation model with comprehensive basic knowledge can learn compact representations from pretraining radiographic data for better cross-modal alignment. MOTOR unifies the understanding and generation, which are two kinds of core intelligence of an AI system, into a single medical foundation model, to flexibly handle more diverse medical tasks. To enable a comprehensive evaluation and facilitate further research, we construct a medical multimodal benchmark including a wide range of downstream tasks, such as chest x-ray report generation and medical visual question answering. Extensive experiments on our benchmark show that MOTOR obtains promising results through simple task-oriented adaptation. The visualization shows that the injected knowledge successfully highlights key information in the medical data, demonstrating the excellent interpretability of MOTOR. Our MOTOR successfully mimics the human practice of fulfilling a "medical student" to accelerate the process of becoming a "specialist". We believe that our work makes a significant stride in realizing MAGI.

cs.AI↗

arXiv2023

SN 2020udy: a SN Iax with strict limits on interaction consistent with a helium-star companion

Early observations of transient explosions can provide vital clues to their progenitor origins. In this paper we present the nearby Type Iax (02cx-like) supernova (SN), SN 2020udy that was discovered within hours ($\sim$7 hr) of estimated first light. An extensive dataset of ultra-violet, optical, and near-infrared observations was obtained, covering out to $\sim$150 d after explosion. SN 2020udy peaked at -17.86$\pm$0.43 mag in the r band and evolved similarly to other 'luminous' SNe Iax, such as SNe 2005hk and 2012Z. Its well-sampled early light curve allows strict limits on companion interaction to be placed. Main-sequence companion stars with masses of 2 and 6 M$_\odot$ are ruled out at all viewing angles, while a helium-star companion is allowed from a narrow range of angles (140-180$^\circ$ away from the companion). The spectra and light curves of SN2020udy are in good agreement with those of the 'N5def' deflagration model of a near Chandrasekhar-mass carbon-oxygen white dwarf. However, as has been seen in previous studies of similar luminosity events, SN 2020udy evolves slower than the model. Broad-band linear polarisation measurements taken at and after peak are consistent with no polarisation, in agreement with the predictions of the companion-star configuration from the early light curve measurements. The host galaxy environment is low metallicity and is consistent with a young stellar population. Overall, we find the most plausible explosion scenario to be the incomplete disruption of a CO white dwarf near the Chandrasekhar-mass limit, with a helium-star companion.

astro-ph.HE↗

arXiv2023

DETR with Additional Global Aggregation for Cross-domain Weakly Supervised Object Detection

This paper presents a DETR-based method for cross-domain weakly supervised object detection (CDWSOD), aiming at adapting the detector from source to target domain through weak supervision. We think DETR has strong potential for CDWSOD due to an insight: the encoder and the decoder in DETR are both based on the attention mechanism and are thus capable of aggregating semantics across the entire image. The aggregation results, i.e., image-level predictions, can naturally exploit the weak supervision for domain alignment. Such motivated, we propose DETR with additional Global Aggregation (DETR-GA), a CDWSOD detector that simultaneously makes "instance-level + image-level" predictions and utilizes "strong + weak" supervisions. The key point of DETR-GA is very simple: for the encoder / decoder, we respectively add multiple class queries / a foreground query to aggregate the semantics into image-level predictions. Our query-based aggregation has two advantages. First, in the encoder, the weakly-supervised class queries are capable of roughly locating the corresponding positions and excluding the distraction from non-relevant regions. Second, through our design, the object queries and the foreground query in the decoder share consensus on the class semantics, therefore making the strong and weak supervision mutually benefit each other for domain alignment. Extensive experiments on four popular cross-domain benchmarks show that DETR-GA significantly improves CSWSOD and advances the states of the art (e.g., 29.0% --> 79.4% mAP on PASCAL VOC --> Clipart_all dataset).

cs.CV↗

arXiv2023

Early-Time Ultraviolet and Optical Hubble Space Telescope Spectroscopy of the Type II Supernova 2022wsp

We report early-time ultraviolet (UV) and optical spectroscopy of the young, nearby Type II supernova (SN) 2022wsp obtained by the Hubble Space Telescope (HST)/STIS at about 10 and 20 days after the explosion. The SN 2022wsp UV spectra are compared to those of other well-observed Type II/IIP SNe, including the recently studied Type IIP SN 2021yja. Both SNe exhibit rapid cooling and similar evolution during early phases, indicating a common behavior among SNe II. Radiative-transfer modeling of the spectra of SN 2022wsp with the TARDIS code indicates a steep radial density profile in the outer layer of the ejecta, a supersolar metallicity, and a relatively high total extinction of E(B-V) = 0.35 mag. The early-time evolution of the photospheric velocity and temperature derived from the modeling agree with the behavior observed from other previously studied cases. The strong suppression of hydrogen Balmer lines in the spectra suggests interaction with a pre-existing circumstellar environment could be occurring at early times. In the SN 2022wsp spectra, the absorption component of the Mg II P Cygni profile displays a double-trough feature on day +10 that disappears by day +20. The shape is well reproduced by the model without fine-tuning the parameters, suggesting that the secondary blueward dip is a metal transition that originates in the SN ejecta.

astro-ph.HE↗

arXiv2023

Explanation Strategies for Image Classification in Humans vs. Current Explainable AI

Explainable AI (XAI) methods provide explanations of AI models, but our understanding of how they compare with human explanations remains limited. In image classification, we found that humans adopted more explorative attention strategies for explanation than the classification task itself. Two representative explanation strategies were identified through clustering: One involved focused visual scanning on foreground objects with more conceptual explanations diagnostic for inferring class labels, whereas the other involved explorative scanning with more visual explanations rated higher for effectiveness. Interestingly, XAI saliency-map explanations had the highest similarity to the explorative attention strategy in humans, and explanations highlighting discriminative features from invoking observable causality through perturbation had higher similarity to human strategies than those highlighting internal features associated with higher class score. Thus, humans differ in information and strategy use for explanations, and XAI methods that highlight features informing observable causality match better with human explanations, potentially more accessible to users.

cs.HC↗

arXiv2023

Stringy scaling of n-point hard string scattering amplitudes

Motivated by the recent calculation of the SL(K+3,C) symmetry of n-point Lauricella string scattering amplitudes (SSA) of open bosonic string theory, we calculate ratios of the solvable infinite linear relations of arbitrary n-point hard SSA (HSSA). We discover a general stringy scaling behavior for all n-point HSSA to all string loop orders. For the special case of n=4, the stringy scaling behavior reduces to the infinite linear relations and constant ratios among HSSA conjectured by Gross [8] and later corrected and calculated by the method of decoupling of zero-norm states [11].

hep-th↗

arXiv2023

GIF: A General Graph Unlearning Strategy via Influence Function

With the greater emphasis on privacy and security in our society, the problem of graph unlearning -- revoking the influence of specific data on the trained GNN model, is drawing increasing attention. However, ranging from machine unlearning to recently emerged graph unlearning methods, existing efforts either resort to retraining paradigm, or perform approximate erasure that fails to consider the inter-dependency between connected neighbors or imposes constraints on GNN structure, therefore hard to achieve satisfying performance-complexity trade-offs. In this work, we explore the influence function tailored for graph unlearning, so as to improve the unlearning efficacy and efficiency for graph unlearning. We first present a unified problem formulation of diverse graph unlearning tasks \wrt node, edge, and feature. Then, we recognize the crux to the inability of traditional influence function for graph unlearning, and devise Graph Influence Function (GIF), a model-agnostic unlearning method that can efficiently and accurately estimate parameter changes in response to a $ε$-mass perturbation in deleted data. The idea is to supplement the objective of the traditional influence function with an additional loss term of the influenced neighbors due to the structural dependency. Further deductions on the closed-form solution of parameter changes provide a better understanding of the unlearning mechanism. We conduct extensive experiments on four representative GNN models and three benchmark datasets to justify the superiority of GIF for diverse graph unlearning tasks in terms of unlearning efficacy, model utility, and unlearning efficiency. Our implementations are available at \url{https://github.com/wujcan/GIF-torch/}.

cs.LG↗

arXiv2023

Synthetic non-Abelian gauge fields for non-Hermitian systems

Non-Abelian gauge fields are versatile tools for synthesizing topological phenomena but have so far been mostly studied in Hermitian systems, where gauge flux has to be defined from a closed loop in order for gauge fields, whether Abelian or non-Abelian, to become physically meaningful. We show that this condition can be relaxed in non-Hermitian systems by proposing and studying a generalized Hatano--Nelson model with imbalanced non-Abelian hopping. Despite lacking gauge flux in one dimension, non-Abelian gauge fields create rich non-Hermitian topological consequences. Under only nearest-neighbor coupling, non-Abelian gauge fields enable Hopf-link bulk braiding topology, whose phase transition accompanies the emergence of exceptional points (EPs). At both ends of an open chain, non-Abelian gauge fields lead to the simultaneous presence of non-Hermitian skin modes, whose population can be effectively tuned. Asymptotic analysis shows that this tuning mechanism stems from the interplay between the Abelian Hatano--Nelson coupling and effective high-order hopping, which becomes substantial near the EP phase transition condition. The predicted non-Hermitian phenomena, enabled by non-Abelian gauge fields, could be realized in synthetic dimensional optical platforms such as time-multiplexed photonic mesh lattices and driven ring resonators.

physics.optics↗

arXiv2023

The $SL(K+3,\mathbb{C})$ symmetry of string scatterings from D-branes

By using the solvability of Lauricella function $F_{D}^{(K)}\left( α;β_{1},...,β_{K};γ;x_{1},...,x_{K}\right) $ with nonpositive integer $β_{J}$, we show that each scattering or decay process of string and D-brane states at \textit{arbitrary} mass levels can be expressed in terms of a single Lauricella function. This result extends the previous exact $SL(K+3,\mathbb{C})$ symmetry of tree-level open bosonic string theory to include the D-brane.

hep-th↗