arXiv subjects

Yi Yang

Publications and source records attributed to Yi Yang.

At least 757 records · Page 42Linked to original sources

arXiv2022

Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives

Reasoning about causal and temporal event relations in videos is a new destination of Video Question Answering (VideoQA).The major stumbling block to achieve this purpose is the semantic gap between language and video since they are at different levels of abstraction. Existing efforts mainly focus on designing sophisticated architectures while utilizing frame- or object-level visual representations. In this paper, we reconsider the multi-modal alignment problem in VideoQA from feature and sample perspectives to achieve better performance. From the view of feature,we break down the video into trajectories and first leverage trajectory feature in VideoQA to enhance the alignment between two modalities. Moreover, we adopt a heterogeneous graph architecture and design a hierarchical framework to align both trajectory-level and frame-level visual feature with language feature. In addition, we found that VideoQA models are largely dependent on language priors and always neglect visual-language interactions. Thus, two effective yet portable training augmentation strategies are designed to strengthen the cross-modal correspondence ability of our model from the view of sample. Extensive results show that our method outperforms all the state-of-the-art models on the challenging NExT-QA benchmark, which demonstrates the effectiveness of the proposed method.

cs.CV↗

arXiv2022

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

In this paper, we focus on unsupervised learning for Video Object Segmentation (VOS) which learns visual correspondence (i.e., the similarity between pixel-level features) from unlabeled videos. Previous methods are mainly based on the contrastive learning paradigm, which optimize either in image level or pixel level. Image-level optimization (e.g., the spatially pooled feature of ResNet) learns robust high-level semantics but is sub-optimal since the pixel-level features are optimized implicitly. By contrast, pixel-level optimization is more explicit, however, it is sensitive to the visual quality of training data and is not robust to object deformation. To complementarily perform these two levels of optimization in a unified framework, we propose the In-aNd-Out (INO) generative learning from a purely generative perspective with the help of naturally designed class tokens and patch tokens in Vision Transformer (ViT). Specifically, for image-level optimization, we force the out-view imagination from local to global views on class tokens, which helps capture high-level semantics, and we name it as out-generative learning. As to pixel-level optimization, we perform in-view masked image modeling on patch tokens, which recovers the corrupted parts of an image via inferring its fine-grained structure, and we term it as in-generative learning. To discover the temporal information better, we additionally force the inter-frame consistency from both feature and affinity matrix levels. Extensive experiments on DAVIS-2017 val and YouTube-VOS 2018 val show that our INO outperforms previous state-of-the-art methods by significant margins. Code is available: https://github.com/pansanity666/INO_VOS

cs.CV↗

arXiv2022

Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation

Recently, diffusion models have been proven to perform remarkably well in text-to-image synthesis tasks in a number of studies, immediately presenting new study opportunities for image generation. Google's Imagen follows this research trend and outperforms DALLE2 as the best model for text-to-image generation. However, Imagen merely uses a T5 language model for text processing, which cannot ensure learning the semantic information of the text. Furthermore, the Efficient UNet leveraged by Imagen is not the best choice in image processing. To address these issues, we propose the Swinv2-Imagen, a novel text-to-image diffusion model based on a Hierarchical Visual Transformer and a Scene Graph incorporating a semantic layout. In the proposed model, the feature vectors of entities and relationships are extracted and involved in the diffusion model, effectively improving the quality of generated images. On top of that, we also introduce a Swin-Transformer-based UNet architecture, called Swinv2-Unet, which can address the problems stemming from the CNN convolution operations. Extensive experiments are conducted to evaluate the performance of the proposed model by using three real-world datasets, i.e., MSCOCO, CUB and MM-CelebA-HQ. The experimental results show that the proposed Swinv2-Imagen model outperforms several popular state-of-the-art methods.

cs.CV↗

arXiv2022

Sparse Teachers Can Be Dense with Knowledge

Recent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the student-friendliness should be taken into consideration to realize a truly knowledgable teacher. Based on a pilot study, we find that over-parameterized teachers can produce expressive yet student-unfriendly knowledge and are thus limited in overall knowledgableness. To remove the parameters that result in student-unfriendliness, we propose a sparse teacher trick under the guidance of an overall knowledgable score for each teacher parameter. The knowledgable score is essentially an interpolation of the expressiveness and student-friendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Extensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines.

cs.CL↗

arXiv2022

Feature-Proxy Transformer for Few-Shot Segmentation

Few-shot segmentation (FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of "feature extractor $+$ linear classification head" and proposes a novel Feature-Proxy Transformer (FPTrans) method, in which the "proxy" is the vector representing a semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from the bottom to top layers using a novel prompting strategy. 2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods.

cs.CV↗

arXiv2022

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e., maximizing p(class|pixel feature). This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets. More impressively, without any modification, GMMSeg even performs well on open-world datasets. We believe this work brings fundamental insights into the related fields.

cs.CV↗

arXiv2022

Page Curve of Effective Hawking Radiation

We study the generalized entanglement entropy in the higher dimensional two-sided eternal black hole by double holography. By introducing an end-of-the-world ETW brane, which defines the time-dependent effective Hawking radiation region, we find a new type of Ryu-Takayanagi surface besides the Hartman-Maldacena surface and the island Ryu-Takayanagi surface known previously. We study the phase transition among the three Ryu-Takayanagi surfaces at different temperatures and obtain the phase diagram as well as the Page curve.

hep-th↗

arXiv2022

An 18.9-minute Blue Large-Amplitude Pulsator Crossing the 'Hertzsprung Gap' of Hot Subdwarfs

Blue large-amplitude pulsators (BLAPs) represent a new and rare class of hot pulsating stars with unusually large amplitudes and short periods. Up to now, only 24 confirmed BLAPs have been identified from more than one billion monitored stars, including a group with pulsation period longer than $\sim 20$ min (classical BLAPs, hereafter) and the other group with pulsation period below $\sim 8$ min. The evolutionary path that could give rise to such kinds of stellar configurations is unclear. Here we report on a comprehensive study of the peculiar BLAP discovered by the Tsinghua University - Ma Huateng Telescopes for Survey (TMTS), TMTS J035143.63+584504.2 (TMTS-BLAP-1). This new BLAP has an 18.9 min pulsation period and is similar to the BLAPs with a low surface gravity and an extended helium-enriched envelope, suggesting that it is a low-gravity BLAP at the shortest-period end. In particular, the long-term monitoring data reveal that this pulsating star has an unusually large rate of period change, P_dot/P=2.2e-6/yr. Such a significant and positive value challenges its origins from both helium-core pre-white-dwarfs and core helium-burning subdwarfs, but is consistent with that derived from shell helium-burning subdwarfs. The particular pulsation period and unusual rate of period change indicate that TMTS-BLAP-1 is at a short-lived (~10^6 yr) phase of shell-helium ignition before the stable shell-helium burning; in other words, TMTS-BLAP-1 is going through a "Hertzsprung gap" of hot subdwarfs.

astro-ph.SR↗

arXiv2022

Generating Safe Autonomous Decision-Making in ROS

The Robot Operating System (ROS) is a widely used framework for building robotic systems. It offers a wide variety of reusable packages and a pattern for new developments. It is up to developers how to combine these elements and integrate them with decision-making for autonomous behavior. The feature of such decision-making that is in general valued the most is safety assurance. In this research preview, we present a formal approach for generating safe autonomous decision-making in ROS. We first describe how to improve our existing static verification approach to verify multi-goal multi-agent decision-making. After that, we describe how to transition from the improved static verification approach to the proposed runtime verification approach. An initial implementation of this research proposal yields promising results.

cs.MA↗

arXiv2022

Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes

The manual annotation for large-scale point clouds costs a lot of time and is usually unavailable in harsh real-world scenarios. Inspired by the great success of the pre-training and fine-tuning paradigm in both vision and language tasks, we argue that pre-training is one potential solution for obtaining a scalable model to 3D point cloud downstream tasks as well. In this paper, we, therefore, explore a new self-supervised learning method, called Mixing and Disentangling (MD), for 3D point cloud representation learning. As the name implies, we mix two input shapes and demand the model learning to separate the inputs from the mixed shape. We leverage this reconstruction task as the pretext optimization objective for self-supervised learning. There are two primary advantages: 1) Compared to prevailing image datasets, eg, ImageNet, point cloud datasets are de facto small. The mixing process can provide a much larger online training sample pool. 2) On the other hand, the disentangling process motivates the model to mine the geometric prior knowledge, eg, key points. To verify the effectiveness of the proposed pretext task, we build one baseline network, which is composed of one encoder and one decoder. During pre-training, we mix two original shapes and obtain the geometry-aware embedding from the encoder, then an instance-adaptive decoder is applied to recover the original shapes from the embedding. Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks. The proposed method has improved the empirical performance on both ModelNet-40 and ShapeNet-Part datasets in terms of point cloud classification and segmentation tasks. We further conduct ablation studies to explore the effect of each component and verify the generalization of our proposed strategy by harnessing different backbones.

cs.CV↗

arXiv2022

Boost CTR Prediction for New Advertisements via Modeling Visual Content

Existing advertisements click-through rate (CTR) prediction models are mainly dependent on behavior ID features, which are learned based on the historical user-ad interactions. Nevertheless, behavior ID features relying on historical user behaviors are not feasible to describe new ads without previous interactions with users. To overcome the limitations of behavior ID features in modeling new ads, we exploit the visual content in ads to boost the performance of CTR prediction models. Specifically, we map each ad into a set of visual IDs based on its visual content. These visual IDs are further used for generating the visual embedding for enhancing CTR prediction models. We formulate the learning of visual IDs into a supervised quantization problem. Due to a lack of class labels for commercial images in advertisements, we exploit image textual descriptions as the supervision to optimize the image extractor for generating effective visual IDs. Meanwhile, since the hard quantization is non-differentiable, we soften the quantization operation to make it support the end-to-end network training. After mapping each image into visual IDs, we learn the embedding for each visual ID based on the historical user-ad interactions accumulated in the past. Since the visual ID embedding depends only on the visual content, it generalizes well to new ads. Meanwhile, the visual ID embedding complements the ad behavior ID embedding. Thus, it can considerably boost the performance of the CTR prediction models previously relying on behavior ID features for both new ads and ads that have accumulated rich user behaviors. After incorporating the visual ID embedding in the CTR prediction model of Baidu online advertising, the average CTR of ads improves by 1.46%, and the total charge increases by 1.10%.

cs.IR↗

arXiv2022

Adaptive Boosting for Domain Adaptation: Towards Robust Predictions in Scene Segmentation

Domain adaptation is to transfer the shared knowledge learned from the source domain to a new environment, i.e., target domain. One common practice is to train the model on both labeled source-domain data and unlabeled target-domain data. Yet the learned models are usually biased due to the strong supervision of the source domain. Most researchers adopt the early-stopping strategy to prevent over-fitting, but when to stop training remains a challenging problem since the lack of the target-domain validation set. In this paper, we propose one efficient bootstrapping method, called Adaboost Student, explicitly learning complementary models during training and liberating users from empirical early stopping. Adaboost Student combines the deep model learning with the conventional training strategy, i.e., adaptive boosting, and enables interactions between learned models and the data sampler. We adopt one adaptive data sampler to progressively facilitate learning on hard samples and aggregate "weak" models to prevent over-fitting. Extensive experiments show that (1) Without the need to worry about the stopping time, AdaBoost Student provides one robust solution by efficient complementary model learning during training. (2) AdaBoost Student is orthogonal to most domain adaptation methods, which can be combined with existing approaches to further improve the state-of-the-art performance. We have achieved competitive results on three widely-used scene segmentation domain adaptation benchmarks.

cs.CV↗

arXiv2022

Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Tickets

Over-parameterized models, typically pretrained language models (LMs), have shown an appealing expressive power due to their small learning bias. However, the huge learning capacity of LMs can also lead to large learning variance. In a pilot study, we find that, when faced with multiple domains, a critical portion of parameters behave unexpectedly in a domain-specific manner while others behave in a domain-general one. Motivated by this phenomenon, we for the first time posit that domain-general parameters can underpin a domain-general LM that can be derived from the original LM. To uncover the domain-general LM, we propose to identify domain-general parameters by playing lottery tickets (dubbed doge tickets). In order to intervene the lottery, we propose a domain-general score, which depicts how domain-invariant a parameter is by associating it with the variance. Comprehensive experiments are conducted on the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets obtains an improved out-of-domain generalization in comparison with a range of competitive baselines. Analysis results further hint the existence of domain-general parameters and the performance consistency of doge tickets.

cs.CL↗

arXiv2022

Tree-based Text-Vision BERT for Video Search in Baidu Video Advertising

The advancement of the communication technology and the popularity of the smart phones foster the booming of video ads. Baidu, as one of the leading search engine companies in the world, receives billions of search queries per day. How to pair the video ads with the user search is the core task of Baidu video advertising. Due to the modality gap, the query-to-video retrieval is much more challenging than traditional query-to-document retrieval and image-to-image search. Traditionally, the query-to-video retrieval is tackled by the query-to-title retrieval, which is not reliable when the quality of tiles are not high. With the rapid progress achieved in computer vision and natural language processing in recent years, content-based search methods becomes promising for the query-to-video retrieval. Benefited from pretraining on large-scale datasets, some visionBERT methods based on cross-modal attention have achieved excellent performance in many vision-language tasks not only in academia but also in industry. Nevertheless, the expensive computation cost of cross-modal attention makes it impractical for large-scale search in industrial applications. In this work, we present a tree-based combo-attention network (TCAN) which has been recently launched in Baidu's dynamic video advertising platform. It provides a practical solution to deploy the heavy cross-modal attention for the large-scale query-to-video search. After launching tree-based combo-attention network, click-through rate gets improved by 2.29\% and conversion rate get improved by 2.63\%.

cs.CV↗

arXiv2022

Non-Abelian nonsymmorphic chiral symmetries

The Hofstadter model exemplifies a large class of physical systems characterized by particles hopping on a lattice immersed in a gauge field. Recent advancements on various synthetic platforms have enabled highly-controllable simulations of such systems with tailored gauge fields featuring complex spatial textures. These synthetic gauge fields could introduce synthetic symmetries that do not appear in electronic materials. Here, in an SU(2) non-Abelian Hofstadter model, we theoretically show the emergence of multiple nonsymmorphic chiral symmetries, which combine an internal unitary anti-symmetry with fractional spatial translation. Depending on the values of the gauge fields, the nonsymmorphic chiral symmetries can exhibit non-Abelian algebra and protect Kramer quartet states in the bulk band structure, creating general four-fold degeneracy at all momenta. These nonsymmorphic chiral symmetries protect double Dirac semimetals at zero energy, which become gapped into quantum confined insulating phases upon introducing a boundary. Moreover, the parity of the system size can determine whether the resulting insulating phase is trivial or topological. Our work indicates a pathway for creating topology via synthetic symmetries emergent from synthetic gauge fields.

cond-mat.mes-hall↗

arXiv2022

ReLER@ZJU-Alibaba Submission to the Ego4D Natural Language Queries Challenge 2022

In this report, we present the ReLER@ZJU-Alibaba submission to the Ego4D Natural Language Queries (NLQ) Challenge in CVPR 2022. Given a video clip and a text query, the goal of this challenge is to locate a temporal moment of the video clip where the answer to the query can be obtained. To tackle this task, we propose a multi-scale cross-modal transformer and a video frame-level contrastive loss to fully uncover the correlation between language queries and video clips. Besides, we propose two data augmentation strategies to increase the diversity of training samples. The experimental results demonstrate the effectiveness of our method. The final submission ranked first on the leaderboard.

cs.CV↗

arXiv2022

Spectropolarimetry of the Thermonuclear Supernova 2021rhu: High Calcium Polarization 79 Days After Peak Luminosity

We report spectropolarimetric observations of the Type Ia supernova (SN) 2021rhu at four epochs: $-$7, +0, +36, and +79 days relative to its $B$-band maximum luminosity. A wavelength-dependent continuum polarization peaking at $3890 \pm 93$ Angstroms and reaching a level of $p_{\rm max}=1.78% \pm 0.02$% was found. The peak of the polarization curve is bluer than is typical in the Milky Way, indicating a larger proportion of small dust grains along the sightline to the SN. After removing the interstellar polarization, we found a pronounced increase of the polarization in the CaII near-infrared triplet, from $\sim$0.3% at day $-$7 to $\sim$2.5% at day +79. No temporal evolution in high-resolution flux spectra across the NaID and CaIIH&K features was seen from days +39 to +74, indicating that the late-time increase in polarization is intrinsic to the SN as opposed to being caused by scattering of SN photons in circumstellar or interstellar matter. We suggest that an explanation for the late-time rise of the CaII near-infrared triplet polarization may be the alignment of calcium atoms in a weak magnetic field through optical excitation/pumping by anisotropic radiation from the SN.

astro-ph.HE↗

arXiv2022

Free-electron-light interactions in nanophotonics

When impinging on optical structures or passing in their vicinity, free electrons can spontaneously emit electromagnetic radiation, a phenomenon generally known as cathodoluminescence. Free-electron radiation comes in many guises: Cherenkov, transition, and Smith-Purcell radiation, but also electron scintillation, commonly referred to as incoherent cathodoluminescence. While those effects have been at the heart of many fundamental discoveries and technological developments in high-energy physics in the past century, their recent demonstration in photonic and nanophotonic systems has attracted a lot of attention. Those developments arose from predictions that exploit nanophotonics for novel radiation regimes, now becoming accessible thanks to advances in nanofabrication. In general, the proper design of nanophotonic structures can enable shaping, control, and enhancement of free-electron radiation, for any of the above-mentioned effects. Free-electron radiation in nanophotonics opens the way to promising applications, such as widely-tunable integrated light sources from x-ray to THz frequencies, miniaturized particle accelerators, and highly sensitive high-energy particle detectors. Here, we review the emerging field of free-electron radiation in nanophotonics. We first present a general, unified framework to describe free-electron light-matter interaction in arbitrary nanophotonic systems. We then show how this framework sheds light on the physical underpinnings of many methods in the field used to control and enhance free-electron radiation. Namely, the framework points to the central role played by the photonic eigenmodes in controlling the output properties of free-electron radiation (e.g., frequency, directionality, and polarization). [... see full abstract in paper]

physics.optics↗