arXiv subjects

Wei Ji

Publications and source records attributed to Wei Ji.

At least 127 records · Page 7Linked to original sources

arXiv2024

Grounding is All You Need? Dual Temporal Grounding for Video Dialog

In the realm of video dialog response generation, the understanding of video content and the temporal nuances of conversation history are paramount. While a segment of current research leans heavily on large-scale pretrained visual-language models and often overlooks temporal dynamics, another delves deep into spatial-temporal relationships within videos but demands intricate object trajectory pre-extractions and sidelines dialog temporal dynamics. This paper introduces the Dual Temporal Grounding-enhanced Video Dialog model (DTGVD), strategically designed to merge the strengths of both dominant approaches. It emphasizes dual temporal relationships by predicting dialog turn-specific temporal regions, filtering video content accordingly, and grounding responses in both video and dialog contexts. One standout feature of DTGVD is its heightened attention to chronological interplay. By recognizing and acting upon the dependencies between different dialog turns, it captures more nuanced conversational dynamics. To further bolster the alignment between video and dialog temporal dynamics, we've implemented a list-wise contrastive learning strategy. Within this framework, accurately grounded turn-clip pairings are designated as positive samples, while less precise pairings are categorized as negative. This refined classification is then funneled into our holistic end-to-end response generation mechanism. Evaluations using AVSD@DSTC-7 and AVSD@DSTC-8 datasets underscore the superiority of our methodology.

cs.CV↗

arXiv2024

Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach

A plethora of text-guided image editing methods has recently been developed by leveraging the impressive capabilities of large-scale diffusion-based generative models especially Stable Diffusion. Despite the success of diffusion models in producing high-quality images, their application to small object generation has been limited due to difficulties in aligning cross-modal attention maps between text and these objects. Our approach offers a training-free method that significantly mitigates this alignment issue with local and global attention guidance , enhancing the model's ability to accurately render small objects in accordance with textual descriptions. We detail the methodology in our approach, emphasizing its divergence from traditional generation techniques and highlighting its advantages. What's more important is that we also provide~\textit{SOEBench} (Small Object Editing), a standardized benchmark for quantitatively evaluating text-based small object generation collected from \textit{MSCOCO} and \textit{OpenImage}. Preliminary results demonstrate the effectiveness of our method, showing marked improvements in the fidelity and accuracy of small object generation compared to existing models. This advancement not only contributes to the field of AI and computer vision but also opens up new possibilities for applications in various industries where precise image generation is critical. We will release our dataset on our project page: \href{https://soebench.github.io/}{https://soebench.github.io/}.

cs.CV↗

arXiv2024

Controllable Synthesis of Submillimeter Ultrathin 2D Ferromagnetic Cr5Te8 Nanosheets by GaTe-Assisted CVD

2D non-van der Waals (vdW) Cr5Te8 has attracted widespread research interest for its air stability and thickness-dependent magnetic properties. However, exploring new methods for growing larger-scale ultrathin 2D Cr5Te8 remains challenging. Here, we selected GaTe powder as precursor to supply Te monomers and fabricated large-scale 2D Cr5Te8 nanosheets. By optimizing the growth temperature and source-substrate distance (DSS), we successfully achieved Cr5Te8 nanosheets with lateral sizes of up to ~0.19 mm and corresponding thicknesses down to ~4.8 nm. The role of GaTe enhances the efficient Te atoms concentration, which promoted the lateral growth of Cr5Te8 nanosheets. Furthermore, our findings reveal the presence of Cr5Te8 nanosheets exhibiting serrated edges and a stacked structure like wedding cakes. Magnetic property measurement revealed the intense out-of-plane ferromagnetism in Cr5Te8, with the Curie temperature (TC) of 172 K. This work paves a way for the controllable growth of the submillimeter ultrathin 2D ferromagnetic crystalline and lays the foundation for the future synthesis of millimeter ultrathin ferromagnets.

cond-mat.mes-hall↗

arXiv2024

Spin-dependent exotic interactions

Novel interactions beyond the four known fundamental forces in nature (electromagnetic, gravitational, strong and weak interactions), may arise due to "new physics" beyond the standard model, manifesting as a "fifth force". This review is focused on spin-dependent fifth forces mediated by exotic bosons such as spin-0 axions and axionlike particles and spin-1 Z' bosons, dark photons, or paraphotons. Many of these exotic bosons are candidates to explain the nature of dark matter and dark energy, and their interactions may violate fundamental symmetries. Spin-dependent interactions between fermions mediated by the exchange of exotic bosons have been investigated in a variety of experiments, particularly at the low-energy frontier. Experimental methods and tools used to search for exotic spin-dependent interactions, such as atomic comagnetometers, torsion balances, nitrogen-vacancy spin sensors, and precision atomic and molecular spectroscopy, are described. A complete set of interaction potentials, derived based on quantum field theory with minimal assumptions and characterized in terms of reduced coupling constants, are presented. A comprehensive summary of existing experimental and observational constraints on exotic spin-dependent interactions is given, illustrating the current research landscape and promising directions of further research.

hep-ph↗

arXiv2024

Normalized Solutions to Schrödinger Equations with General Nonlinearities in Bounded Domains via a Global Bifurcation Approach

We obtain the existence, nonexistence and multiplicity of positive solutions with prescribed mass for nonlinear Schrödinger equations in bounded domains via a global bifurcation approach. The nonlinearities in this paper can be mass supercritical, critical, subcritical or some mixes of these cases, and the equation can be autonomous or non-autonomous. This generalizes a result in Noris, Tavares and Verzini [\emph{Anal. PDE}, 7 (8) (2014) 1807-1838], where the equation is autonomous with homogeneous nonlinearities. Besides, we have proven some orbital stability or instability results.

math.AP↗

arXiv2024

Personalized Knowledge Tracing through Student Representation Reconstruction and Class Imbalance Mitigation

Knowledge tracing is a technique that predicts students' future performance by analyzing their learning process through historical interactions with intelligent educational platforms, enabling a precise evaluation of their knowledge mastery. Recent studies have achieved significant progress by leveraging powerful deep neural networks. These models construct complex input representations using questions, skills, and other auxiliary information but overlook individual student characteristics, which limits the capability for personalized assessment. Additionally, the available datasets in the field exhibit class imbalance issues. The models that simply predict all responses as correct without substantial effort can yield impressive accuracy. In this paper, we propose PKT, a novel approach for personalized knowledge tracing. PKT reconstructs representations from sequences of interactions with a tutoring platform to capture latent information about the students. Moreover, PKT incorporates focal loss to improve prioritize minority classes, thereby achieving more balanced predictions. Extensive experimental results on four publicly available educational datasets demonstrate the advanced predictive performance of PKT in comparison with 16 state-of-the-art models. To ensure the reproducibility of our research, the code is publicly available at https://anonymous.4open.science/r/PKT.

cs.LG↗

arXiv2024

Kagome bands and magnetism in MoTe$_{2-x}$ kagome monolayers

Kagome lattices facilitate various quantum phases, yet in bulk materials, their kagome flat-bands often interact with bulk bands, suppressing kagome electronic characteristics for hosting these phases. Here, we use density-functional-theory calculations to predict the geometric and electronic structures, as well as the topological and magnetic properties, of a series of MoTe$_{2-x}$ kagome monolayers formed by mirror-twin-boundary (MTB) loops. We analyze ten MTB-loop configurations of varying sizes and arrangements to assess their impact on various properties. Within the intrinsic bandgap of MoTe$_{2}$, we identify two sets of kagome bands, primarily originating from in-plane and out-of-plane Mo d-orbitals at MTB-loop edges and -vertices, respectively. Three configurations exhibit superior stability, while three others show comparable stability. Among these, four display bandgaps and potentially non-zero Z$_{2}$ topological invariants, suggesting possible topological phases, while the remaining two are metallic and feature Stoner magnetization. These findings guide the design of kagome-based two-dimensional materials with tunable electronic, topological, and magnetic properties.

cond-mat.mtrl-sci↗

arXiv2024

SpeechEE: A Novel Benchmark for Speech Event Extraction

Event extraction (EE) is a critical direction in the field of information extraction, laying an important foundation for the construction of structured knowledge bases. EE from text has received ample research and attention for years, yet there can be numerous real-world applications that require direct information acquisition from speech signals, online meeting minutes, interview summaries, press releases, etc. While EE from speech has remained under-explored, this paper fills the gap by pioneering a SpeechEE, defined as detecting the event predicates and arguments from a given audio speech. To benchmark the SpeechEE task, we first construct a large-scale high-quality dataset. Based on textual EE datasets under the sentence, document, and dialogue scenarios, we convert texts into speeches through both manual real-person narration and automatic synthesis, empowering the data with diverse scenarios, languages, domains, ambiences, and speaker styles. Further, to effectively address the key challenges in the task, we tailor an E2E SpeechEE system based on the encoder-decoder architecture, where a novel Shrinking Unit module and a retrieval-aided decoding mechanism are devised. Extensive experimental results on all SpeechEE subsets demonstrate the efficacy of the proposed model, offering a strong baseline for the task. At last, being the first work on this topic, we shed light on key directions for future research.

cs.MM↗

arXiv2024

Semantic Alignment for Multimodal Large Language Models

Research on Multi-modal Large Language Models (MLLMs) towards the multi-image cross-modal instruction has received increasing attention and made significant progress, particularly in scenarios involving closely resembling images (e.g., change captioning). Existing MLLMs typically follow a two-step process in their pipelines: first, extracting visual tokens independently for each input image, and then aligning these visual tokens from different images with the Large Language Model (LLM) in its textual feature space. However, the independent extraction of visual tokens for each image may result in different semantics being prioritized for different images in the first step, leading to a lack of preservation of linking information among images for subsequent LLM analysis. This issue becomes more serious in scenarios where significant variations exist among the images (e.g., visual storytelling). To address this challenge, we introduce Semantic Alignment for Multi-modal large language models (SAM). By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM. As the test bed, we propose a large-scale dataset named MmLINK consisting of 69K samples. Different from most existing datasets for MLLMs fine-tuning, our MmLINK dataset comprises multi-modal instructions with significantly diverse images. Extensive experiments on the group captioning task and the storytelling task prove the effectiveness of our SAM model, surpassing the state-of-the-art methods by a large margin (+37% for group captioning and +22% for storytelling on CIDEr score). Project page: https://mccartney01.github.io/SAM.

cs.CV↗

arXiv2024

De-fine: Decomposing and Refining Visual Programs with Auto-Feedback

Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks. Unlike end-to-end models that need task-specific data, it advances in performing visual processing and reasoning in an unsupervised manner. Current visual programming methods generate programs in a single pass for each task where the ability to evaluate and optimize based on feedback, unfortunately, is lacking, which consequentially limits their effectiveness for complex, multi-step problems. Drawing inspiration from benders decomposition, we introduce De-fine, a training-free framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. This model-agnostic approach can improve logical reasoning performance by integrating the strengths of multiple models. Our experiments across various visual tasks show that De-fine creates more robust programs. Moreover, viewing each feedback module as an independent agent will yield fresh prospects for the field of agent research.

cs.CV↗

arXiv2024

Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching

Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning visual and textual data. To address this pressing need, we introduce GeoText-1652, a new natural language-guided geo-localization benchmark. This dataset is systematically constructed through an interactive human-computer process leveraging Large Language Model (LLM) driven annotation techniques in conjunction with pre-trained vision models. GeoText-1652 extends the established University-1652 image dataset with spatial-aware text annotations, thereby establishing one-to-one correspondences between image, text, and bounding box elements. We further introduce a new optimization objective to leverage fine-grained spatial associations, called blending spatial matching, for region-level spatial relation matching. Extensive experiments reveal that our approach maintains a competitive recall rate comparing other prevailing cross-modality methods. This underscores the promising potential of our approach in elevating drone control and navigation through the seamless integration of natural language commands in real-world scenarios.

cs.CV↗

arXiv2024

Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation

The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.

cs.CV↗

arXiv2024

Real-space topology-engineering of skyrmionic spin textures in a van der Waals ferromagnet Fe3GaTe2

Realizing magnetic skyrmions in two-dimensional (2D) van der Waals (vdW) ferromagnets offers unparalleled prospects for future spintronic applications. The room-temperature ferromagnet Fe3GaTe2 provides an ideal platform for tailoring these magnetic solitons. Here, skyrmions of distinct topological charges are artificially introduced and spatially engineered using magnetic force microscopy (MFM). The skyrmion lattice is realized by specific field-cooling process, and can be further controllably erased and painted via delicate manipulation of tip stray field. The skyrmion lattice with opposite topological charges (S = +1 or -1) can be tailored at the target regions to form topological skyrmion junctions (TSJs) with specific configurations. The delicate interplay of TSJs and spin-polarized device current were finally investigated via the in-situ transport measurements, alongside the topological stability of TSJs. Our results demonstrate that Fe3GaTe2 not only serves as a potential building block for room-temperature skyrmion-based spintronic devices, but also presents promising prospects for Fe3GaTe2-based heterostructures with the engineered topological spin textures.

cond-mat.mtrl-sci↗

arXiv2024

Regulated magnetic anisotropy and charge density wave in uniformly fabricated Janus CrTeSe monolayer

Two-dimensional materials with Janus structure host novel physical properties due to their inversional symmetry breaking. However, it remains elusive to synthesize Janus monolayer crystals with tailored long-range magnetic orders. Here, we have developed a general method to fabricate uniform Janus CrTeSe monolayers by selective selenization of preformed CrTe2 monolayers with molecular beam epitaxy. The uniform Janus structure of CrTeSe with high crystal quality is confirmed by high-resolution scanning transmission electron microscopy. Spin-polarized scanning tunneling microscopy/spectroscopy measurements unveil that the Janus CrTeSe undergoes a charge density wave (CDW) transition and a robust antiferromagnetic order. The magnetic anisotropy of CrTeSe is drastically altered compared to monolayer CrTe2 by the breaking symmetries induced from the Janus structure and the CDW transition, as is substantiated with first principles calculations. Our research achieves the construction of large-area Janus structures, and artificially tailors the electronic and magnetic properties of Janus systems at the two-dimensional limit.

cond-mat.mtrl-sci↗

arXiv2024

DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Driving

In autonomous driving, deep models have shown remarkable performance across various visual perception tasks with the demand of high-quality and huge-diversity training datasets. Such datasets are expected to cover various driving scenarios with adverse weather, lighting conditions and diverse moving objects. However, manually collecting these data presents huge challenges and expensive cost. With the rapid development of large generative models, we propose DriveDiTFit, a novel method for efficiently generating autonomous Driving data by Fine-tuning pre-trained Diffusion Transformers (DiTs). Specifically, DriveDiTFit utilizes a gap-driven modulation technique to carefully select and efficiently fine-tune a few parameters in DiTs according to the discrepancy between the pre-trained source data and the target driving data. Additionally, DriveDiTFit develops an effective weather and lighting condition embedding module to ensure diversity in the generated data, which is initialized by a nearest-semantic-similarity initialization approach. Through progressive tuning scheme to refined the process of detail generation in early diffusion process and enlarging the weights corresponding to small objects in training loss, DriveDiTFit ensures high-quality generation of small moving objects in the generated data. Extensive experiments conducted on driving datasets confirm that our method could efficiently produce diverse real driving data. The source codes will be available at https://github.com/TtuHamg/DriveDiTFit.

cs.CV↗

arXiv2024

Ferromagnetism and correlated insulating states in monolayer Mo33Te56

Kagome lattices have an inherent two-dimensional nature. Despite previous realizations in the monolayer limit, their abilities to drive emergent electronic states such as correlated insulators have remained unobserved. Here, we report the experimental realization of a new structural phase of monolayer Mo33Te56, characterized by its virtually global uniformity as a mirror-twin boundary loop superlattice embedded in an H-MoTe2 monolayer. Through a combination of scanning tunnelling microscopy (STM) and theoretical calculations, we unveil a kagome geometry along with multiple associated sets of kagome flat bands. Crucially, the partial filling of these kagome bands induces ferromagnetism as revealed by spin-polarized STM, and leads to a correlated insulating state exhibiting a hard gap as large as 15 meV. Our findings represent a major advance in kagome materials, offering a framework with clearer band structures and more intrinsic two-dimensional properties for exploring flat-band physics.

cond-mat.mtrl-sci↗

arXiv2024

Constraining Ultralight Dark Matter through an Accelerated Resonant Search

Experiments aimed at detecting ultralight dark matter typically rely on resonant effects, which are sensitive to the dark matter mass that matches the resonance frequency. In this study, we investigate the nucleon couplings of ultralight axion dark matter using a magnetometer operating in a nuclear magnetic resonance (NMR) mode. Our approach involves the use of a $^{21}$Ne spin-based sensor, which features the lowest nuclear magnetic moment among noble-gas spins. This configuration allows us to achieve an ultrahigh sensitivity of 0.73 fT/Hz$^{1/2}$ at around 5 Hz, corresponding to energy resolution of approximately 1.5$\times 10^{-23}\,\rm{eV/Hz^{1/2}}$. Our analysis reveals that under certain conditions it is beneficial to scan the frequency with steps significantly larger than the resonance width. The analytical results are in agreement with experimental data and the scan strategy is potentially applicable to other resonant searches. Further, our study establishes stringent constraints on axion-like particles (ALP) in the 4.5--15.5 Hz Compton-frequency range coupling to neutrons and protons, improving on prior work by several-fold. Within a band around 4.6--6.6 Hz and around 7.5 Hz, our laboratory findings surpass astrophysical limits derived from neutron-star cooling. Hence, we demonstrate an accelerated resonance search for ultralight dark matter, achieving an approximately 30-fold increase in scanning step while maintaining competitive sensitivity.

hep-ph↗

arXiv2024

Axion-Like Particle Detection in Alkali-Noble-Gas Haloscopes

Revealing the essence of dark matter (DM) and dark energy is essential for understanding our universe. Ultralight (rest energy $<$10 eV) bosonic particles, including pseudoscalar axions and axion-like particles (ALPs) have emerged among leading candidates to explain the composition of DM and searching for them has become an important part of precision-measurement science. Ultrahigh-sensitivity alkali-noble-gas based comagnetometers and magnetometers are being used as powerful haloscopes, i.e., devices designed to search for DM present in the galactic halo. A broad variety of such devices include clock-comparison comagnetometers, self-compensating comagnetometers, hybrid-spin-resonance magnetometer, spin-exchange-relaxation-free magnetometers, nuclear magnetic-resonance magnetometers, Floquet magnetometers, masers, as well as devices like the cosmic axion spin-precession experiment (CASPEr) using liquid $^{129}$Xe, prepolarized via spin-exchange optical pumping with rubidium atoms. The combination of alkali metal and noble gas allows one to take the best advantage of the complementary properties of the two spin systems. This review summarizes the operational principles, experimental setups and the successful explorations of new physics using these haloscopes. Additionally, some limiting factors are pointed out for further improvement.

hep-ex↗