Search arXiv⌕ Search

arXiv subjects

Wei Ji

Publications and source records attributed to Wei Ji.

At least 163 records · Page 9Linked to original sources
arXiv2024

Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape

Accurately estimating the 3D pose and shape is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. However, research in this area is held back by the lack of a comprehensive and diverse dataset with high-quality 3D pose and shape annotations. In this paper, we propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. All annotations were labeled and checked manually in a multi-stage process to ensure highest quality results. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models. Our experimental results demonstrate that predicting the 3D shape and pose of animals across species remains a very challenging task, despite significant advances in human pose estimation. Our results further demonstrate that synthetic pre-training is a viable strategy to boost the model performance. Overall, Animal3D opens new directions for facilitating future research in animal 3D pose and shape estimation, and is publicly available.

cs.CV↗
arXiv2024

Levitated ferromagnetic magnetometer with energy resolution well below $\hbar$

A quantum limit on the measurement of magnetic field has been recently pointed out, stating that the so-called Energy Resolution $E_\mathrm{R}$ is bounded to $E_\mathrm{R} \gtrsim \hbar$. This limit holds indeed true for the vast majority of existing quantum magnetometers, including SQUIDs, solid state spins and optically pumped atomic magnetometers. However, it can be surpassed by highly correlated spin systems, as recently demonstrated with a single-domain spinor Bose-Einstein Condensate. Here we show that similar and potentially much better resolution can be achieved with a hard ferromagnet levitated above a superconductor at cryogenic temperature. We demonstrate $E_\mathrm{R}=\left( 0.064 \pm 0.010 \right) \, \hbar$ and anticipate that $E_\mathrm{R}<10^{-3} \, \hbar$ is within reach with near-future improvements. This finding opens the way to new applications in condensed matter, biophysics and fundamental science. In particular, we propose an experiment to search for axionlike dark matter and project a sensitivity orders of magnitude better than in previous searches.

quant-ph↗
arXiv2023

Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.

cs.CV↗
arXiv2023

MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer

The Diffusion Probabilistic Model (DPM) has recently gained popularity in the field of computer vision, thanks to its image generation applications, such as Imagen, Latent Diffusion Models, and Stable Diffusion, which have demonstrated impressive capabilities and sparked much discussion within the community. Recent investigations have further unveiled the utility of DPM in the domain of medical image analysis, as underscored by the commendable performance exhibited by the medical image segmentation model across various tasks. Although these models were originally underpinned by a UNet architecture, there exists a potential avenue for enhancing their performance through the integration of vision transformer mechanisms. However, we discovered that simply combining these two models resulted in subpar performance. To effectively integrate these two cutting-edge techniques for the Medical image segmentation, we propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities. Through comprehensive evaluation, our approach demonstrates superiority over prior state-of-the-art (SOTA) methodologies. Code is released at https://github.com/KidsWithTokens/MedSegDiff

eess.IV↗
arXiv2023

NExT-Chat: An LMM for Chat, Detection and Segmentation

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat.

cs.CV↗
arXiv2023

Identification of Carbon Stars from LAMOST DR7

Carbon stars are excellent kinematic tracers of galaxies and play important roles in understanding the evolution of the Galaxy. Therefore, it is worthwhile to search for them in a large amount of spectra. In this work, we build a new carbon star catalog based on the LAMOST DR7 spectra. The catalog contains 4542 spectra of 3546 carbon stars, identified through line index and near-infrared color-color diagrams. Through visual inspection of the spectra, we further subclassify them into 925 C--H, 384 C--R, 608 C--N, and 1292 Ba stars. However, 437 stars could not be sub-classified due to their low signal-to-noise. Moreover, by comparing with LAMOST DR7 pipeline we find 567 more carbon stars and visually sub-classify them. We find that on the $J-H$ vs. $H-K_{\rm s}$ two-color diagram, C--N stars can be reliably distinguished from the other three sub-types. Additionally, by utilizing the Gaia distance, we study the distribution of carbon stars in the H-R diagram and identify 258 dwarf carbon stars by the criterion $M_{\rm G}>$5.0\,mag. Finally, we present the spatial distribution in Galactic coordinates of the 3546 carbon stars. The majority of C-N, C-R, and Ba stars are distributed at low Galactic latitudes, while most C--H and dC stars distribute at high Galactic latitudes.

astro-ph.GA↗
arXiv2023

Domain-wise Invariant Learning for Panoptic Scene Graph Generation

Panoptic Scene Graph Generation (PSG) involves the detection of objects and the prediction of their corresponding relationships (predicates). However, the presence of biased predicate annotations poses a significant challenge for PSG models, as it hinders their ability to establish a clear decision boundary among different predicates. This issue substantially impedes the practical utility and real-world applicability of PSG models. To address the intrinsic bias above, we propose a novel framework to infer potentially biased annotations by measuring the predicate prediction risks within each subject-object pair (domain), and adaptively transfer the biased annotations to consistent ones by learning invariant predicate representation embeddings. Experiments show that our method significantly improves the performance of benchmark models, achieving a new state-of-the-art performance, and shows great generalization and effectiveness on PSG dataset.

cs.CV↗
arXiv2023

VPGTrans: Transfer Visual Prompt Generator across LLMs

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the MLLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing MLLMs for the target MLLM. In this work, we for the first time investigate the VPG transferability across LLMs, and explore a solution to reduce the cost of VPG transfer. We first study the VPG transfer across different LLM sizes (e.g., small-to-large), and across different LLM types, through which we diagnose the key factors to maximize the transfer efficiency. Based on our observation, we design a two-stage transfer framework named VPGTrans, which is simple yet highly effective. Through extensive experiments, we demonstrate that VPGTrans helps significantly speed up the transfer learning process without compromising performance. Remarkably, it helps achieve the VPG transfer from BLIP-2 OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and 10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from scratch. Further, a series of intriguing findings and potential rationales behind them are provided and discussed. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

cs.CV↗
arXiv2023

A Brief Review of Single Event Burnout Failure Mechanisms and Design Tolerances of Silicon Carbide MOSFETs

Radiation hardening of the MOSFET is of the highest priority for sustaining high-power systems in the space radiation environment. SiC-based power electronics are being looked at as a strong alternative for high power spaceborne power electronic systems. The SiC MOSFET has been shown to be most prone to SEB of the radiation effects. The knowledge of SiC MOSFET device degradation and failure mechanisms are reviewed. Additionally, the viability of rad-tolerant SiC MOSFET designs and the methods of SEB simulation are evaluated. A merit system is proposed to consider the performance of radiation tolerance and nominal electrical performance. Criteria needed for high-fidelity SEB simulation are also reviewed. This paper stands as a necessary analytical review to intercede the development of rad-hard power devices for space and extreme environment applications.

physics.ins-det↗
arXiv2023

Towards Complex-query Referring Image Segmentation: A Novel Benchmark

Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms. However, there has been a lack of research investigating how existing algorithms should be benchmarked with complex language queries, which include more informative descriptions of surrounding objects and backgrounds (\eg \textit{"the black car."} vs. \textit{"the black car is parking on the road and beside the bus."}). Given the significant improvement in the semantic understanding capability of large pre-trained models, it is crucial to take a step further in RIS by incorporating complex language that resembles real-world applications. To close this gap, building upon the existing RefCOCO and Visual Genome datasets, we propose a new RIS benchmark with complex queries, namely \textbf{RIS-CQ}. The RIS-CQ dataset is of high quality and large scale, which challenges the existing RIS with enriched, specific and informative queries, and enables a more realistic scenario of RIS research. Besides, we present a nichetargeting method to better task the RIS-CQ, called dual-modality graph alignment model (\textbf{\textsc{DuMoGa}}), which outperforms a series of RIS methods.

cs.CV↗
arXiv2023

Spin-resolved imaging of atomic-scale helimagnetism in monolayer NiI2

Identifying intrinsic noncollinear magnetic order in monolayer van der Waals (vdW) crystals is highly desirable for understanding the delicate magnetic interactions at reduced spatial constraints and miniaturized spintronic applications, but remains elusive in experiments. Here, we achieved spin-resolved imaging of helimagnetism at atomic scale in monolayer NiI2 crystals, that were grown on graphene-covered SiC(0001) substrate, using spin-polarized scanning tunneling microscopy. Our experiments identify the existence of a spin spiral state with canted plane in monolayer NiI2. The spin modulation Q vector of the spin spiral is determined as (0.2203, 0, 0), which is different from its bulk value or its in-plane projection, but agrees well with our first principles calculations. The spin spiral surprisingly indicates collective spin switching behavior under magnetic field, whose origin is ascribed to the incommensurability between the spin spiral and the crystal lattice. Our work unambiguously identifies the helimagnetic state in monolayer NiI2, paving the way for illuminating its expected type-II multiferroic order and developing spintronic devices based on vdW magnets.

cond-mat.mes-hall↗
arXiv2023

Intralayer Negative Poisson's Ratio in Two-Dimensional Black Arsenic by Strain Engineering

Negative Poisson's ratio as the anomalous characteristic generally exists in artificial architectures, such as re-entrant and honeycomb structures. The structures with negative Poisson's ratio have attracted intensive attention due to their unique auxetic effect and many promising applications in shear resistant and energy absorption fields. However, experimental observation of negative Poisson's ratio in natural materials barely happened, although various two-dimensional layered materials are predicted in theory. Herein, we report the anisotropic Raman response and the intrinsic intralayer negative Poisson's ratio of two-dimensional natural black arsenic (b-As) via strain engineering strategy. The results were evident by the detailed Raman spectrum of b-As under uniaxial strain together with density functional theory calculations. It is found that b-As was softer along the armchair than zigzag direction. The anisotropic mechanical features and van der Waals interactions play essential roles in strain-dependent Raman shifts and negative Poisson's ratio in the natural b-As along zigzag direction. This work may shed a light on the mechanical properties and potential applications of two-dimensional puckered materials.

cond-mat.mtrl-sci↗
arXiv2023

Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

Scene Graph Generation (SGG) aims to extract relationships in images for vision understanding. Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates. Existing re-balancing strategies try to handle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets. In this paper, we propose a Cross-modal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way. The proposed CaCao can be applied in a plug-and-play fashion and automatically strengthen existing SGG to tackle the long-tailed problem. Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner. Comprehensive experiments on three benchmark datasets show that CaCao consistently boosts the performance of multiple scene graph generation models in a model-agnostic way. Moreover, our Epic achieves competitive performance on open-world predicate prediction. The data and code for this paper are publicly available.

cs.CV↗
arXiv2023

Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models

Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models. Though effective, it is particularly problematic in the few-shot scenario, where prompt tuning performance is sensitive to the initialization and requires a time-consuming process to find a good initialization, thus restricting the fast adaptation ability of the pre-training models. In addition, prompt tuning could undermine the generalizability of the pre-training models, because the learnable prompt tokens are easy to overfit to the limited training samples. To address these issues, we introduce a novel Gradient-RegulAted Meta-prompt learning (GRAM) framework that jointly meta-learns an efficient soft prompt initialization for better adaptation and a lightweight gradient regulating function for strong cross-domain generalizability in a meta-learning paradigm using only the unlabeled image-text pre-training data. Rather than designing a specific prompt tuning method, our GRAM can be easily incorporated into various prompt tuning methods in a model-agnostic way, and comprehensive experiments show that GRAM brings about consistent improvement for them in several settings (i.e., few-shot learning, cross-domain generalization, cross-dataset generalization, etc.) over 11 datasets. Further, experiments show that GRAM enables the orthogonal methods of textual and visual prompt tuning to work in a mutually-enhanced way, offering better generalizability beyond the uni-modal prompt tuning methods.

cs.CV↗
arXiv2023

Online Distillation-enhanced Multi-modal Transformer for Sequential Recommendation

Multi-modal recommendation systems, which integrate diverse types of information, have gained widespread attention in recent years. However, compared to traditional collaborative filtering-based multi-modal recommendation systems, research on multi-modal sequential recommendation is still in its nascent stages. Unlike traditional sequential recommendation models that solely rely on item identifier (ID) information and focus on network structure design, multi-modal recommendation models need to emphasize item representation learning and the fusion of heterogeneous data sources. This paper investigates the impact of item representation learning on downstream recommendation tasks and examines the disparities in information fusion at different stages. Empirical experiments are conducted to demonstrate the need to design a framework suitable for collaborative learning and fusion of diverse information. Based on this, we propose a new model-agnostic framework for multi-modal sequential recommendation tasks, called Online Distillation-enhanced Multi-modal Transformer (ODMT), to enhance feature interaction and mutual learning among multi-source input (ID, text, and image), while avoiding conflicts among different features during training, thereby improving recommendation accuracy. To be specific, we first introduce an ID-aware Multi-modal Transformer module in the item representation learning stage to facilitate information interaction among different features. Secondly, we employ an online distillation training strategy in the prediction optimization stage to make multi-source data learn from each other and improve prediction robustness. Experimental results on a stream media recommendation dataset and three e-commerce recommendation datasets demonstrate the effectiveness of the proposed two modules, which is approximately 10% improvement in performance compared to baseline models.

cs.IR↗
arXiv2023

Designing optoelectronic properties by on-surface synthesis: formation and electronic structure of an iron-terpyridine macromolecular complex

Supramolecular chemistry protocols applied on surfaces offer compelling avenues for atomic scale control over organic-inorganic interface structures. In this approach, adsorbate-surface interactions and two-dimensional confinement can lead to morphologies and properties that differ dramatically from those achieved via conventional synthetic approaches. Here, we describe the bottom-up, on-surface synthesis of one-dimensional coordination nanostructures based on an iron (Fe)-terpyridine (tpy) interaction borrowed from functional metal-organic complexes used in photovoltaic and catalytic applications. Thermally activated diffusion of sequentially deposited ligands and metal atoms, and intra-ligand conformational changes, lead to Fe-tpy coordination and formation of these nanochains. Low-temperature Scanning Tunneling Microscopy and Density Functional Theory were used to elucidate the atomic-scale morphology of the system, providing evidence of a linear tri-Fe linkage between facing, coplanar tpy groups. Scanning Tunneling Spectroscopy reveals highest occupied orbitals with dominant contributions from states located at the Fe node, and ligand states that mostly contribute to the lowest unoccupied orbitals. This electronic structure yields potential for hosting photo-induced metal-to-ligand charge transfer in the visible/near-infrared. The formation of this unusual tpy/tri-Fe/tpy coordination motif has not been observed for wet chemistry synthesis methods, and is mediated by the bottom-up on-surface approach used here.

cond-mat.mtrl-sci↗
arXiv2023

In Defense of Clip-based Video Relation Detection

Video Visual Relation Detection (VidVRD) aims to detect visual relationship triplets in videos using spatial bounding boxes and temporal boundaries. Existing VidVRD methods can be broadly categorized into bottom-up and top-down paradigms, depending on their approach to classifying relations. Bottom-up methods follow a clip-based approach where they classify relations of short clip tubelet pairs and then merge them into long video relations. On the other hand, top-down methods directly classify long video tubelet pairs. While recent video-based methods utilizing video tubelets have shown promising results, we argue that the effective modeling of spatial and temporal context plays a more significant role than the choice between clip tubelets and video tubelets. This motivates us to revisit the clip-based paradigm and explore the key success factors in VidVRD. In this paper, we propose a Hierarchical Context Model (HCM) that enriches the object-based spatial context and relation-based temporal context based on clips. We demonstrate that using clip tubelets can achieve superior performance compared to most video-based methods. Additionally, using clip tubelets offers more flexibility in model designs and helps alleviate the limitations associated with video tubelets, such as the challenging long-term object tracking problem and the loss of temporal information in long-term tubelet feature compression. Extensive experiments conducted on two challenging VidVRD benchmarks validate that our HCM achieves a new state-of-the-art performance, highlighting the effectiveness of incorporating advanced spatial and temporal context modeling within the clip-based paradigm.

cs.CV↗
arXiv2023

Room-Temperature Ferromagnetism in Fe-doped SnSe Bulk Single Crystalline Semiconductor

The quest for pragmatic room-temperature (RT) magnetic semiconductors (MSs) with a suitable bandgap constitutes one of the contemporary opportunities to be exploited. This may provide a materials platform for to bring new-generation ideal information device technologies into real-world applications where the otherwise conventionally separately utilized charge and spin are simultaneously exploited. Here we present RT ferromagnetism in an Fe-doped SnSe (Fe:SnSe) van der Waals (vdW) single crystalline ferromagnetic semiconductor (FMS) with a semiconducting bandgap of ~1.19 eV (comparable to those of Si and GaAs). The synthesized Fe:SnSe single crystals feature a dilute Fe content of less than 1.0 at%, a Curie temperature of ~310 K, a layered vdW structure identical to that of pristine SnSe, and the absence of in-gap defect states. The Fe:SnSe vdW diluted magnetic semiconductor (DMS) single crystals are grown using a simple temperature-gradient melt-growth process, in which the magnetic Fe atom doping is realized uniquely using FeI2 as the dopant precursor whose melting point is low with respect to crystal growth, and which in principle possesses industrially unlimited scalability. Our work adds a new member in the family of long-searching RT magnetic semiconductors, and may establish a generalized strategy for large-volume production of related DMSs.

cond-mat.mtrl-sci↗