arXiv subjects

Zhenyu Zhang

Publications and source records attributed to Zhenyu Zhang.

At least 217 records · Page 12Linked to original sources

arXiv2023

Polarization multistates in antiferroelectric van der Waals materials

The bistability of charge polarization in ferroelectric materials has long been the basis of ferroelectric devices. However, the ferroelectricity tends to be vanishing as the thickness of materials is reduced to a few nanometers or thinner due to the depolarization field. Instead, they show a paraelectric or an antiferroelectric ordering in the ultra-thin limit, which is unfavorable for their applications in devices. Here we uncover polarization multistates in thin films of van der Waals materials, in which the individual monolayers have an antiferroelectric ordering with out-of-plane polarizations. This property results from a unique combination of the polarization and layer degrees of freedom. Using first-principles calculations, we demonstrate that bilayers and trilayers of the CuInP$_2$S$_6$ family possess quintuple and septuple polarization states., respectively. Our climbing image nudged elastic band calculations for the bilayers and trilayers of CuInP$_2$S$_6$ and CuCrP$_2$S$_6$ further show that the states can be transformed into each other under appropriate external electric fields, for which a unique layer-selective half-layer-by-half-layer flipping mechanism governs the transformings. Our study opens up a door to design unusual polarization states using intrinsic degrees of freedom of layered antiferroelectrics for the next-generation ferroelectric devices that go beyond the bistability paradigm.

cond-mat.mtrl-sci↗

arXiv2023

H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Large Language Models (LLMs), despite their recent impressive accomplishments, are notably cost-prohibitive to deploy, particularly for applications involving long-content generation, such as dialogue systems and story writing. Often, a large amount of transient state information, referred to as the KV cache, is stored in GPU memory in addition to model parameters, scaling linearly with the sequence length and batch size. In this paper, we introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the noteworthy observation that a small portion of tokens contributes most of the value when computing attention scores. We call these tokens Heavy Hitters (H$_2$). Through a comprehensive investigation, we find that (i) the emergence of H$_2$ is natural and strongly correlates with the frequent co-occurrence of tokens in the text, and (ii) removing them results in significant performance degradation. Based on these insights, we propose Heavy Hitter Oracle (H$_2$O), a KV cache eviction policy that dynamically retains a balance of recent and H$_2$ tokens. We formulate the KV cache eviction as a dynamic submodular problem and prove (under mild assumptions) a theoretical guarantee for our novel eviction algorithm which could help guide future work. We validate the accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of tasks. Our implementation of H$_2$O with 20% heavy hitters improves the throughput over three leading inference systems DeepSpeed Zero-Inference, Hugging Face Accelerate, and FlexGen by up to 29$\times$, 29$\times$, and 3$\times$ on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the latency by up to 1.9$\times$. The code is available at https://github.com/FMInference/H2O.

cs.LG↗

arXiv2023

A study on the impact of pre-trained model on Just-In-Time defect prediction

Previous researchers conducting Just-In-Time (JIT) defect prediction tasks have primarily focused on the performance of individual pre-trained models, without exploring the relationship between different pre-trained models as backbones. In this study, we build six models: RoBERTaJIT, CodeBERTJIT, BARTJIT, PLBARTJIT, GPT2JIT, and CodeGPTJIT, each with a distinct pre-trained model as its backbone. We systematically explore the differences and connections between these models. Specifically, we investigate the performance of the models when using Commit code and Commit message as inputs, as well as the relationship between training efficiency and model distribution among these six models. Additionally, we conduct an ablation experiment to explore the sensitivity of each model to inputs. Furthermore, we investigate how the models perform in zero-shot and few-shot scenarios. Our findings indicate that each model based on different backbones shows improvements, and when the backbone's pre-training model is similar, the training resources that need to be consumed are much more closer. We also observe that Commit code plays a significant role in defect detection, and different pre-trained models demonstrate better defect detection ability with a balanced dataset under few-shot scenarios. These results provide new insights for optimizing JIT defect prediction tasks using pre-trained models and highlight the factors that require more attention when constructing such models. Additionally, CodeGPTJIT and GPT2JIT achieved better performance than DeepJIT and CC2Vec on the two datasets respectively under 2000 training samples. These findings emphasize the effectiveness of transformer-based pre-trained models in JIT defect prediction tasks, especially in scenarios with limited training data.

cs.SE↗

arXiv2023

Simple but Effective Unsupervised Classification for Specified Domain Images: A Case Study on Fungi Images

High-quality labeled datasets are essential for deep learning. Traditional manual annotation methods are not only costly and inefficient but also pose challenges in specialized domains where expert knowledge is needed. Self-supervised methods, despite leveraging unlabeled data for feature extraction, still require hundreds or thousands of labeled instances to guide the model for effective specialized image classification. Current unsupervised learning methods offer automatic classification without prior annotation but often compromise on accuracy. As a result, efficiently procuring high-quality labeled datasets remains a pressing challenge for specialized domain images devoid of annotated data. Addressing this, an unsupervised classification method with three key ideas is introduced: 1) dual-step feature dimensionality reduction using a pre-trained model and manifold learning, 2) a voting mechanism from multiple clustering algorithms, and 3) post-hoc instead of prior manual annotation. This approach outperforms supervised methods in classification accuracy, as demonstrated with fungal image data, achieving 94.1% and 96.7% on public and private datasets respectively. The proposed unsupervised classification method reduces dependency on pre-annotated datasets, enabling a closed-loop for data classification. The simplicity and ease of use of this method will also bring convenience to researchers in various fields in building datasets, promoting AI applications for images in specialized domains.

cs.CV↗

arXiv2023

Polarized images of charged particles in vortical motions around a magnetized Kerr black hole

In this work, we study the images of a Kerr black hole (BH) immersed in uniform magnetic fields, illuminated by the synchrotron radiation of charged particles in the jet. We particularly focus on the spontaneously vortical motions (SVMs) of charged particles in the jet region and investigate the polarized images of electromagnetic radiations from the trajectories along SVMs. We notice that there is a critical value $ω_c$ for charged particle released at a given initial position and subjected an outward force, and once $|qB_0/m|=|ω_B|>|ω_c|$ charged particles can move along SVMs in the jet region. We obtain the polarized images of the electromagnetic radiations from the trajectories along SVMs. Our simplified model suggests that the SVM radiations can act as the light source to illuminate the BH and form a photon ring structure.

gr-qc↗

arXiv2023

Metallic Quantized Anomalous Hall Effect without Chiral Edge States

The quantum anomalous Hall effect (QAHE) is a topological state of matter with a quantized Hall resistance. It has been observed in some two-dimensional insulating materials such as magnetic topological insulator films and twisted bilayer graphene. These materials are insulating in the bulk, but possess chiral edge states carrying the edge current around the systems. Here we discover a metallic QAHE in a topological insulator film with magnetic sandwich heterostructure, in which the Hall conductance is quantized to $e^{2}/h$, but the longitudinal conductance remains finite. This effect is attributed to the existence of a pair of massless Dirac cones of surface fermions, with each contributing half of the Hall conductance due to quantum anomaly. It is not characterized by a Chern number and not associated to any chiral edge states. Our study offers novel insights into topological transport phenomena and topological metallic states of matter.

cond-mat.mes-hall↗

arXiv2023

Quasiparticle and Transport Properties of Disordered Bilayer Graphene

In recent experimental and theoretical studies of graphene, disorder scattering processes have been suggested to play an important role in its electronic and transport properties. In the preceding paper, it has been shown that the nonperturbative momentum-space Lanczos method is able to accurately describe all the multiple impurity scattering events and account for the quasiparticle and transport properties of disordered monolayer graphene. In the present study, we expand the range of applicability of this recursive method by numerically investigating the quasiparticle and transport properties of Bernal-stacked bilayer graphene in the presence of scalar Anderson disorder. The results are further compared with the findings of the same system using a self-consistent Born approximation, as well as the central findings in the preceding paper for monolayer graphene. It is found that in both systems, proper inclusions of all the scattering events are needed in order to reliably capture the role of disorder via multiple impurity scattering. In particular, the quasiparticle residue is shown to decrease sharply near the charge neutrality point, suggesting that the system is either a marginal Fermi liquid or a non-Fermi liquid. Furthermore, we reveal the dependences of the transport properties of disordered bilayer graphene on the carrier density and temperature, and explore the role of interlayer scattering at varying strengths. Our findings help to provide some new angles into the quasiparticle and transport properties of disordered bilayer graphene.

cond-mat.dis-nn↗

arXiv2023

Disorder Effects on the Quasiparticle and Transport Properties of Two-Dimensional Dirac Fermionic Systems

Despite extensive existing studies, a complete understanding of the role of disorder in affecting the physical properties of two-dimensional Dirac fermionic systems remains a standing challenge, largely due to obstacles encountered in treating multiple scattering events for such inherently strong scattering systems. Using graphene as an example and a nonperturbative numerical technique, here we reveal that the low energy quasiparticle properties are considerably modified by multiple scattering processes even in the presence of weak scalar potentials. We extract unified power-law energy dependences of the self-energy with fractional exponents from the weak scattering limit to the strong scattering limit from our numerical analysis, leading to sharp reductions of the quasiparticle residues near the Dirac point, eventually vanishing at the Dirac point. The central findings stay valid when the Anderson-type impurities are replaced by correlated Gaussian- or Yukawa-type disorder with varying correlation lengths. The improved understanding gained here also enables us to provide better interpretations of the experimental observations surrounding the temperature and carrier density dependences of the conductivity in ultra-high mobility graphene samples. The approach demonstrated here is expected to find broad applicability in understanding the role of various other types of impurities in two-dimensional Dirac systems.

cond-mat.mes-hall↗

arXiv2023

Electromagnetic effects on charged particles in NHEK

We investigate the motions of charged particles in the near horizon region of an extreme Kerr black hole with weak electromagnetic fields. There is an enhanced symmetry in the NHEK geometry. We find that when the electromagnetic field respects this enhanced symmetry, which we refer to as the maximally symmetric electromagnetic (MSEM) field, the equations of motion of charged particles get simplified into a set of decoupled first-order differential equations. We discuss the motions of charged particles in two MSEM fields, one being the force-free field and the other being the vacuum fields. Even though the radial motions are similar to the geodesics in NHEK geometry, the angular motions could be affected by the electromagnetic field significantly. In particular, for the vacuum solution which is produced by a weakly charged black hole, there exist stable vortical motions if the electromagnetic parameter is above the critical value $\mB_c = \sqrt{3}$. These vortical motions do not cross the equatorial planes, and the charged particles in them radiate non-thermally. We discuss the corresponding astrophysical implications.

gr-qc↗

arXiv2023

Are Large Kernels Better Teachers than Transformers for ConvNets?

This paper reveals a new appeal of the recently emerged large-kernel Convolutional Neural Networks (ConvNets): as the teacher in Knowledge Distillation (KD) for small-kernel ConvNets. While Transformers have led state-of-the-art (SOTA) performance in various fields with ever-larger models and labeled data, small-kernel ConvNets are considered more suitable for resource-limited applications due to the efficient convolution operation and compact weight sharing. KD is widely used to boost the performance of small-kernel ConvNets. However, previous research shows that it is not quite effective to distill knowledge (e.g., global information) from Transformers to small-kernel ConvNets, presumably due to their disparate architectures. We hereby carry out a first-of-its-kind study unveiling that modern large-kernel ConvNets, a compelling competitor to Vision Transformers, are remarkably more effective teachers for small-kernel ConvNets, due to more similar architectures. Our findings are backed up by extensive experiments on both logit-level and feature-level KD ``out of the box", with no dedicated architectural nor training recipe modifications. Notably, we obtain the \textbf{best-ever pure ConvNet} under 30M parameters with \textbf{83.1\%} top-1 accuracy on ImageNet, outperforming current SOTA methods including ConvNeXt V2 and Swin V2. We also find that beneficial characteristics of large-kernel ConvNets, e.g., larger effective receptive fields, can be seamlessly transferred to students through this large-to-small kernel distillation. Code is available at: \url{https://github.com/VITA-Group/SLaK}.

cs.CV↗

arXiv2023

Hybrid Transformer and CNN Attention Network for Stereo Image Super-resolution

Multi-stage strategies are frequently employed in image restoration tasks. While transformer-based methods have exhibited high efficiency in single-image super-resolution tasks, they have not yet shown significant advantages over CNN-based methods in stereo super-resolution tasks. This can be attributed to two key factors: first, current single-image super-resolution transformers are unable to leverage the complementary stereo information during the process; second, the performance of transformers is typically reliant on sufficient data, which is absent in common stereo-image super-resolution algorithms. To address these issues, we propose a Hybrid Transformer and CNN Attention Network (HTCAN), which utilizes a transformer-based network for single-image enhancement and a CNN-based network for stereo information fusion. Furthermore, we employ a multi-patch training strategy and larger window sizes to activate more input pixels for super-resolution. We also revisit other advanced techniques, such as data augmentation, data ensemble, and model ensemble to reduce overfitting and data bias. Finally, our approach achieved a score of 23.90dB and emerged as the winner in Track 1 of the NTIRE 2023 Stereo Image Super-Resolution Challenge.

cs.CV↗

arXiv2023

OPDN: Omnidirectional Position-aware Deformable Network for Omnidirectional Image Super-Resolution

360° omnidirectional images have gained research attention due to their immersive and interactive experience, particularly in AR/VR applications. However, they suffer from lower angular resolution due to being captured by fisheye lenses with the same sensor size for capturing planar images. To solve the above issues, we propose a two-stage framework for 360° omnidirectional image superresolution. The first stage employs two branches: model A, which incorporates omnidirectional position-aware deformable blocks (OPDB) and Fourier upsampling, and model B, which adds a spatial frequency fusion module (SFF) to model A. Model A aims to enhance the feature extraction ability of 360° image positional information, while Model B further focuses on the high-frequency information of 360° images. The second stage performs same-resolution enhancement based on the structure of model A with a pixel unshuffle operation. In addition, we collected data from YouTube to improve the fitting ability of the transformer, and created pseudo low-resolution images using a degradation network. Our proposed method achieves superior performance and wins the NTIRE 2023 challenge of 360° omnidirectional image super-resolution.

eess.IV↗

arXiv2023

Revealing the nature of hidden charm pentaquarks with machine learning

We study the nature of the hidden charm pentaquarks, i.e. the $P_c(4312)$, $P_c(4440)$ and $P_c(4457)$, with a neural network approach in pionless effective field theory. In this framework, the normal $χ^2$ fitting approach cannot distinguish the quantum numbers of the $P_c(4440)$ and $P_c(4457)$. In contrast to that, the neural network-based approach can discriminate them. In addition, we also illustrate the role of each experimental data bin of the invariant $J/ψp$ mass distribution on the underlying physics in both neural network and fitting methods. Their similarities and differences demonstrate that neural network methods can use data information more effectively and directly. This study provides more insights about how the neural network-based approach predicts the nature of exotic states from the mass spectrum.

hep-ph↗

arXiv2023

The third peak structure in the double $J/ψ$ spectrum

Recently, the CMS and ATLAS collaborations have reported their $J/ψJ/ψ$ and $J/ψψ^\prime$ invariant mass distributions, respectively, for searching for fully charmed tetraquarks. Both of them reported the existence of a peak structure around $7.2~\gev$. In this article, we exhibit the role of the $η_c(2S)η_c(2S)$ channel, which is close to this peak position, by studying the $J/ψJ/ψ$ and $J/ψψ^\prime$ invariant mass distributions for the potential quantum numbers $J^{PC}=0^{++}$ or $2^{++}$ by considering both the coherence and incoherence with the background contribution. All these frameworks can describe the experimental data very well, however with different pole structures. For instance, in the case of $2^{++}$ description of these structures, there always exists a pole slightly below the $J/ψJ/ψ$ threshold. For the $0^{++}$ case, a similar pole can also be found below the $J/ψJ/ψ$ threshold however with a much lower mass. More importantly, the number of the poles is found to be case-dependent. For the $0^{++}$ ($2^{++}$) case, the peak structure around $7.2~\gev$ can (cannot) be produced due to the presence (absence) of the $η_c(2S)η_c(2S)$ channel. Although the pole positions are case-dependent, the relation between the peak structure in the $J/ψJ/ψ$ invariant mass distribution and the dip structure in the $J/ψψ^\prime$ invariant mass distribution around $7.2~\gev$ is unambiguous. We suggest experimentalists to detailed scan both the $J/ψJ/ψ$ and the $J/ψψ^\prime$ invariant mass distributions, especially around $7.2~\gev$ to probe the nature of the third fully charmed state.

hep-ph↗

arXiv2023

ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts

Recent progress in diffusion models has revolutionized the popular technology of text-to-image generation. While existing approaches could produce photorealistic high-resolution images with text conditions, there are still several open problems to be solved, which limits the further improvement of image fidelity and text relevancy. In this paper, we propose ERNIE-ViLG 2.0, a large-scale Chinese text-to-image diffusion model, to progressively upgrade the quality of generated images by: (1) incorporating fine-grained textual and visual knowledge of key elements in the scene, and (2) utilizing different denoising experts at different denoising stages. With the proposed mechanisms, ERNIE-ViLG 2.0 not only achieves a new state-of-the-art on MS-COCO with zero-shot FID score of 6.75, but also significantly outperforms recent models in terms of image fidelity and image-text alignment, with side-by-side human evaluation on the bilingual prompt set ViLG-300.

cs.CV↗

arXiv2023

Learning Versatile 3D Shape Generation with Improved AR Models

Auto-Regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space. While this approach has been extended to the 3D domain for powerful shape generation, it still has two limitations: expensive computations on volumetric grids and ambiguous auto-regressive order along grid dimensions. To overcome these limitations, we propose the Improved Auto-regressive Model (ImAM) for 3D shape generation, which applies discrete representation learning based on a latent vector instead of volumetric grids. Our approach not only reduces computational costs but also preserves essential geometric details by learning the joint distribution in a more tractable order. Moreover, thanks to the simplicity of our model architecture, we can naturally extend it from unconditional to conditional generation by concatenating various conditioning inputs, such as point clouds, categories, images, and texts. Extensive experiments demonstrate that ImAM can synthesize diverse and faithful shapes of multiple categories, achieving state-of-the-art performance.

cs.CV↗

arXiv2023

Phonon-Mediated ${\bf S}$-Wave Superconductivity in the Kagome Metal CsV$_3$Sb$_5$ under Pressure

The nature of the superconducting pairing state in the pristine phase of a compressed kagome metal CsV$_3$Sb$_5$ under pressure is studied by the Migdal-Eliashberg formalism and density-functional theory calculations. We find that the superconducting gap distribution driven by electron-phonon coupling is nodeless and anisotropic. It is revealed that the hybridized V 3$d$ and Sb 5$p$ orbitals are strongly coupled to the V-V bond-stretching and V-Sb bond-bending phonon modes, giving rise to a wide spread of superconducting gap depending on its associated Fermi-surface sheets and momentum. Specifically, the superconducting gaps associated with V 3$d_{xy,x^2-y^2,z^2}$ and 3$d_{xz,yz}$ orbitals are larger in their average magnitude and more widely spread compared to that associated with the Sb 5$p_z$ orbital. Our findings demonstrate that the superconductivity of compressed CsV$_3$Sb$_5$ can be explained by the anisotropic multiband pairing mechanism with conventional phonon-mediated $s$-wave symmetry, evidenced by recent experimental observations under pressure as well as at ambient pressure.

cond-mat.supr-con↗

arXiv2023

Graph Transformer GANs for Graph-Constrained House Generation

We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for the challenging graph-constrained house generation task. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and self-attentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. Finally, we propose a novel graph-based cycle-consistency loss that aims at maintaining the relative spatial relationships between ground truth and predicted graphs. Experiments on two challenging graph-constrained house generation tasks (i.e., house layout and roof generation) with two public datasets demonstrate the effectiveness of GTGAN in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on both tasks.

cs.CV↗