Search arXiv⌕ Search

arXiv subjects

Cheng Peng

Publications and source records attributed to Cheng Peng.

At least 145 records · Page 8Linked to original sources
arXiv2023

On the Impact of Cross-Domain Data on German Language Models

Traditionally, large language models have been either trained on general web crawls or domain-specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domain datasets. To examine the significance of prioritizing data diversity over quality, we present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data. Through training a series of models ranging between 122M and 750M parameters on both datasets, we conduct a comprehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained on the cross-domain dataset outperform those trained on quality data alone, leading to improvements up to $4.45\%$ over the previous state-of-the-art. The models are available at https://huggingface.co/ikim-uk-essen

cs.CL↗
arXiv2023

Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction

Objective To develop soft prompt-based learning algorithms for large language models (LLMs), examine the shape of prompts, prompt-tuning using frozen/unfrozen LLMs, transfer learning, and few-shot learning abilities. Methods We developed a soft prompt-based LLM model and compared 4 training strategies including (1) fine-tuning without prompts; (2) hard-prompt with unfrozen LLMs; (3) soft-prompt with unfrozen LLMs; and (4) soft-prompt with frozen LLMs. We evaluated 7 pretrained LLMs using the 4 training strategies for clinical concept and relation extraction on two benchmark datasets. We evaluated the transfer learning ability of the prompt-based learning algorithms in a cross-institution setting. We also assessed the few-shot learning ability. Results and Conclusion When LLMs are unfrozen, GatorTron-3.9B with soft prompting achieves the best strict F1-scores of 0.9118 and 0.8604 for concept extraction, outperforming the traditional fine-tuning and hard prompt-based models by 0.6~3.1% and 1.2~2.9%, respectively; GatorTron-345M with soft prompting achieves the best F1-scores of 0.8332 and 0.7488 for end-to-end relation extraction, outperforming the other two models by 0.2~2% and 0.6~11.7%, respectively. When LLMs are frozen, small (i.e., 345 million parameters) LLMs have a big gap to be competitive with unfrozen models; scaling LLMs up to billions of parameters makes frozen LLMs competitive with unfrozen LLMs. For cross-institute evaluation, soft prompting with a frozen GatorTron-8.9B model achieved the best performance. This study demonstrates that (1) machines can learn soft prompts better than humans, (2) frozen LLMs have better few-shot learning ability and transfer learning ability to facilitate muti-institution applications, and (3) frozen LLMs require large models.

cs.CL↗
arXiv2023

Project Aria: A New Tool for Egocentric Multi-Modal AI Research

Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. These future devices will need to be all-day wearable in a socially acceptable form-factor to support always available, context-aware and personalized AI applications. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area. In this paper, we describe the Aria device hardware including its sensor configuration and the corresponding software tools that enable recording and processing of such data.

cs.HC↗
arXiv2023

From Stoner to Local Moment Magnetism in Atomically Thin Cr2Te3

The field of two-dimensional (2D) ferromagnetism has been proliferating over the past few years, with ongoing interests in basic science and potential applications in spintronic technology. However, a high-resolution spectroscopic study of the 2D ferromagnet is still lacking due to the small size and air sensitivity of the exfoliated nanoflakes. Here, we report a thickness-dependent ferromagnetism in epitaxially grown Cr2Te3 thin films and investigate the evolution of the underlying electronic structure by synergistic angle-resolved photoemission spectroscopy, scanning tunneling microscopy, x-ray absorption spectroscopy, and first-principle calculations. A conspicuous ferromagnetic transition from Stoner to Heisenberg-type is directly observed in the atomically thin limit, indicating that dimensionality is a powerful tuning knob to manipulate the novel properties of 2D magnetism. Monolayer Cr2Te3 retains robust ferromagnetism, but with a suppressed Curie temperature, due to the drastic drop in the density of states near the Fermi level. Our results establish atomically thin Cr2Te3 as an excellent platform to explore the dual nature of localized and itinerant ferromagnetism in 2D magnets.

cond-mat.mtrl-sci↗
arXiv2023

PhaBOX: A web server for identifying and characterizing phage contigs in metagenomic data

Motivation: There is accumulating evidence showing the important roles of bacteriophages (phages) in regulating the structure and functions of the microbiome. However, lacking an easy-to-use and integrated phage analysis software hampers microbiome-related research from incorporating phages in the analysis. Results: In this work, we developed a web server, PhaBOX, which can comprehensively identify and analyze phage contigs in metagenomic data. It supports integrated phage analysis, including phage contig identification from the metagenomic assembly, lifestyle prediction, taxonomic classification, and host prediction. Instead of treating the algorithms as a black box, PhaBOX also supports visualization of the essential features for making predictions. The web server is designed with a user-friendly graphical interface that enables both informatics-trained and non-specialist users to analyze phages in microbiome data with ease. Availability: The web server of PhaBOX is available via: https://phage.ee.cityu.edu.hk. The source code of PhaBOX is available at: https://github.com/KennthShang/PhaBOX Contact: yannisun@cityu.edu.hk

q-bio.GN↗
arXiv2023

Divergence Based Quadrangle and Applications

This paper introduces a novel framework for assessing risk and decision-making in the presence of uncertainty, the \emph{$φ$-Divergence Quadrangle}. This approach expands upon the traditional Risk Quadrangle, a model that quantifies uncertainty through four key components: \emph{risk, deviation, regret}, and \emph{error}. The $φ$-Divergence Quadrangle incorporates the $φ$-divergence as a measure of the difference between probability distributions, thereby providing a more nuanced understanding of risk. Importantly, the $φ$-Divergence Quadrangle is closely connected with the distributionally robust optimization based on the $φ$-divergence approach through the duality theory of convex functionals. To illustrate its practicality and versatility, several examples of the $φ$-Divergence Quadrangle are provided, including the Quantile Quadrangle. The final portion of the paper outlines a case study implementing regression with the Entropic Value-at-Risk Quadrangle. The proposed $φ$-Divergence Quadrangle presents a refined methodology for understanding and managing risk, contributing to the ongoing development of risk assessment and management strategies.

q-fin.RM↗
arXiv2023

Machine learning enabled experimental design and parameter estimation for ultrafast spin dynamics

Advanced experimental measurements are crucial for driving theoretical developments and unveiling novel phenomena in condensed matter and material physics, which often suffer from the scarcity of facility resources and increasing complexities. To address the limitations, we introduce a methodology that combines machine learning with Bayesian optimal experimental design (BOED), exemplified with x-ray photon fluctuation spectroscopy (XPFS) measurements for spin fluctuations. Our method employs a neural network model for large-scale spin dynamics simulations for precise distribution and utility calculations in BOED. The capability of automatic differentiation from the neural network model is further leveraged for more robust and accurate parameter estimation. Our numerical benchmarks demonstrate the superior performance of our method in guiding XPFS experiments, predicting model parameters, and yielding more informative measurements within limited experimental time. Although focusing on XPFS and spin fluctuations, our method can be adapted to other experiments, facilitating more efficient data collection and accelerating scientific discoveries.

cond-mat.mtrl-sci↗
arXiv2023

A Study of Generative Large Language Model for Medical Research and Healthcare

There is enormous enthusiasm and concerns in using large language models (LLMs) in healthcare, yet current assumptions are all based on general-purpose LLMs such as ChatGPT. This study develops a clinical generative LLM, GatorTronGPT, using 277 billion words of mixed clinical and English text with a GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical natural language processing for medical research. Synthetic NLP models trained using GatorTronGPT generated text outperform NLP models trained using real-world clinical text. Physicians Turing test using 1 (worst) to 9 (best) scale shows that there is no significant difference in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights on the opportunities and challenges of LLMs for medical research and healthcare.

cs.CL↗
arXiv2023

On Ultrafast X-ray Methods for Magnetism

With the introduction of x-ray free electron laser sources around the world, new scientific approaches for visualizing matter at fundamental length and time-scales have become possible. As it relates to magnetism and "magnetic-type" systems, advanced methods are being developed for studying ultrafast magnetic responses on the time-scales at which they occur. We describe three capabilities which have the potential to seed new directions in this area and present original results from each: pump-probe x-ray scattering with low energy excitation, x-ray photon fluctuation spectroscopy, and ultrafast diffuse x-ray scattering. By combining these experimental techniques with advanced modeling together with machine learning, we describe how the combination of these domains allows for a new understanding in the field of magnetism. Finally, we give an outlook for future areas of investigation and the newly developed instruments which will take us there.

cond-mat.mtrl-sci↗
arXiv2023

Capturing dynamical correlations using implicit neural representations

The observation and description of collective excitations in solids is a fundamental issue when seeking to understand the physics of a many-body system. Analysis of these excitations is usually carried out by measuring the dynamical structure factor, S(Q, $ω$), with inelastic neutron or x-ray scattering techniques and comparing this against a calculated dynamical model. Here, we develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. We benchmark this approach on a Linear Spin Wave Theory (LSWT) simulator and advanced inelastic neutron scattering data from the square-lattice spin-1 antiferromagnet La$_2$NiO$_4$. We find that the model predicts the unknown parameters with excellent agreement relative to analytical fitting. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data, without the need for human-guided peak finding and fitting algorithms. This prototypical approach promises a new technology for this field to automatically detect and refine more advanced models for ordered quantum systems.

cond-mat.str-el↗
arXiv2023

Superconductivity in lightly doped Hubbard model on honeycomb lattice

We have performed large-scale density-matrix renormalization group studies of the lightly doped Hubbard model on the honeycomb lattice on long three and four-leg cylinders. We find that the ground state of the system upon lightly doping is consistent with that of a superconducting state with coexisting quasi-long-range superconducting and charge density wave orders. Both the superconducting and charge density wave correlations decay as a power law at long distances with corresponding exponents $K_{sc}<2$ and $K_c<2$. On the contrary, the spin-spin and single-particle correlations decay exponentially, although with relatively long correlation lengths.

cond-mat.str-el↗
arXiv2023

Factor Model of Mixtures

This paper proposes a new approach to estimating the distribution of a response variable conditioned on observing some factors. The proposed approach possesses desirable properties of flexibility, interpretability, tractability and extendability. The conditional quantile function is modeled by a mixture (weighted sum) of basis quantile functions, with the weights depending on factors. The calibration problem is formulated as a convex optimization problem. It can be viewed as conducting quantile regressions for all confidence levels simultaneously while avoiding quantile crossing by definition. The calibration problem is equivalent to minimizing the continuous ranked probability score (CRPS). Based on the canonical polyadic (CP) decomposition of tensors, we propose a dimensionality reduction method that reduces the rank of the parameter tensor and propose an alternating algorithm for estimation. Additionally, based on Risk Quadrangle framework, we generalize the approach to conditional distributions defined by Conditional Value-at-Risk (CVaR), expectile and other functions of uncertainty measures. Although this paper focuses on using splines as the weight functions, it can be extended to neural networks. Numerical experiments demonstrate the effectiveness of our approach.

stat.ME↗
arXiv2023

Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension

Objective: To develop a natural language processing system that solves both clinical concept extraction and relation extraction in a unified prompt-based machine reading comprehension (MRC) architecture with good generalizability for cross-institution applications. Methods: We formulate both clinical concept extraction and relation extraction using a unified prompt-based MRC architecture and explore state-of-the-art transformer models. We compare our MRC models with existing deep learning models for concept extraction and end-to-end relation extraction using two benchmark datasets developed by the 2018 National NLP Clinical Challenges (n2c2) challenge (medications and adverse drug events) and the 2022 n2c2 challenge (relations of social determinants of health [SDoH]). We also evaluate the transfer learning ability of the proposed MRC models in a cross-institution setting. We perform error analyses and examine how different prompting strategies affect the performance of MRC models. Results and Conclusion: The proposed MRC models achieve state-of-the-art performance for clinical concept and relation extraction on the two benchmark datasets, outperforming previous non-MRC transformer models. GatorTron-MRC achieves the best strict and lenient F1-scores for concept extraction, outperforming previous deep learning models on the two datasets by 1%~3% and 0.7%~1.3%, respectively. For end-to-end relation extraction, GatorTron-MRC and BERT-MIMIC-MRC achieve the best F1-scores, outperforming previous deep learning models by 0.9%~2.4% and 10%-11%, respectively. For cross-institution evaluation, GatorTron-MRC outperforms traditional GatorTron by 6.4% and 16% for the two datasets, respectively. The proposed method is better at handling nested/overlapped concepts, extracting relations, and has good portability for cross-institute applications.

cs.CL↗
arXiv2023

Portfolio Optimization on Multivariate Regime Switching GARCH Model with Normal Tempered Stable Innovation

This paper uses simulation-based portfolio optimization to mitigate the left tail risk of the portfolio. The contribution is twofold. (i) We propose the Markov regime-switching GARCH model with multivariate normal tempered stable innovation (MRS-MNTS-GARCH) to accommodate fat tails, volatility clustering and regime switch. The volatility of each asset independently follows the regime-switch GARCH model, while the correlation of joint innovation of the GARCH models follows the Hidden Markov Model. (ii) We use tail risk measures, namely conditional value-at-risk (CVaR) and conditional drawdown-at-risk (CDaR), in the portfolio optimization. The optimization is performed with the sample paths simulated by the MRS-MNTS-GARCH model. We conduct an empirical study on the performance of optimal portfolios. Out-of-sample tests show that the optimal portfolios with tail measures outperform the optimal portfolio with standard deviation measure and the equally weighted portfolio in various performance measures. The out-of-sample performance of the optimal portfolios is also more robust to suboptimality on the efficient frontier.

q-fin.RM↗
arXiv2023

PhaVIP: Phage VIrion Protein classification based on chaos game representation and Vision Transformer

Motivation: As viruses that mainly infect bacteria, phages are key players across a wide range of ecosystems. Analyzing phage proteins is indispensable for understanding phages' functions and roles in microbiomes. High-throughput sequencing enables us to obtain phages in different microbiomes with low cost. However, compared to the fast accumulation of newly identified phages, phage protein classification remains difficult. In particular, a fundamental need is to annotate virion proteins, the structural proteins such as major tail, baseplate etc. Although there are experimental methods for virion protein identification, they are too expensive or time-consuming, leaving a large number of proteins unclassified. Thus, there is a great demand to develop a computational method for fast and accurate phage virion protein classification. Results: In this work, we adapted the state-of-the-art image classification model, Vision Transformer, to conduct virion protein classification. By encoding protein sequences into unique images using chaos gaming representation, we can leverage Vision Transformer to learn both local and global features from sequence ``images''. Our method, PhaVIP, has two main functions: classifying PVP and non-PVP sequences and annotating the types of PVP, such as capsid and tail. We tested PhaVIP on several datasets with increasing difficulty and benchmarked it against alternative tools. The experimental results show that PhaVIP has superior performance. After validating the performance of PhaVIP, we investigated two applications that can use the output of PhaVIP: phage taxonomy classification and phage host prediction. The results show the benefit of using classified proteins rather than all proteins.

q-bio.GN↗
arXiv2022

A fugacity-based Lattice Boltzmann method for multicomponent multiphase systems

The free energy model can extend the Lattice Boltzmann method to multiphase systems. However, there is a lack of models capable of simulating multicomponent multiphase fluids with partial miscibility. In addition, existing models cannot be generalized to honor thermodynamic information provided by any multicomponent equation of state of choice. In this paper, we introduce a free energy Lattice Boltzmann model where the forcing term is determined by the fugacity of the species, the thermodynamic property that connects species partial pressure to chemical potential calculations. By doing so, we are able to carry out multicomponent multiphase simulations of partially miscible fluids and generalize the methodology for use with any multicomponent equation of state of interest. We test this fugacity-based Lattice Boltzmann method for the cases of vapor-liquid equilibrium for two and three-component mixtures in various temperature and pressure conditions. We demonstrate that the model is able to reliably reproduce phase densities and compositions as predicted by multicomponent thermodynamics and can reproduce different characteristic pressure-composition and temperature-composition envelopes with a high degree of accuracy. We also demonstrate that the model can offer accurate predictions under dynamic conditions.

physics.flu-dyn↗
arXiv2022

Bridging quantum many-body scar and quantum integrability in Ising chains with transverse and longitudinal fields

Quantum many-body scar (QMBS) and quantum integrability(QI) have been recognized as two distinct mechanisms for the breakdown of eigenstate thermalization hypothesis(ETH) in an isolated system. In this work, we reveal a smooth route to connect these two ETH-breaking mechanisms in the Ising chain with transverse and longitudinal fields. Specifically, starting from an initial Ising anti-ferromagnetic state, we find that the dynamical system undergoes a smooth non-thermal crossover from QMBS to QI by changing the Ising coupling($J$) and longitudinal field($h$) simultaneously while keeping their ratio fixed, which corresponds to the Rydberg Hamiltonian with an arbitrary nearest-neighbor repulsion. Deviating from this ratio, we further identify a continuous thermalization trajectory in ($h,J$) plane that is exactly given by the Ising transition line, signifying an intimate relation between thermalization and quantum critical point. Finally, we map out a completely different dynamical phase diagram starting from an initial ferromagnetic state, where the thermalization is shown to be equally facilitated by the resonant spin-flip at special ratios of $J$ and $h$. By bridging QMBS and QI in Ising chains, our results demonstrate the breakdown of ETH in much broader physical settings, which also suggest an alternative way to characterize quantum phase transition via thermalization in non-equilibrium dynamics.

cond-mat.quant-gas↗
arXiv2022

A Geometric-Relational Deep Learning Framework for BIM Object Classification

Interoperability issue is a significant problem in Building Information Modeling (BIM). Object type, as a kind of critical semantic information needed in multiple BIM applications like scan-to-BIM and code compliance checking, also suffers when exchanging BIM data or creating models using software of other domains. It can be supplemented using deep learning. Current deep learning methods mainly learn from the shape information of BIM objects for classification, leaving relational information inherent in the BIM context unused. To address this issue, we introduce a two-branch geometric-relational deep learning framework. It boosts previous geometric classification methods with relational information. We also present a BIM object dataset IFCNet++, which contains both geometric and relational information about the objects. Experiments show that our framework can be flexibly adapted to different geometric methods. And relational features do act as a bonus to general geometric learning methods, obviously improving their classification performance, thus reducing the manual labor of checking models and improving the practical value of enriched BIM models.

cs.CV↗