arXiv subjects

Fei Huang

Publications and source records attributed to Fei Huang.

At least 325 records · Page 18Linked to original sources

arXiv2024

DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever

Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.

cs.CL↗

arXiv2024

LORE++: Logical Location Regression Network for Table Structure Recognition with Pre-training

Table structure recognition (TSR) aims at extracting tables in images into machine-understandable formats. Recent methods solve this problem by predicting the adjacency relations of detected cell boxes or learning to directly generate the corresponding markup sequences from the table images. However, existing approaches either count on additional heuristic rules to recover the table structures, or face challenges in capturing long-range dependencies within tables, resulting in increased complexity. In this paper, we propose an alternative paradigm. We model TSR as a logical location regression problem and propose a new TSR framework called LORE, standing for LOgical location REgression network, which for the first time regresses logical location as well as spatial location of table cells in a unified network. Our proposed LORE is conceptually simpler, easier to train, and more accurate than other paradigms of TSR. Moreover, inspired by the persuasive success of pre-trained models on a number of computer vision and natural language processing tasks, we propose two pre-training tasks to enrich the spatial and logical representations at the feature level of LORE, resulting in an upgraded version called LORE++. The incorporation of pre-training in LORE++ has proven to enjoy significant advantages, leading to a substantial enhancement in terms of accuracy, generalization, and few-shot capability compared to its predecessor. Experiments on standard benchmarks against methods of previous paradigms demonstrate the superiority of LORE++, which highlights the potential and promising prospect of the logical location regression paradigm for TSR.

cs.CV↗

arXiv2024

Unifying Structured Data as Graph for Data-to-Text Pre-Training

Data-to-text (D2T) generation aims to transform structured data into natural language text. Data-to-text pre-training has proved to be powerful in enhancing D2T generation and yields impressive performances. However, previous pre-training methods either oversimplified structured data into a sequence without considering input structures or designed training objectives tailored for a specific data structure (e.g., table or knowledge graph). In this paper, we unify different types of structured data (i.e., table, key-value data, knowledge graph) into the graph format and cast different data-to-text generation tasks as graph-to-text generation. To effectively exploit the structural information of the input graph, we propose a structure-enhanced pre-training method for D2T generation by designing a structure-enhanced Transformer. Concretely, we devise a position matrix for the Transformer, encoding relative positional information of connected nodes in the input graph. In addition, we propose a new attention matrix to incorporate graph structures into the original Transformer by taking the available explicit connectivity structure into account. Extensive experiments on six benchmark datasets show the effectiveness of our model. Our source codes are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/unid2t.

cs.CL↗

arXiv2023

A Survey on Out-of-Distribution Detection in NLP

Out-of-distribution (OOD) detection is essential for the reliable and safe deployment of machine learning systems in the real world. Great progress has been made over the past years. This paper presents the first review of recent advances in OOD detection with a particular focus on natural language processing approaches. First, we provide a formal definition of OOD detection and discuss several related fields. We then categorize recent algorithms into three classes according to the data they used: (1) OOD data available, (2) OOD data unavailable + in-distribution (ID) label available, and (3) OOD data unavailable + ID label unavailable. Third, we introduce datasets, applications, and metrics. Finally, we summarize existing work and present potential future research topics.

cs.CL↗

arXiv2023

EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data

Large Language Models (LLMs) pre-trained on massive corpora have exhibited remarkable performance on various NLP tasks. However, applying these models to specific domains still poses significant challenges, such as lack of domain knowledge, limited capacity to leverage domain knowledge and inadequate adaptation to domain-specific data formats. Considering the exorbitant cost of training LLMs from scratch and the scarcity of annotated data within particular domains, in this work, we focus on domain-specific continual pre-training of LLMs using E-commerce domain as an exemplar. Specifically, we explore the impact of continual pre-training on LLMs employing unlabeled general and E-commercial corpora. Furthermore, we design a mixing strategy among different data sources to better leverage E-commercial semi-structured data. We construct multiple tasks to assess LLMs' few-shot In-context Learning ability and their zero-shot performance after instruction tuning in E-commerce domain. Experimental results demonstrate the effectiveness of continual pre-training of E-commerce LLMs and the efficacy of our devised data mixing strategy.

cs.CL↗

arXiv2023

Transforming Visual Scene Graphs to Image Captions

We propose to Transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a homogeneous encoder-decoder unlike the previous heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TSG. The code is in: https://github.com/GaryJiajia/TSG.

cs.CV↗

arXiv2023

Analysis of the data for $γp \to f_1(1285) p$ photoproduction

The photoproduction of $f_1(1285)$ meson off proton is investigated within an effective Lagrangian approach. The $t$-channel $ρ$- and $ω$-exchange diagrams, $u$-channel nucleon-exchange diagram, generalized contact term, and $s$-channel pole diagrams of nucleon and a minimal number of nucleon resonances are taken into account in constructing the reaction amplitudes to describe the experimental data. Three different models, i.e., the Feynman model, the Regge model, and the interpolated Regge model, are employed where the $t$-channel reaction amplitudes are constructed in Feynman type, Regge type, and interpolated Regge type, respectively. The results show that in neither Feynman model with two nucleon resonances introduced nor interpolated Regge model with one nucleon resonances included, can the available data for $γp \to f_1(1285) p$ be satisfactorily reproduced. Nevertheless, in the Regge model, when any one of the $N(1990){7/2}^+$, $N(2000){5/2}^+$, $N(2040){3/2}^+$, $N(2060){5/2}^-$, $N(2100){1/2}^+$, $N(2120){3/2}^-$, $N(2190){7/2}^-$, $N(2300){1/2}^+$, and $N(2570){5/2}^-$ resonances is considered, the data can be well described. The resulted resonance parameters are consistent with those advocated in Particle Data Group (PDG) review. Further analysis shows that in high-energy region, the peaks of $γp \to f_1(1285) p$ differential cross sections at forward angles are dominated by the contributions from $t$-channel $ρ$- and $ω$-exchange diagrams, while in low-energy region, the $s$-channel pole diagrams of resonances also provide significant contributions to the $γp \to f_1(1285) p$ cross sections.

hep-ph↗

arXiv2023

Short rainbow cycles in edge-colored graphs

A famous conjecture of Caccetta and Häggkvist (CHC) states that a directed graph $D$ with $n$ vertices and minimum outdegree at least $r$ has a directed cycle of length at most $\lceil \frac{n}{r}\rceil$. In 2017, Aharoni proposed the following generalization: an edge-colored graph $G$ with $n$ vertices, $n$ color classes of size at least $r$ has a rainbow cycle of length at most $\lceil \frac{n}{r}\rceil$. Since CHC can be seen as the case of Aharoni's Conjecture: color classes in the color partition are monochromatic stars centered at distinct vertices, one way to study Aharoni's Conjecture is to structure the color classes as each color class is either a star, a triangle or contains a matching of size 2. Guo improved the upper bound in Aharoni's Conjecture to $O(\log n)$ in some mixed cases when the color classes are not necessarily stars. In this paper, we extend Guo's results. Our main result is as follows: Let $G$ an edge-colored graph on $n$ vertices and $n$ color classes, if at least $αn$ color classes are either a matching of size 2 or a triangle for $α>\frac{1}{2}$, then $G$ contains a rainbow cycle of length $O(\log n)$. We also prove that the $\log n$ bound is the right order of magnitude.

math.CO↗

arXiv2023

Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years. In particular, Codex and ChatGPT have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications. To mitigate this gap, we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. Besides, we also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: https://bird-bench.github.io/.

cs.CL↗

arXiv2023

Effects of $N(2080){3/2}^-$ and $N(2270)3/2^-$ molecules on $K^\ast Σ$ photoproduction

In the present work, we re-analyze the available data for $γp\to K^{\ast +}Σ^0$ and $γp \to K^{\ast 0}Σ^+$ by considering the contributions from the $N(2080){3/2}^-$ and $N(2270)3/2^-$ molecules instead of any nucleon resonances in the $s$ channel, where the $N(2080)3/2^-$ was proposed to be a $K^\ast Σ$ molecule as the strange partner of the $P_c^+(4457)$ hadronic molecular state, and the $N(2270)3/2^-$ was assumed to be a $K^*Σ^*$ molecule as the strange partner of the $\bar{D}^\ast Σ^\ast_c$ bound states that are predicated as members in the same heavy-quark spin symmetry multiplet as the $P_c$ states. It turns out that all the available cross-section data can be well reproduced, indicating that the molecular structures of the possible $N(2080){3/2}^-$ and $N(2270)3/2^-$ states are compatible with the available data for $K^\astΣ$ photoproduction reactions. Further analysis shows that for both $γp\to K^{\ast +}Σ^0$ and $γp \to K^{\ast 0}Σ^+$ reactions, the $N(2080){3/2}^-$ exchange provides dominant contributions to the cross-sections in the near-threshold energy region, and significant contributions from the $N(2270)3/2^-$ exchange to the cross-sections in the higher energy region are also found. Predictions of the beam asymmetry $Σ$, target asymmetry $T$, and recoil baryon asymmetry $P$ are presented and compared with those from our previous work. Measurements of the data on these observables are called on to further constrain the reaction mechanisms of $K^\astΣ$ photoproduction reactions and to verify the molecular scenario of the $N(2080){3/2}^-$ and $N(2270)3/2^-$ states.

nucl-th↗

arXiv2023

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily focus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collaboration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modality collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experiments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.

cs.CL↗

arXiv2023

Causal Document-Grounded Dialogue Pre-training

The goal of document-grounded dialogue (DocGD) is to generate a response by grounding the evidence in a supporting document in accordance with the dialogue context. This process involves four variables that are causally connected. Recently, task-specific pre-training has greatly boosted performances on many downstream tasks. Existing DocGD methods, however, continue to rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To tackle this issue, we are the first to present a causally-complete dataset construction strategy for building million-level DocGD pre-training corpora. To better capture causality, we further propose a causally-perturbed pre-training strategy, which introduces causal perturbations on the variables and optimizes the overall causal effect. Experiments on three benchmark datasets demonstrate that our causal pre-training achieves considerable and consistent improvements under fully-supervised, low-resource, few-shot, and zero-shot settings.

cs.CL↗

arXiv2023

Global analysis of measured and unmeasured hadronic two-body weak decays of antitriplet charmed baryons

A large amount of data on hadronic two body weak decays of anti-triplet charmed baryons $T_{c\bar 3}$ to an octet baryon $T_8$ and an octet or singlet pseudoscalar meson $P$, $T_{c \bar 3} \to T_8 P$, have been measured. The SU(3) flavor symmetry has been applied to study these decays to obtain insights about weak interactions for charm physics. However not all such decays needed to determine the SU(3) irreducible amplitudes have been measured forbidding a complete global analysis. Previously, it has been shown that data from measured decays can be used to do a global fit to determine all except one parity violating and one parity conserving amplitudes of the relevant SU(3) irreducible amplitudes causing 8 hadronic two body weak decay channels involving $Ξ^0_c$ to $η$ or $η'$ transitions undetermined. It is important to obtain information about these decays in order to guide experimental searches. In this work using newly measured decay modes by BESIII and Belle in 2022, we carry out a global analysis and parameterize the unknown amplitudes to provide the ranges for the branching ratios of the 8 undetermined decays. Our results indicate that the SU(3) flavor symmetry can explain the measured data exceptionally well, with a remarkable minimal $χ^2/d.o.f.$ of 1.21 and predict 80 observables in 45 decays for future experimental data to test. We then vary the unknown SU(3) amplitudes to obtain the allowed range of branching ratios for the 8 undetermined decays. We find that some of them are within reach of near future experimental capabilities. We urge our experimental colleagues to carry out related searches.

hep-ph↗

arXiv2023

SPA: A Graph Spectral Alignment Perspective for Domain Adaptation

Unsupervised domain adaptation (UDA) is a pivotal form in machine learning to extend the in-domain model to the distinctive target domains where the data distributions differ. Most prior works focus on capturing the inter-domain transferability but largely overlook rich intra-domain structures, which empirically results in even worse discriminability. In this work, we introduce a novel graph SPectral Alignment (SPA) framework to tackle the tradeoff. The core of our method is briefly condensed as follows: (i)-by casting the DA problem to graph primitives, SPA composes a coarse graph alignment mechanism with a novel spectral regularizer towards aligning the domain graphs in eigenspaces; (ii)-we further develop a fine-grained message propagation module -- upon a novel neighbor-aware self-training mechanism -- in order for enhanced discriminability in the target domain. On standardized benchmarks, the extensive experiments of SPA demonstrate that its performance has surpassed the existing cutting-edge DA methods. Coupled with dense model analysis, we conclude that our approach indeed possesses superior efficacy, robustness, discriminability, and transferability. Code and data are available at: https://github.com/CrownX/SPA.

cs.CV↗

arXiv2023

EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning

Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose EMMA-X: an EM-like Multilingual pre-training Algorithm, to learn (X)Cross-lingual universals with the aid of excessive multilingual non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of EMMA-X over advanced models.

cs.CL↗

arXiv2023

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.

cs.CL↗

arXiv2023

Improving Question Generation with Multi-level Content Planning

This paper addresses the problem of generating questions from a given context and an answer, specifically focusing on questions that require multi-hop reasoning across an extended context. Previous studies have suggested that key phrase selection is essential for question generation (QG), yet it is still challenging to connect such disjointed phrases into meaningful questions, particularly for long context. To mitigate this issue, we propose MultiFactor, a novel QG framework based on multi-level content planning. Specifically, MultiFactor includes two components: FA-model, which simultaneously selects key phrases and generates full answers, and Q-model which takes the generated full answer as an additional input to generate questions. Here, full answer generation is introduced to connect the short answer with the selected key phrases, thus forming an answer-aware summary to facilitate QG. Both FA-model and Q-model are formalized as simple-yet-effective Phrase-Enhanced Transformers, our joint model for phrase selection and text generation. Experimental results show that our method outperforms strong baselines on two popular QG datasets. Our code is available at https://github.com/zeaver/MultiFactor.

cs.CL↗

arXiv2023

Diversify Question Generation with Retrieval-Augmented Style Transfer

Given a textual passage and an answer, humans are able to ask questions with various expressions, but this ability is still challenging for most question generation (QG) systems. Existing solutions mainly focus on the internal knowledge within the given passage or the semantic word space for diverse content planning. These methods, however, have not considered the potential of external knowledge for expression diversity. To bridge this gap, we propose RAST, a framework for Retrieval-Augmented Style Transfer, where the objective is to utilize the style of diverse templates for question generation. For training RAST, we develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward. Here, the consistency reward is computed by a Question-Answering (QA) model, whereas the diversity reward measures how much the final output mimics the retrieved template. Experimental results show that our method outperforms previous diversity-driven baselines on diversity while being comparable in terms of consistency scores. Our code is available at https://github.com/gouqi666/RAST.

cs.CL↗