arXiv subjects

Hao Zhou

Publications and source records attributed to Hao Zhou.

At least 253 records · Page 14Linked to original sources

arXiv2024

PruneVid: Visual Token Pruning for Efficient Video Large Language Models

In this paper, we introduce PruneVid, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding. Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs. To address this issue, we introduce a training-free method that 1) minimizes video redundancy by merging spatial-temporal tokens, and 2) leverages LLMs' reasoning capabilities to selectively prune visual features relevant to question tokens, enhancing model efficiency. We validate our method across multiple video benchmarks, which demonstrate that PruneVid can prune over 80% of tokens while maintaining competitive performance combined with different model networks. This highlights its superior effectiveness and efficiency compared to existing pruning methods. Code: https://github.com/Visual-AI/PruneVid.

cs.CV↗

arXiv2024

AI-Driven Accelerated Discovery of Intercalation-type Cathode Materials for Magnesium Batteries

Magnesium-ion batteries hold promise as future energy storage solution, yet current Mg cathodes are challenged by low voltage and specific capacity. Herein, we present an AI-driven workflow for discovering high-performance Mg cathode materials. Utilizing the common characteristics of various ionic intercalation-type electrodes, we design and train a Crystal Graph Convolutional Neural Network model that can accurately predicts electrode voltages for various ions with mean absolute errors (MAE) between 0.25 and 0.33 V. By deploying the trained model to stable Mg compounds from Materials Project and GNoME AI dataset, we identify 160 high voltage structures out of 15,308 candidates with voltages above 3.0 V and volumetric capacity over 800 Ah/L. We further train a precise NequIP model to facilitate accurate and rapid simulations of Mg ionic conductivity. From the 160 high voltage structures, the machine learning molecular dynamics simulations have selected 23 cathode materials with both high energy density and high ionic conductivity. This AI-driven workflow dramatically boosts the efficiency and precision of material discovery for multivalent ion batteries, paving the way for advanced Mg battery development.

cond-mat.mtrl-sci↗

arXiv2024

Video Creation by Demonstration

We explore a novel video creation experience, namely Video Creation by Demonstration. Given a demonstration video and a context image from a different scene, we generate a physically plausible video that continues naturally from the context image and carries out the action concepts from the demonstration. To enable this capability, we present $δ$-Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction. Unlike most existing video generation controls that are based on explicit signals, we adopts the form of implicit latent control for maximal flexibility and expressiveness required by general videos. By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process with minimal appearance leakage. Empirically, $δ$-Diffusion outperforms related baselines in terms of both human preference and large-scale machine evaluations, and demonstrates potentials towards interactive world simulation. Sampled video generation results are available at https://delta-diffusion.github.io/.

cs.CV↗

arXiv2024

Question Answering for Decisionmaking in Green Building Design: A Multimodal Data Reasoning Method Driven by Large Language Models

In recent years, the critical role of green buildings in addressing energy consumption and environmental issues has become widely acknowledged. Research indicates that over 40% of potential energy savings can be achieved during the early design stage. Therefore, decision-making in green building design (DGBD), which is based on modeling and performance simulation, is crucial for reducing building energy costs. However, the field of green building encompasses a broad range of specialized knowledge, which involves significant learning costs and results in low decision-making efficiency. Many studies have already applied artificial intelligence (AI) methods to this field. Based on previous research, this study innovatively integrates large language models with DGBD, creating GreenQA, a question answering framework for multimodal data reasoning. Utilizing Retrieval Augmented Generation, Chain of Thought, and Function Call methods, GreenQA enables multimodal question answering, including weather data analysis and visualization, retrieval of green building cases, and knowledge query. Additionally, this study conducted a user survey using the GreenQA web platform. The results showed that 96% of users believed the platform helped improve design efficiency. This study not only effectively supports DGBD but also provides inspiration for AI-assisted design.

cs.AI↗

arXiv2024

Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement

Large Language Models (LLMs) have achieved remarkable progress in recent years; however, their excellent performance is still largely limited to major world languages, primarily English. Many LLMs continue to face challenges with multilingual tasks, especially when it comes to low-resource languages. To address this issue, we introduced Marco-LLM: Massive multilingual training for cross-lingual enhancement LLM. We have collected a substantial amount of multilingual data for several low-resource languages and conducted extensive continual pre-training using the Qwen2 models. This effort has resulted in a multilingual LLM named Marco-LLM. Through comprehensive evaluations on various multilingual benchmarks, including MMMLU, AGIEval, Belebele, Flores-200, XCOPA and many others, Marco-LLM has demonstrated substantial improvements over state-of-the-art LLMs. Furthermore, Marco-LLM achieved substantial enhancements in any-to-any machine translation tasks, showing the effectiveness of our multilingual LLM. Marco-LLM is a pioneering multilingual LLM designed to not only perform exceptionally well in multilingual tasks, including low-resource languages, but also maintain strong performance in English and other major languages, closing the performance gap between high- and low-resource language capabilities. By bridging languages, this effort demonstrates our dedication to ensuring LLMs work accurately across various languages.

cs.CL↗

arXiv2024

HLAT: High-quality Large Language Model Pre-trained on AWS Trainium

Getting large language models (LLMs) to perform well on the downstream tasks requires pre-training over trillions of tokens. This typically demands a large number of powerful computational devices in addition to a stable distributed training framework to accelerate the training. The growing number of applications leveraging AI/ML led to a scarcity of the expensive conventional accelerators (such as GPUs), which emphasizes the need for the alternative specialized-accelerators that are scalable and cost-efficient. AWS Trainium is the second-generation machine learning accelerator purposely built for training large deep learning models. However, training LLMs with billions of parameters on AWS Trainium is challenging due to its relatively nascent software ecosystem. In this paper, we showcase HLAT: a family of 7B and 70B decoder-only LLMs pre-trained using 4096 AWS Trainium accelerators over 1.8 trillion tokens. The performance of HLAT is benchmarked against popular open source models including LLaMA and OpenLLaMA, which have been trained on NVIDIA GPUs and Google TPUs, respectively. On various evaluation tasks, we show that HLAT achieves model quality on par with the baselines of similar model size. We also open-source all the training scripts and configurations of HLAT (https://github.com/awslabs/HLAT) and share the best practice of using the NeuronX Distributed Training (NxDT), a customized distributed training library for AWS Trainium. Our work demonstrates that AWS Trainium powered by NxDT is able to successfully pre-train state-of-the-art LLM models with high performance and cost-effectiveness.

cs.CL↗

arXiv2024

Operating a Multi-Level Molecular Dimer Switch through Precise Tip-Molecule Control

Controlling structural transitions between molecular configurations is crucial for advancing functional molecular electronics. While reversible switching of bistable two-state molecules has been achieved, creating molecular systems that can be controllably switched between multiple configurations often requires complex synthetic methods, presenting a much greater challenge. In this study, we showcase a straightforward yet effective strategy to create and control transitions between multiple molecular structural states by forming a surface-bound molecular dimer. Using low-temperature scanning tunneling microscopy, we induce and characterize the structural transitions of a pyrrolidine dimer on a Cu(100) surface. The intermolecular interactions open new energy transfer channels, enabling the excitation through pathways that were inaccessible in monomers. The occupation of different molecular states is highly sensitive to both the energy of the tunneling electrons and the interaction with the STM tip. By precisely adjusting the tip-molecule distance, we can select the most probable structural configuration based on sample bias, thereby achieving on-demand control of this molecular dimer switch. This work highlights an approach that leverages both intermolecular and molecule-environment interactions to create and control an artificially fabricated molecular device.

physics.chem-ph↗

arXiv2024

Impacts of Point Defects on Shallow Doping in Cubic Boron Arsenide: A First Principles Study

Cubic boron arsenide (BAs) stands out as a promising material for advanced electronics, thanks to its exceptional thermal conductivity and ambipolar mobility. However, effective control of p- and n-type doping in BAs poses a significant challenge, mostly as a result of the influence of defects. In the present study, we employed density functional theory (DFT) to explore the impacts of the common point defects and impurities on p-type doping of Be$_\text{B}$ and Si$_\text{As}$, and on n-type doping of Si$_\text{B}$ and Se$_\text{As}$. We found that the most favorable point defects formed by C, O, and Si are C$_\text{As}$, O$_\text{B}$O$_\text{As}$, Si$_\text{As}$, C$_\text{As}$Si$_\text{B}$, and O$_\text{B}$Si$_\text{As}$, which have formation energies of less than $1.5$ eV. While the O impurity detrimentally affects both n- and p-type dopings, C and Si impurities are harmful for n-type dopings. Interestingly, the antisite defect pair As$_\text{B}$B$_\text{As}$ benefits both p- and n-type doping. The doping limitation analysis presented in this study can potentially pave the way for strategic development in the area of BAs-based electronics.

cond-mat.mtrl-sci↗

arXiv2024

A stabilized nonconforming finite element method for the surface biharmonic problem

This paper presents a novel stabilized nonconforming finite element method for solving the surface biharmonic problem. The method extends the New-Zienkiewicz-type (NZT) element to polyhedral (approximated) surfaces by employing the Piola transform to establish the connection of vertex gradients across adjacent elements. Key features of the surface NZT finite element space include its $H^1$-relative conformity and weak $H({\rm div})$ conformity, allowing for stabilization without the use of artificial parameters. Under the assumption that the exact solution and the dual problem possess only $H^3$ regularity, we establish optimal error estimates in the energy norm and provide, for the first time, a comprehensive analysis yielding optimal second-order convergence in the broken $H^1$ norm. Numerical experiments are provided to support the theoretical results.

math.NA↗

arXiv2024

Penumbra-Effect Induced Spectral Mixing in X-ray Computed Tomography: A Multi-Ray Spectrum Estimation Model and Subsampled Weighting Algorithm

Purpose: With the development of spectral CT, several novel spectral filters have been introduced to modulate the spectra, such as split filters and spectral modulators. However, due to the finite size of the focal spot of X-ray source, these filters cause spectral mixing in the penumbra region. Traditional spectrum estimation methods fail to account for it, resulting in reduced spectral accuracy. Methods: To address this challenge, we develop a multi-ray spectrum estimation model and propose an Adaptive Subsampled WeIghting of Filter Thickness (A-SWIFT) method. First, we estimate the unfiltered spectrum using traditional methods. Next, we model the final spectra as a weighted summation of spectra attenuated by multiple filters. The weights and equivalent lengths are obtained by X-ray transmission measurements taken with altered spectra using different kVp or flat filters. Finally, the spectra are approximated by using the multi-ray model. To mimic the penumbra effect, we used a spectral modulator (0.2 mm Mo, 0.6 mm Mo) and a split filter (0.07 mm Au, 0.7 mm Sn) in simulations, and used a copper modulator and a molybdenum modulator (0.2 mm, 0.6 mm) in experiments. Results: Simulation results show that the mean energy bias in the penumbra region decreased from 7.43 keV using the previous SCFM method (Spectral Compensation for Modulator) to 0.72 keV using the A-SWIFT method for the split filter, and from 1.98 keV to 0.61 keV for the spectral modulator. In experiments, the root mean square error of the selected ROIs was decreased from 77 to 7 Hounsfield units (HU) for the pure water phantom with a molybdenum modulator, and from 85 to 21 HU with a copper modulator. Conclusion: Based on a multi-ray spectrum estimation model, the A-SWIFT method provides an accurate and robust approach for spectrum estimation in penumbra region of CT systems utilizing spectral filters.

physics.med-ph↗

arXiv2024

MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

Studying protein mutations within amino acid sequences holds tremendous significance in life sciences. Protein language models (PLMs) have demonstrated strong capabilities in broad biological applications. However, due to architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies. To address these issues, we present MutaPLM, a unified framework for interpreting and navigating protein mutations with protein language models. MutaPLM introduces a protein delta network that captures explicit protein mutation representations within a unified feature space, and a transfer learning pipeline with a chain-of-thought (CoT) strategy to harvest protein mutation knowledge from biomedical texts. We also construct MutaDescribe, the first large-scale protein mutation dataset with rich textual annotations, which provides cross-modal supervision signals. Through comprehensive experiments, we demonstrate that MutaPLM excels at providing human-understandable explanations for mutational effects and prioritizing novel mutations with desirable properties. Our code, model, and data are open-sourced at https://github.com/PharMolix/MutaPLM.

cs.LG↗

arXiv2024

Vision-Language Models Can Self-Improve Reasoning via Reflection

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R3V, which iteratively enhances the model's Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23 to 60 percent over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation.

cs.LG↗

arXiv2024

P$^2$C$^2$Net: PDE-Preserved Coarse Correction Network for efficient prediction of spatiotemporal dynamics

When solving partial differential equations (PDEs), classical numerical methods often require fine mesh grids and small time stepping to meet stability, consistency, and convergence conditions, leading to high computational cost. Recently, machine learning has been increasingly utilized to solve PDE problems, but they often encounter challenges related to interpretability, generalizability, and strong dependency on rich labeled data. Hence, we introduce a new PDE-Preserved Coarse Correction Network (P$^2$C$^2$Net) to efficiently solve spatiotemporal PDE problems on coarse mesh grids in small data regimes. The model consists of two synergistic modules: (1) a trainable PDE block that learns to update the coarse solution (i.e., the system state), based on a high-order numerical scheme with boundary condition encoding, and (2) a neural network block that consistently corrects the solution on the fly. In particular, we propose a learnable symmetric Conv filter, with weights shared over the entire model, to accurately estimate the spatial derivatives of PDE based on the neural-corrected system state. The resulting physics-encoded model is capable of handling limited training data (e.g., 3--5 trajectories) and accelerates the prediction of PDE solutions on coarse spatiotemporal grids while maintaining a high accuracy. P$^2$C$^2$Net achieves consistent state-of-the-art performance with over 50\% gain (e.g., in terms of relative prediction error) across four datasets covering complex reaction-diffusion processes and turbulent flows.

math.NA↗

arXiv2024

VideoGLUE: Video General Understanding Evaluation of Foundation Models

We evaluate the video understanding capabilities of existing foundation models (FMs) using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition,temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring an FM for downstream tasks. Furthermore, we jointly profile FMs' efficacy and efficiency when adapting to general video understanding tasks using cost measurements during both training and inference. Our main findings areas follows. First, task-specialized models significantly outperform the seven FMs studied in this work, in sharp contrast to what FMs have achieved in natural language and image understanding. Second, video-native FMs, whose pretraining data mainly contains the video modality, are generally better than image-native FMs in classifying motion-rich videos, localizing actions in time, and understanding a video of more than one action. Third, the video-native FMs can perform well on video tasks under light adaptations to downstream tasks (e.g., freezing the FM backbones), while image-native FMs win in full end-to-end finetuning. The first two observations reveal the need and tremendous opportunities to conduct research on video-focused FMs, and the last confirms that both tasks and adaptation methods matter when it comes to the evaluation of FMs. Our code is released under: https://github.com/tensorflow/models/tree/master/official/projects/videoglue.

cs.CV↗

arXiv2024

Ruddlesden-Popper chalcogenides push the limit of mechanical stiffness and glass-like thermal conductivity in single crystals

Insulating materials featuring ultralow thermal conductivity for diverse applications also require robust mechanical properties. Conventional thinking, however, which correlates strong bonding with high atomic-vibration-mediated heat conduction, led to diverse weakly bonded materials that feature ultralow thermal conductivity and low elastic moduli. One must, therefore, search for strongly-bonded single crystals in which heat transport is impeded by other means. Here, we report intrinsic, glass-like, ultralow thermal conductivity and ultrahigh elastic-modulus/thermal-conductivity ratio in single-crystalline Ruddlesden-Popper Ban+1ZrnS3n+1, n = 2,3, which are derivatives of BaZrS3. Their key features are strong anharmonicity and intra-unit-cell rock-salt blocks. The latter produce strongly bonded intrinsic superlattices, impeding heat conduction by broadband reduction of phonon velocities and mean free paths and concomitant strong phonon localization. The present study initiates a paradigm of mechanically stiff phonon glasses.

cond-mat.mtrl-sci↗

arXiv2024

Optimal Bailouts and Strategic Debt Forgiveness in Financial Networks

A financial system is represented by a network, where nodes correspond to banks, and directed labeled edges correspond to debt contracts between banks. Once a payment schedule has been defined, where we assume that a bank cannot refuse a payment towards one of its lenders if it has sufficient funds, the liquidity of the system is defined as the sum of total payments made in the network. Maximizing systemic liquidity is a natural objective of any financial authority, so, we study the setting where the financial authority offers bailout money to some bank(s) or forgives the debts of others in order to maximize liquidity, and examine efficient ways to achieve this. We investigate the approximation ratio provided by the greedy bailout policy compared to the optimal one, and we study the computational hardness of finding the optimal debt-removal and budget-constrained optimal bailout policy, respectively. We also study financial systems from a game-theoretic standpoint. We observe that the removal of some incoming debt might be in the best interest of a bank, if that helps one of its borrowers remain solvent and avoid costs related to default. Assuming that a bank's well-being (i.e., utility) is aligned with the incoming payments they receive from the network, we define and analyze a game among banks who want to maximize their utility by strategically giving up some incoming payments. In addition, we extend the previous game by considering bailout payments. After formally defining the above games, we prove results about the existence and quality of pure Nash equilibria, as well as the computational complexity of finding such equilibria.

cs.GT↗

arXiv2024

An Agile Large-Workspace Teleoperation Interface Based on Human Arm Motion and Force Estimation

Teleoperation can transfer human perception and cognition to a slave robot to cope with some complex tasks, in which the agility and flexibility of the interface play an important role in mapping human intention to the robot. In this paper, we developed an agile large-workspace teleoperation interface by estimating human arm behavior. Using the wearable sensor, namely the inertial measurement unit and surface electromyography armband, we can capture the human arm motion and force information, thereby intuitively controlling the manipulation of the robot. The control principle of our wearable interface includes two parts: (1) the arm incremental kinematics and (2) the grasping recognition. Moreover, we developed a teleoperation framework with a time synchronization mechanism for the real-time application. We conducted experimental comparisons with a versatile haptic device (Omega 7) to verify the effectiveness of our interface and framework. Seven subjects are invited to complete three different tasks: free motion, handover, and pick-and-place action (each task ten times), and the total number of tests is 420. Objectively, we used the task completion time and success rate to compare the performance of the two interfaces quantitatively. In addition, to quantify the operator experience, we used the NASA Task Load Index to assess their subjective feelings. The results showed that the proposed interface achieved a competitive performance with a better operating experience.

cs.RO↗

arXiv2024

Secure Ranging with IEEE 802.15.4z HRP UWB

Secure ranging refers to the capability of upper-bounding the actual physical distance between two devices with reliability. This is essential in a variety of applications, including to unlock physical systems. In this work, we will look at secure ranging in the context of ultra-wideband impulse radio (UWB-IR) as specified in IEEE 802.15.4z (a.k.a. 4z). In particular, an encrypted waveform, i.e. the scrambled timestamp sequence (STS), is defined in the high rate pulse repetition frequency (HRP) mode of operation in 4z for secure ranging. This work demonstrates the security analysis of 4z HRP when implemented with an adequate receiver design and shows the STS waveform can enable secure ranging. We first review the STS receivers adopted in previous studies and analyze their security vulnerabilities. Then we present a reference STS receiver and prove that secure ranging can be achieved by employing the STS waveform in 4z HRP. The performance bounds of the reference secure STS receiver are also characterized. Numerical experiments corroborate the analyses and demonstrate the security of the reference STS receiver.

cs.CR↗