arXiv subjects

Tao Luo

Publications and source records attributed to Tao Luo.

At least 91 records · Page 5Linked to original sources

arXiv2025

Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks

Understanding the relation between deep and shallow neural networks is extremely important for the theoretical study of deep learning. In this work, we discover an embedding principle in depth that loss landscape of an NN "contains" all critical points of the loss landscapes for shallower NNs. The key tool for our discovery is the critical lifting operator proposed in this work that maps any critical point of a network to critical manifolds of any deeper network while preserving the outputs. This principle provides new insights to many widely observed behaviors of DNNs. Regarding the easy training of deep networks, we show that local minimum of an NN can be lifted to strict saddle points of a deeper NN. Regarding the acceleration effect of batch normalization, we demonstrate that batch normalization helps avoid the critical manifolds lifted from shallower NNs by suppressing layer linearization. We also prove that increasing training data shrinks the lifted critical manifolds, which can result in acceleration of training as demonstrated in experiments. Overall, our discovery of the embedding principle in depth uncovers the depth-wise hierarchical structure of deep learning loss landscape, which serves as a solid foundation for the further study about the role of depth for DNNs.

cs.LG↗

arXiv2025

Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

Under mild assumptions, we investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. Utilizing novel techniques, we demonstrate: (i) how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows; and (ii) the local convergence properties and rate of gradient flow dynamics. Our results indicate that two-layer neural networks can be locally recovered in the regime of overparameterization.

cs.LG↗

arXiv2025

Strong solutions to the 3-D compressible MHD equations with density-dependent viscosities in exterior domains with far-field vacuum

This paper investigates the existence and uniqueness of local strong solutions to the three-dimensional compressible magnetohydrodynamic (MHD) equations with density-dependent viscosities in an exterior domain. The system models the dynamics of electrically conducting fluids, such as plasmas, and incorporates the effects of magnetic fields on fluid motion. We focus on the case where the viscosity coefficients are proportional to the fluid density, and the far-field density approaches vacuum. By introducing a reformulation of the problem using new variables to handle the degeneracy near vacuum, we establish the local well-posedness of strong solutions for arbitrarily large initial data, even in the presence of far-field vacuum. Our analysis employs energy estimates, elliptic regularity theory, and a careful treatment of the Navier-slip boundary conditions for the velocity and perfect conductivity conditions for the magnetic field. To the best of our knowledge, such results are not available even for the Cauchy problem to the 3-D compressible MHD equations with degenerate viscosities.

math.AP↗

arXiv2025

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

The optimal objective is a fundamental aspect of reinforcement learning (RL), as it determines how policies are evaluated and optimized. While total return maximization is the ideal objective in RL, discounted return maximization is the practical objective due to its stability. This can lead to a misalignment of objectives. To better understand the problem, we theoretically analyze the performance gap between the policy maximizes the total return and the policy maximizes the discounted return. Our analysis reveals that increasing the discount factor can be ineffective at eliminating this gap when environment contains cyclic states,a frequent scenario. To address this issue, we propose two alternative approaches to align the objectives. The first approach achieves alignment by modifying the terminal state value, treating it as a tunable hyper-parameter with its suitable range defined through theoretical analysis. The second approach focuses on calibrating the reward data in trajectories, enabling alignment in practical Deep RL applications using off-policy algorithms. This method enhances robustness to the discount factor and improve performance when the trajectory length is large. Our proposed methods demonstrate that adjusting reward data can achieve alignment, providing an insight that can be leveraged to design new optimization objectives to fundamentally enhance the performance of RL algorithms.

cs.LG↗

arXiv2025

Linear Programming Bounds on $k$-Uniform States

The existence of $k$-uniform states has been a widely studied problem due to their applications in several quantum information tasks and their close relation to combinatorial objects like Latin squares and orthogonal arrays. With the machinery of quantum enumerators and linear programming, we establish several improved non-existence results and bounds on $k$-uniform states. 1. First, for any fixed $l\geq 1$ and $q\geq 2$, we show that there exists a constant $c$ such that $(\left\lfloor{n/2}\right\rfloor-l)$-uniform states in $(\mathbb{C}^q)^{\otimes n}$ do not exist when $n\geq cq^2+o(q^2)$. The constant $c$ equals $4$ when $l=1$ and $6$ when $l=2$, which generalizes Scott's bound (2004) for $l=0$. 2. Second, when $n$ is sufficiently large, we show that there exists a constant $θ<1/2$ for each $q \le 9$, such that $k$-uniform states in $(\mathbb{C}^q)^{\otimes n}$ exist only when $k\leq θn$. In particular, this provides the first bound (to the best of our knowledge) of $k$ for $4\leq q\leq 9$ and confirms a conjecture posed by Shi et al. (2023) when $q=5$ in a stronger form. 3. Finally, we improve the shadow bounds given by Shi et al. (2023) by a constant for $q = 3,4,5$ and small $n$. When $q=4$, our results can update some bounds listed in the code tables maintained by Grassl (2007--2024).

quant-ph↗

arXiv2025

Isotropic superconductivity in pressurized trilayer nickelate La4Ni3O10

Evidence of superconductivity (SC) has recently been reported in pressurized La3Ni2O7 and La4Ni3O10, providing a new platform to explore high-temperature superconductivity. However, while zero resistance state has been observed, experimental characterization of the superconducting properties of pressurized nickelates is still limited and experimentally challenging. Here, we present the first full temperature dependence of the upper critical field Hc2 measurement in La4Ni3O10 single crystal, achieved by combining high magnetic field and high-pressure techniques. Remarkably, the Hc2 of La4Ni3O10 is nearly isotropic, with the anisotropic parameter monotonically increasing from 1.4 near Tc to 1 at lower temperatures. By analyzing the Hc2 using the two-band model, we uncover that the anisotropic diffusivity of the bands, primarily originating from d(z2 ) and d(x2-y2 ) orbitals, is well compensated, resulting in an unusually isotropic superconducting state. These findings provide critical experimental evidence that underscores the significant role of the d(z2 ) orbital in enabling superconductivity in pressurized Ruddlesden-Popper nickelates.

cond-mat.supr-con↗

arXiv2025

Harnessing the Power of Vibration Motors to Develop Miniature Untethered Robotic Fishes

Miniature underwater robots play a crucial role in the exploration and development of marine resources, particularly in confined spaces and high-pressure deep-sea environments. This study presents the design, optimization, and performance of a miniature robotic fish, powered by the oscillation of bio-inspired fins. These fins feature a rigid-flexible hybrid structure and use an eccentric rotating mass (ERM) vibration motor as the excitation source to generate high-frequency unidirectional oscillations that induce acoustic streaming for propulsion. The drive mechanism, powered by miniature ERM vibration motors, eliminates the need for complex mechanical drive systems, enabling complete isolation of the entire drive system from the external environment and facilitating the miniaturization of the robotic fish. A compact, untethered robotic fish, measuring 85*60*45 mm^3, is equipped with three bio-inspired fins located at the pectoral and caudal positions. Experimental results demonstrate that the robotic fish achieves a maximum forward swimming speed of 1.36 body lengths (BL) per second powered by all fins and minimum turning radius of 0.6 BL when powered by a single fin. These results underscore the significance of employing the ERM vibration motor in advancing the development of highly maneuverable, miniature untethered underwater robots for various marine exploration tasks.

cs.RO↗

arXiv2024

Exploring Viewing Modalities in Cinematic Virtual Reality: A Systematic Review and Meta-Analysis of Challenges in Evaluating User Experience

Cinematic Virtual Reality (CVR) is a narrative-driven VR experience that uses head-mounted displays with a 360-degree field of view. Previous research has explored different viewing modalities to enhance viewers' CVR experience. This study conducted a systematic review and meta-analysis focusing on how different viewing modalities, including intervened rotation, avatar assistance, guidance cues, and perspective shifting, influence the CVR experience. The study has screened 3444 papers (between 01/01/2013 and 17/06/2023) and selected 45 for systematic review, 13 of which also for meta-analysis. We conducted separate random-effects meta-analysis and applied Robust Variance Estimation to examine CVR viewing modalities and user experience outcomes. Evidence from experiments was synthesized as differences between standardized mean differences (SMDs) of user experience of control group ("Swivel-Chair" CVR) and experiment groups. To our surprise, we found inconsistencies in the effect sizes across different studies, even with the same viewing modalities. Moreover, in these studies, terms such as "presence," "immersion," and "narrative engagement" were often used interchangeably. Their irregular use of questionnaires, overreliance on self-developed questionnaires, and incomplete data reporting may have led to unrigorous evaluations of CVR experiences. This study contributes to Human-Computer Interaction (HCI) research by identifying gaps in CVR research, emphasizing the need for standardization of terminologies and methodologies to enhance the reliability and comparability of future CVR research.

cs.HC↗

arXiv2024

Overview frequency principle/spectral bias in deep learning

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

cs.LG↗

arXiv2024

On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages

The multi-stage phenomenon in the training loss curves of neural networks has been widely observed, reflecting the non-linearity and complexity inherent in the training process. In this work, we investigate the training dynamics of neural networks (NNs), with particular emphasis on the small initialization regime, identifying three distinct stages observed in the loss curve during training: the initial plateau stage, the initial descent stage, and the secondary plateau stage. Through rigorous analysis, we reveal the underlying challenges contributing to slow training during the plateau stages. While the proof and estimate for the emergence of the initial plateau were established in our previous work, the behaviors of the initial descent and secondary plateau stages had not been explored before. Here, we provide a more detailed proof for the initial plateau, followed by a comprehensive analysis of the initial descent stage dynamics. Furthermore, we examine the factors facilitating the network's ability to overcome the prolonged secondary plateau, supported by both experimental evidence and heuristic reasoning. Finally, to clarify the link between global training trends and local parameter adjustments, we use the Wasserstein distance to track the fine-scale evolution of weight amplitude distribution.

cs.LG↗

arXiv2024

Enabling Energy-Efficient Deployment of Large Language Models on Memristor Crossbar: A Synergy of Large and Small

Large language models (LLMs) have garnered substantial attention due to their promising applications in diverse domains. Nevertheless, the increasing size of LLMs comes with a significant surge in the computational requirements for training and deployment. Memristor crossbars have emerged as a promising solution, which demonstrated a small footprint and remarkably high energy efficiency in computer vision (CV) models. Memristors possess higher density compared to conventional memory technologies, making them highly suitable for effectively managing the extreme model size associated with LLMs. However, deploying LLMs on memristor crossbars faces three major challenges. Firstly, the size of LLMs increases rapidly, already surpassing the capabilities of state-of-the-art memristor chips. Secondly, LLMs often incorporate multi-head attention blocks, which involve non-weight stationary multiplications that traditional memristor crossbars cannot support. Third, while memristor crossbars excel at performing linear operations, they are not capable of executing complex nonlinear operations in LLM such as softmax and layer normalization. To address these challenges, we present a novel architecture for the memristor crossbar that enables the deployment of state-of-the-art LLM on a single chip or package, eliminating the energy and time inefficiencies associated with off-chip communication. Our testing on BERT_Large showed negligible accuracy loss. Compared to traditional memristor crossbars, our architecture achieves enhancements of up to 39X in area overhead and 18X in energy consumption. Compared to modern TPU/GPU systems, our architecture demonstrates at least a 68X reduction in the area-delay product and a significant 69% energy consumption reduction.

cs.AI↗

arXiv2024

Multi-modal Image and Radio Frequency Fusion for Optimizing Vehicle Positioning

In this paper, a multi-modal vehicle positioning framework that jointly localizes vehicles with channel state information (CSI) and images is designed. In particular, we consider an outdoor scenario where each vehicle can communicate with only one BS, and hence, it can upload its estimated CSI to only its associated BS. Each BS is equipped with a set of cameras, such that it can collect a small number of labeled CSI, a large number of unlabeled CSI, and the images taken by cameras. To exploit the unlabeled CSI data and position labels obtained from images, we design an meta-learning based hard expectation-maximization (EM) algorithm. Specifically, since we do not know the corresponding relationship between unlabeled CSI and the multiple vehicle locations in images, we formulate the calculation of the training objective as a minimum matching problem. To reduce the impact of label noises caused by incorrect matching between unlabeled CSI and vehicle locations obtained from images and achieve better convergence, we introduce a weighted loss function on the unlabeled datasets, and study the use of a meta-learning algorithm for computing the weighted loss. Subsequently, the model parameters are updated according to the weighted loss function of unlabeled CSI samples and their matched position labels obtained from images. Simulation results show that the proposed method can reduce the positioning error by up to 61% compared to a baseline that does not use images and uses only CSI fingerprint for vehicle positioning.

eess.SP↗

arXiv2024

Multi-modal Data based Semi-Supervised Learning for Vehicle Positioning

In this paper, a multi-modal data based semi-supervised learning (SSL) framework that jointly use channel state information (CSI) data and RGB images for vehicle positioning is designed. In particular, an outdoor positioning system where the vehicle locations are determined by a base station (BS) is considered. The BS equipped with several cameras can collect a large amount of unlabeled CSI data and a small number of labeled CSI data of vehicles, and the images taken by cameras. Although the collected images contain partial information of vehicles (i.e. azimuth angles of vehicles), the relationship between the unlabeled CSI data and its azimuth angle, and the distances between the BS and the vehicles captured by images are both unknown. Therefore, the images cannot be directly used as the labels of unlabeled CSI data to train a positioning model. To exploit unlabeled CSI data and images, a SSL framework that consists of a pretraining stage and a downstream training stage is proposed. In the pretraining stage, the azimuth angles obtained from the images are considered as the labels of unlabeled CSI data to pretrain the positioning model. In the downstream training stage, a small sized labeled dataset in which the accurate vehicle positions are considered as labels is used to retrain the model. Simulation results show that the proposed method can reduce the positioning error by up to 30% compared to a baseline where the model is not pretrained.

eess.SP↗

arXiv2024

Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers

Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods leverage neural networks as basis functions to approximate PDE solutions. However, training such networks can be challenging, often resulting in limited accuracy. In this paper, we investigate the training dynamics of neural network-based PDE solvers with a focus on the impact of initialization techniques. We assess training difficulty by analyzing the eigenvalue distribution of the kernel and apply the concept of effective rank to quantify this difficulty, where a larger effective rank correlates with faster convergence of the training error. Building upon this, we discover through theoretical analysis and numerical experiments that two initialization techniques, partition of unity (PoU) and variance scaling (VS), enhance the effective rank, thereby accelerating the convergence of training error. Furthermore, comprehensive experiments using popular PDE-solving frameworks, such as PINN, Deep Ritz, and the operator learning framework DeepOnet, confirm that these initialization techniques consistently speed up convergence, in line with our theoretical findings.

math.NA↗

arXiv2024

Towards Energy- and Cost-Efficient 6G Networks

As the world enters the journey toward the 6th generation (6G) of wireless technology, the promises of ultra-high data rates, unprecedented low latency, and a massive surge in connected devices require crucial exploration of network energy saving (NES) solutions to minimize the carbon footprint and overall energy usage of future cellular networks. On the other hand, network-controlled repeaters (NCRs) have been introduced by 3rd generation partnership project (3GPP) as a cost-effective solution to improve network coverage. However, their impact on network power consumption and energy efficiency has not been thoroughly investigated. This paper studies NES schemes for next-generation 6G networks aided by NCRs and proposes optimal NES strategies aiming at maximizing the overall energy efficiency of the network. Repeaters are shown to allow for power savings at next-generation nodeB (gNB), and offer higher overall energy efficiency (EE) and spectral efficiency (SE), thus providing an energy-efficient and cost-efficient alternative to increase the performance of future 6G networks

cs.NI↗

arXiv2024

UdeerLID+: Integrating LiDAR, Image, and Relative Depth with Semi-Supervised

Road segmentation is a critical task for autonomous driving systems, requiring accurate and robust methods to classify road surfaces from various environmental data. Our work introduces an innovative approach that integrates LiDAR point cloud data, visual image, and relative depth maps derived from images. The integration of multiple data sources in road segmentation presents both opportunities and challenges. One of the primary challenges is the scarcity of large-scale, accurately labeled datasets that are necessary for training robust deep learning models. To address this, we have developed the [UdeerLID+] framework under a semi-supervised learning paradigm. Experiments results on KITTI datasets validate the superior performance.

cs.CV↗

arXiv2024

Exponential stability of a free boundary problem with spherical symmetry for a gas bubble immersed in a bounded incompressible liquid

This paper is mainly concerned with the free boundary problem for an approximate model (for example, arising from the study of sonoluminescence) of a gas bubble of finite mass enclosed within a bounded incompressible viscous liquid, accounting for surface tensions at both the gas-liquid interface and the external free surface of the entire gas-liquid region. It is found that any regular spherically symmetric steady-state solution is characterized by a positive root of a ninth-degree polynomial for which the existence and uniqueness are proved and a one-to-one correspondence between equilibria and pairs of gas mass and liquid volume is established. We prove that these equilibria exhibit nonlinear and exponential asymptotic stability under small perturbations that conserve gas mass and liquid volume, and an equilibrium solution acts as a local minimizer of the energy functional, even under relatively large perturbations, with the proportionality constant determined by the adiabatic constant. Moreover, we construct a global center manifold to apply the center manifold theory. Our results apply to gases and liquids of all sizes. Furthermore, we derive the optimal exponential decay rate for small liquid volumes by analyzing the spectrum bounds of the associated linear operator and show that decreasing the gas mass or increasing the temperature can accelerate the convergence rate, a behavior not seen in unbounded liquid scenarios.

math.AP↗

arXiv2024

UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

Masked Autoencoders (MAE) play a pivotal role in learning potent representations, delivering outstanding results across various 3D perception tasks essential for autonomous driving. In real-world driving scenarios, it's commonplace to deploy multiple sensors for comprehensive environment perception. Despite integrating multi-modal features from these sensors can produce rich and powerful features, there is a noticeable challenge in MAE methods addressing this integration due to the substantial disparity between the different modalities. This research delves into multi-modal Masked Autoencoders tailored for a unified representation space in autonomous driving, aiming to pioneer a more efficient fusion of two distinct modalities. To intricately marry the semantics inherent in images with the geometric intricacies of LiDAR point clouds, we propose UniM$^2$AE. This model stands as a potent yet straightforward, multi-modal self-supervised pre-training framework, mainly consisting of two designs. First, it projects the features from both modalities into a cohesive 3D volume space to intricately marry the bird's eye view (BEV) with the height dimension. The extension allows for a precise representation of objects and reduces information loss when aligning multi-modal features. Second, the Multi-modal 3D Interactive Module (MMIM) is invoked to facilitate the efficient inter-modal interaction during the interaction process. Extensive experiments conducted on the nuScenes Dataset attest to the efficacy of UniM$^2$AE, indicating enhancements in 3D object detection and BEV map segmentation by 1.2\% NDS and 6.5\% mIoU, respectively. The code is available at https://github.com/hollow-503/UniM2AE.

cs.CV↗