arXiv subjects

Sheng Han

Publications and source records attributed to Sheng Han.

13 recordsLinked to original sources

arXiv2026

From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training

Unsupervised pre-training on large-scale datasets has demonstrated significant potential for improving the sample efficiency and performance of Reinforcement Learning (RL). Given the large-scale action-free internet videos, existing methods utilize single-step transition prediction and image reconstruction to learn representations. However, these methods prefer to preserve large-proportion stationary information in the pixel space, neglecting small but crucial information. To preserve enough information in the representation, it is essential to pay equal attention to each element in videos. Specifically, we propose a temporal correlation space to distinguish each element. For implementation, we introduce the Multi-scale Temporal Contrastive Learning (MTCL) method to model multi-scale temporal correlations separately. This approach can balance the attention of different elements and yield more informative representations, effectively supporting policy learning in various downstream tasks. Experimental results demonstrate that our method improves sample efficiency and asymptotic performance across various downstream tasks.

cs.LG↗

arXiv2025

Think How Your Teammates Think: Active Inference Can Benefit Decentralized Execution

In multi-agent systems, explicit cognition of teammates' decision logic serves as a critical factor in facilitating coordination. Communication (i.e., ``\textit{Tell}'') can assist in the cognitive development process by information dissemination, yet it is inevitably subject to real-world constraints such as noise, latency, and attacks. Therefore, building the understanding of teammates' decisions without communication remains challenging. To address this, we propose a novel non-communication MARL framework that realizes the construction of cognition through local observation-based modeling (i.e., \textit{``Think''}). Our framework enables agents to model teammates' \textbf{active inference} process. At first, the proposed method produces three teammate portraits: perception-belief-action. Specifically, we model the teammate's decision process as follows: 1) Perception: observing environments; 2) Belief: forming beliefs; 3) Action: making decisions. Then, we selectively integrate the belief portrait into the decision process based on the accuracy and relevance of the perception portrait. This enables the selection of cooperative teammates and facilitates effective collaboration. Extensive experiments on the SMAC, SMACv2, MPE, and GRF benchmarks demonstrate the superior performance of our method.

cs.MA↗

arXiv2025

Improving monotonic optimization in heterogeneous multi-agent reinforcement learning with optimal marginal deterministic policy gradient

In heterogeneous multi-agent reinforcement learning (MARL), achieving monotonic improvement plays a pivotal role in enhancing performance. The HAPPO algorithm proposes a feasible solution by introducing a sequential update scheme, which requires independent learning with No Parameter-sharing (NoPS). However, heterogeneous MARL generally requires Partial Parameter-sharing (ParPS) based on agent grouping to achieve high cooperative performance. Our experiments prove that directly combining ParPS with the sequential update scheme leads to the policy updating baseline drift problem, thereby failing to achieve improvement. To solve the conflict between monotonic improvement and ParPS, we propose the Optimal Marginal Deterministic Policy Gradient (OMDPG) algorithm. First, we replace the sequentially computed $Q_{\psi}^s(s,a_{1:i})$ with the Optimal Marginal Q (OMQ) function $\phi_{\psi}^*(s,a_{1:i})$ derived from Q-functions. This maintains MAAD's monotonic improvement while eliminating the conflict through optimal joint action sequences instead of sequential policy ratio calculations. Second, we introduce the Generalized Q Critic (GQC) as the critic function, employing pessimistic uncertainty-constrained loss to optimize different Q-value estimations. This provides the required Q-values for OMQ computation and stable baselines for actor updates. Finally, we implement a Centralized Critic Grouped Actor (CCGA) architecture that simultaneously achieves ParPS in local policy networks and accurate global Q-function computation. Experimental results in SMAC and MAMuJoCo environments demonstrate that OMDPG outperforms various state-of-the-art MARL baselines.

cs.AI↗

arXiv2025

CoDe: Communication Delay-Tolerant Multi-Agent Collaboration via Dual Alignment of Intent and Timeliness

Communication has been widely employed to enhance multi-agent collaboration. Previous research has typically assumed delay-free communication, a strong assumption that is challenging to meet in practice. However, real-world agents suffer from channel delays, receiving messages sent at different time points, termed {\it{Asynchronous Communication}}, leading to cognitive biases and breakdowns in collaboration. This paper first defines two communication delay settings in MARL and emphasizes their harm to collaboration. To handle the above delays, this paper proposes a novel framework, Communication Delay-tolerant Multi-Agent Collaboration (CoDe). At first, CoDe learns an intent representation as messages through future action inference, reflecting the stable future behavioral trends of the agents. Then, CoDe devises a dual alignment mechanism of intent and timeliness to strengthen the fusion process of asynchronous messages. In this way, agents can extract the long-term intent of others, even from delayed messages, and selectively utilize the most recent messages that are relevant to their intent. Experimental results demonstrate that CoDe outperforms baseline algorithms in three MARL benchmarks without delay and exhibits robustness under fixed and time-varying delays.

cs.MA↗

arXiv2024

Improving Global Parameter-sharing in Physically Heterogeneous Multi-agent Reinforcement Learning with Unified Action Space

In a multi-agent system (MAS), action semantics indicates the different influences of agents' actions toward other entities, and can be used to divide agents into groups in a physically heterogeneous MAS. Previous multi-agent reinforcement learning (MARL) algorithms apply global parameter-sharing across different types of heterogeneous agents without careful discrimination of different action semantics. This common implementation decreases the cooperation and coordination between agents in complex situations. However, fully independent agent parameters dramatically increase the computational cost and training difficulty. In order to benefit from the usage of different action semantics while also maintaining a proper parameter-sharing structure, we introduce the Unified Action Space (UAS) to fulfill the requirement. The UAS is the union set of all agent actions with different semantics. All agents first calculate their unified representation in the UAS, and then generate their heterogeneous action policies using different available-action-masks. To further improve the training of extra UAS parameters, we introduce a Cross-Group Inverse (CGI) loss to predict other groups' agent policies with the trajectory information. As a universal method for solving the physically heterogeneous MARL problem, we implement the UAS adding to both value-based and policy-based MARL algorithms, and propose two practical algorithms: U-QMIX and U-MAPPO. Experimental results in the SMAC environment prove the effectiveness of both U-QMIX and U-MAPPO compared with several state-of-the-art MARL methods.

cs.MA↗

arXiv2023

GHQ: Grouped Hybrid Q Learning for Heterogeneous Cooperative Multi-agent Reinforcement Learning

Previous deep multi-agent reinforcement learning (MARL) algorithms have achieved impressive results, typically in homogeneous scenarios. However, heterogeneous scenarios are also very common and usually harder to solve. In this paper, we mainly discuss cooperative heterogeneous MARL problems in Starcraft Multi-Agent Challenges (SMAC) environment. We firstly define and describe the heterogeneous problems in SMAC. In order to comprehensively reveal and study the problem, we make new maps added to the original SMAC maps. We find that baseline algorithms fail to perform well in those heterogeneous maps. To address this issue, we propose the Grouped Individual-Global-Max Consistency (GIGM) and a novel MARL algorithm, Grouped Hybrid Q Learning (GHQ). GHQ separates agents into several groups and keeps individual parameters for each group, along with a novel hybrid structure for factorization. To enhance coordination between groups, we maximize the Inter-group Mutual Information (IGMI) between groups' trajectories. Experiments on original and new heterogeneous maps show the fabulous performance of GHQ compared to other state-of-the-art algorithms.

cs.MA↗

arXiv2022

Conservative Distributional Reinforcement Learning with Safety Constraints

Safety exploration can be regarded as a constrained Markov decision problem where the expected long-term cost is constrained. Previous off-policy algorithms convert the constrained optimization problem into the corresponding unconstrained dual problem by introducing the Lagrangian relaxation technique. However, the cost function of the above algorithms provides inaccurate estimations and causes the instability of the Lagrange multiplier learning. In this paper, we present a novel off-policy reinforcement learning algorithm called Conservative Distributional Maximum a Posteriori Policy Optimization (CDMPO). At first, to accurately judge whether the current situation satisfies the constraints, CDMPO adapts distributional reinforcement learning method to estimate the Q-function and C-function. Then, CDMPO uses a conservative value function loss to reduce the number of violations of constraints during the exploration process. In addition, we utilize Weighted Average Proportional Integral Derivative (WAPID) to update the Lagrange multiplier stably. Empirical results show that the proposed method has fewer violations of constraints in the early exploration process. The final test results also illustrate that our method has better risk control.

cs.LG↗

arXiv2020

Tunable lattice reconstruction and bandwidth of flat bands in magic-angle twisted bilayer graphene

The interplay between interlayer van der Waals interaction and intralayer lattice distortion can lead to structural reconstruction in slightly twisted bilayer graphene (TBG) with the twist angle being smaller than a characteristic angle {\theta}c. Experimentally, the {\theta}c is demonstrated to be very close to the magic angle ({\theta} ~ 1.05{\deg}). In this work, we address the transition between reconstructed and unreconstructed structures of the TBG across the magic angle by using scanning tunnelling microscopy (STM). Our experiment demonstrates that both the two structures are stable in the TBG around the magic angle. By applying a STM tip pulse, we show that the two structures can be switched to each other and the bandwidth of the flat bands, which plays a vital role in the emergent strongly correlated states in the magic-angle TBG, can be tuned. The observed tunable lattice reconstruction and bandwidth of the flat bands provide an extra control knob to manipulate the exotic electronic states of the TBG near the magic angle.

cond-mat.mes-hall↗

arXiv2020

Robust two-dimensional ice on graphene built from finite-length water molecular chains

Interfacial ice on graphene has attracted much attention because it is a model system to study two-dimensional (2D) ice structures on chemically inert substrates. While water-graphene interaction was usually assumed to be negligible, the structures of the 2D ice are believed to be not appreciably perturbed by the graphene substrate. Here we report atomic-resolved characterizations of an exotic 2D ice structure on graphene built from water molecular chains with finite lengths. Our experiments demonstrated that the water molecular chains are exactly orientated along zigzag directions of the graphene substrate, which evidences an anomalously strong interlayer interaction between the 2D ice and the graphene substrate. Moreover, the length of the water molecular chains closely links to the number of graphene layers, indicating layer-number-dependent water-graphene interfacial interactions. Our work highlights the important role of the 2D ice structures on the water-graphene interfacial interactions.

cond-mat.mtrl-sci↗

arXiv2019

Scanning tunneling microscope characterizations of a circular graphene resonator realized with p-p junctions

Using low-temperature high-magnetic-field scanning tunneling microscopy and spectroscopy (STM/STS), we systematically study a graphene quantum dot (GQD) defined by a circular graphene p-p junction. Inside the GQD, we observe a series of quasi-bound states arising from whispering-gallery-mode (WGM) confinement of the circular junction and directly visualize these quasi-bound states down to atomic dimensions. By applying a strong magnetic field, a large jump in energy of the quasi-bound states, which is about one-half the energy spacing between the quasi-bound states, is observed. Such a behavior results from turning on a {\pi} Berry phase of massless Dirac fermions in graphene by a magnetic field. Moreover, our experiment demonstrates that a quasi-bound state splits into two peaks with an energy separation of about 26 meV when the Fermi level crosses the quasi-bound state, indicating that there are strong electron-electron interactions in the GQD.

cond-mat.mes-hall↗

arXiv2015

Sparsest Error Detection via Sparsity Invariant Transformation based $\ell_1$ Minimization

This paper presents a new method, referred to here as the sparsity invariant transformation based $\ell_1$ minimization, to solve the $\ell_0$ minimization problem for an over-determined linear system corrupted by additive sparse errors with arbitrary intensity. Many previous works have shown that $\ell_1$ minimization can be applied to realize sparse error detection in many over-determined linear systems. However, performance of this approach is strongly dependent on the structure of the measurement matrix, which limits application possibility in practical problems. Here, we present a new approach based on transforming the $\ell_0$ minimization problem by a linear transformation that keeps sparsest solutions invariant. We call such a property a sparsity invariant property (SIP), and a linear transformation with SIP is referred to as a sparsity invariant transformation (SIT). We propose the SIT-based $\ell_1$ minimization method by using an SIT in conjunction with $\ell_1$ relaxation on the $\ell_0$ minimization problem. We prove that for any over-determined linear system, there always exists a specific class of SIT's that guarantees a solution to the SIT-based $\ell_1$ minimization is a sparsest-errors solution. Besides, a randomized algorithm based on Monte Carlo simulation is proposed to search for a feasible SIT.

stat.ME↗

arXiv2014

New Accumulative Score Function Based Bound For Sparsity Level of L1 Minimization

This paper discusses a fundamental problem in compressed sensing: the sparse recoverability of L1 minimization with an arbitrary sensing matrix. We develop an new accumulative score function (ASF) to provide a lower bound for the recoverable sparsity level (SL) of a sensing matrix while preserving a low computational complexity. We first define a score function for each row of a matrix, and then ASF sums up large scores until the total score reaches 0.5. Interestingly, the number of involved rows in the summation is a reliable lower bound of SL. It is further proved that ASF provides a sharper bound for SL than coherence We also investigate the underlying relationship between the new ASF and the classical RIC and achieve a RIC-based bound for SL.

math.ST↗

arXiv2014

$l_1$-regularized Outlier Isolation and Regression

This paper proposed a new regression model called $l_1$-regularized outlier isolation and regression (LOIRE) and a fast algorithm based on block coordinate descent to solve this model. Besides, assuming outliers are gross errors following a Bernoulli process, this paper also presented a Bernoulli estimate model which, in theory, should be very accurate and robust due to its complete elimination of affections caused by outliers. Though this Bernoulli estimate is hard to solve, it could be approximately achieved through a process which takes LOIRE as an important intermediate step. As a result, the approximate Bernoulli estimate is a good combination of Bernoulli estimate's accuracy and LOIRE regression's efficiency with several simulations conducted to strongly verify this point. Moreover, LOIRE can be further extended to realize robust rank factorization which is powerful in recovering low-rank component from massive corruptions. Extensive experimental results showed that the proposed method outperforms state-of-the-art methods like RPCA and GoDec in the aspect of computation speed with a competitive performance.

cs.CV↗