arXiv subjects

Hao Hu

Publications and source records attributed to Hao Hu.

At least 127 records · Page 7Linked to original sources

arXiv2023

The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning

Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcement learning (RL) in a principled way remains unclear. In this paper, we address this issue by investigating the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. Further, we propose a novel, Provable Data Sharing algorithm (PDS) to utilize such reward-free data for offline RL. PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm. Our results on various offline RL tasks demonstrate that PDS significantly improves the performance of offline RL algorithms with reward-free data. Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees. We believe our findings will contribute to developing more robust self-supervised RL methods.

cs.LG↗

arXiv2022

Universal Critical Behavior of Percolation in Orientationally Ordered Janus Particles and Other Anisotropic Systems

We combine percolation theory and Monte Carlo simulation to study in two dimensions the connectivity of an equilibrium lattice model of interacting Janus disks which self-assemble into an orientationally ordered stripe phase at low temperature. As the patch size is increased or the temperature is lowered, clusters of patch-connected disks grow, and a percolating cluster emerges at a threshold. In the stripe phase, the critical clusters extend longer in the direction parallel to the stripes than in the perpendicular direction, and percolation is thus anisotropic. It is found that the critical behavior of percolation in the Janus system is consistent with that of standard isotropic percolation, when an appropriate spatial rescaling is made. The rescaling procedure can be applied to understand other anisotropic systems, such as the percolation of aligned rigid rods and of the $q$-state Potts model with anisotropic interactions.

cond-mat.soft↗

arXiv2022

Low-velocity-favored transition radiation

When a charged particle penetrates through an optical interface, photon emissions emerge - a phenomenon known as transition radiation. Being paramount to fundamental physics, transition radiation has enabled many applications from high-energy particle identification to novel light sources. A rule of thumb in transition radiation is that the radiation intensity generally decreases with the particle velocity v; as a result, low-energy particles are not favored in practice. Here we find that there exist situations where transition radiation from particles with extremely low velocities (e.g. v/c<0.001) exhibits comparable intensity as that from high-energy particles (e.g. v/c=0.999), where c is light speed in free space. The comparable radiation intensity implies an extremely high photon extraction efficiency from low-energy particles, up to eight orders of magnitude larger than that from high-energy particles. This exotic phenomenon of low-velocity-favored transition radiation originates from the excitation of Ferrell-Berreman modes in epsilon-near-zero materials. Our findings may provide a promising route towards the design of integrated light sources based on low-energy electrons and specialized detectors for beyond-standard-model particles.

physics.optics↗

arXiv2022

Improving Accuracy Without Losing Interpretability: A ML Approach for Time Series Forecasting

In time series forecasting, decomposition-based algorithms break aggregate data into meaningful components and are therefore appreciated for their particular advantages in interpretability. Recent algorithms often combine machine learning (hereafter ML) methodology with decomposition to improve prediction accuracy. However, incorporating ML is generally considered to sacrifice interpretability inevitably. In addition, existing hybrid algorithms usually rely on theoretical models with statistical assumptions and focus only on the accuracy of aggregate predictions, and thus suffer from accuracy problems, especially in component estimates. In response to the above issues, this research explores the possibility of improving accuracy without losing interpretability in time series forecasting. We first quantitatively define interpretability for data-driven forecasts and systematically review the existing forecasting algorithms from the perspective of interpretability. Accordingly, we propose the W-R algorithm, a hybrid algorithm that combines decomposition and ML from a novel perspective. Specifically, the W-R algorithm replaces the standard additive combination function with a weighted variant and uses ML to modify the estimates of all components simultaneously. We mathematically analyze the theoretical basis of the algorithm and validate its performance through extensive numerical experiments. In general, the W-R algorithm outperforms all decomposition-based and ML benchmarks. Based on P50_QL, the algorithm relatively improves by 8.76% in accuracy on the practical sales forecasts of JD.com and 77.99% on a public dataset of electricity loads. This research offers an innovative perspective to combine the statistical and ML algorithms, and JD.com has implemented the W-R algorithm to make accurate sales predictions and guide its marketing activities.

cs.LG↗

arXiv2022

Reinforcement Learning based Voice Interaction to Clear Path for Robots in Elevator Environment

Efficient use of the space in an elevator is very necessary for a service robot, due to the need for reducing the amount of time caused by waiting for the next elevator. To provide a solution for this, we propose a hybrid approach that combines reinforcement learning (RL) with voice interaction for robot navigation in the scene of entering the elevator. RL provides robots with a high exploration ability to find a new clear path to enter the elevator compared to traditional navigation methods such as Optimal Reciprocal Collision Avoidance (ORCA). The proposed method allows the robot to take an active clear path action towards the elevator whilst a crowd of people stands at the entrance of the elevator wherein there are still lots of space. This is done by embedding a clear path action (voice prompt) into the RL framework, and the proposed navigation policy helps the robot to finish tasks efficiently and safely. Our model approach provides a great improvement in the success rate and reward of entering the elevator compared to state-of-the-art navigation policies without active clear path operation.

cs.RO↗

arXiv2022

Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery

Offline reinforcement learning (RL) enables the agent to effectively learn from logged data, which significantly extends the applicability of RL algorithms in real-world scenarios where exploration can be expensive or unsafe. Previous works have shown that extracting primitive skills from the recurring and temporally extended structures in the logged data yields better learning. However, these methods suffer greatly when the primitives have limited representation ability to recover the original policy space, especially in offline settings. In this paper, we give a quantitative characterization of the performance of offline hierarchical learning and highlight the importance of learning lossless primitives. To this end, we propose to use a \emph{flow}-based structure as the representation for low-level policies. This allows us to represent the behaviors in the dataset faithfully while keeping the expression ability to recover the whole policy space. We show that such lossless primitives can drastically improve the performance of hierarchical policies. The experimental results and extensive ablation studies on the standard D4RL benchmark show that our method has a good representation ability for policies and achieves superior performance in most tasks.

cs.LG↗

arXiv2022

Robust Interior Point Method for Quantum Key Distribution Rate Computation

Security proof methods for quantum key distribution, QKD, that are based on the numerical key rate calculation problem, are powerful in principle. However, the practicality of the methods are limited by computational resources and the efficiency and accuracy of the underlying algorithms for convex optimization. We derive a stable reformulation of the convex nonlinear semidefinite programming, SDP, model for the key rate calculation problems. We use this to develop an efficient, accurate algorithm. The stable reformulation is based on novel forms of facial reduction, FR, for both the linear constraints and nonlinear quantum relative entropy objective function. This allows for a Gauss-Newton type interior-point approach that avoids the need for perturbations to obtain strict feasibility, a technique currently used in the literature. The result is high accuracy solutions with theoretically proven lower bounds for the original QKD from the FR stable reformulation. This provides novel contributions for FR for general SDP. We report on empirical results that dramatically improve on speed and accuracy, as well as solving previously intractable problems.

quant-ph↗

arXiv2022

On the Role of Discount Factor in Offline Reinforcement Learning

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. The discount factor, $γ$, plays a vital role in improving online RL sample efficiency and estimation accuracy, but the role of the discount factor in offline RL is not well explored. This paper examines two distinct effects of $γ$ in offline RL with theoretical analysis, namely the regularization effect and the pessimism effect. On the one hand, $γ$ is a regulator to trade-off optimality with sample efficiency upon existing offline techniques. On the other hand, lower guidance $γ$ can also be seen as a way of pessimism where we optimize the policy's performance in the worst possible models. We empirically verify the above theoretical observation with tabular MDPs and standard D4RL tasks. The results show that the discount factor plays an essential role in the performance of offline RL algorithms, both under small data regimes upon existing offline methods and in large data regimes without other conservative methods.

cs.LG↗

arXiv2022

Optimizing Binary Decision Diagrams with MaxSAT for classification

The growing interest in explainable artificial intelligence (XAI) for critical decision making motivates the need for interpretable machine learning (ML) models. In fact, due to their structure (especially with small sizes), these models are inherently understandable by humans. Recently, several exact methods for computing such models are proposed to overcome weaknesses of traditional heuristic methods by providing more compact models or better prediction quality. Despite their compressed representation of Boolean functions, Binary decision diagrams (BDDs) did not gain enough interest as other interpretable ML models. In this paper, we first propose SAT-based models for learning optimal BDDs (in terms of the number of features) that classify all input examples. Then, we lift the encoding to a MaxSAT model to learn optimal BDDs in limited depths, that maximize the number of examples correctly classified. Finally, we tackle the fragmentation problem by introducing a method to merge compatible subtrees for the BDDs found via the MaxSAT model. Our empirical study shows clear benefits of the proposed approach in terms of prediction quality and intrepretability (i.e., lighter size) compared to the state-of-the-art approaches.

cs.AI↗

arXiv2022

Observation of 2D Cherenkov radiation

For over 80 years of research, the conventional description of free-electron radiation phenomena, such as Cherenkov radiation, has remained unchanged: classical three-dimensional electromagnetic waves. Interestingly, in reduced dimensionality, the properties of free-electron radiation are predicted to fundamentally change. Here, we present the first observation of Cherenkov surface waves, wherein free electrons emit narrow-bandwidth photonic quasiparticles propagating in two-dimensions. The low dimensionality and narrow bandwidth of the effect enable to identify quantized emission events through electron energy loss spectroscopy. Our results support the recent theoretical prediction that free electrons do not always emit classical light and can instead become entangled with the photons they emit. The two-dimensional Cherenkov interaction achieves quantum coupling strengths over two orders of magnitude larger than ever reported, reaching the single-electron-single-photon interaction regime for the first time with free electrons. Our findings pave the way to previously unexplored phenomena in free-electron quantum optics, facilitating bright, free-electron-based quantum emitters of heralded Fock states.

quant-ph↗

arXiv2022

Facial Reduction for Symmetry Reduced Semidefinite Doubly Nonnegative Programs

We consider both facial reduction, \FRp, and symmetry reduction, \SRp, techniques for semidefinite programming, \SDPp. We show that the two together fit surprisingly well in an alternating direction method of multipliers, \ADMMp, approach. In fact, this approach allows for simply adding on nonnegativity constraints, and solving the doubly nonnegative, \DNN, relaxation of many classes of hard combinatorial problems. We also show that the singularity degree remains the same after \SRp, and that the \DNN relaxations considered here have singularity degree one, that is reduced to zero after \FRp. The combination of \FR and \SR leads to a significant improvement in both numerical stability and running time for both the \ADMM and interior point approaches. We test our method on various \DNN relaxations of hard combinatorial problems including quadratic assignment problems with sizes of more than $n=500$. This translates to a semidefinite constraint of order $250,000$ and $625\times 10^8$ nonnegative constrained variables, before applying the reduction techniques.

math.OC↗

arXiv2022

On the Estimation Bias in Double Q-Learning

Double Q-learning is a classical method for reducing overestimation bias, which is caused by taking maximum estimated values in the Bellman operation. Its variants in the deep Q-learning paradigm have shown great promise in producing reliable value prediction and improving learning performance. However, as shown by prior work, double Q-learning is not fully unbiased and suffers from underestimation bias. In this paper, we show that such underestimation bias may lead to multiple non-optimal fixed points under an approximate Bellman operator. To address the concerns of converging to non-optimal stationary solutions, we propose a simple but effective approach as a partial fix for the underestimation bias in double Q-learning. This approach leverages an approximate dynamic programming to bound the target value. We extensively evaluate our proposed method in the Atari benchmark tasks and demonstrate its significant improvement over baseline algorithms.

cs.LG↗

arXiv2021

Percolation thresholds of randomly rotating patchy particles on Archimedean lattices

We study the percolation of randomly rotating patchy particles on $11$ Archimedean lattices in two dimensions. Each vertex of the lattice is occupied by a particle, and in each model the patch size and number are monodisperse. When there are more than one patches on the surface of a particle, they are symmetrically decorated. As the proportion $χ$ of the particle surface covered by the patches increases, the clusters connected by the patches grow and the system percolates at the threshold $χ_c$. We combine Monte Carlo simulations and the critical polynomial method to give precise estimates of $χ_c$ for disks with one to six patches and spheres with one to two patches on the $11$ lattices. For one-patch particles, we find that the order of $χ_c$ values for particles on different lattices is the same as that of threshold values $p_c$ for site percolation on same lattices, which implies that $χ_c$ for one-patch particles mainly depends on the geometry of lattices. For particles with more patches, symmetry become very important in determining $χ_c$. With the estimates of $χ_c$ for disks with one to six patches, by analyses related to symmetry, we are able to give precise values of $χ_c$ for disks with an arbitrary number of patches on all $11$ lattices. The following rules are found for patchy disks on each of these lattices: (i) as the number of patches $n$ increases, values of $χ_c$ repeat in a periodic way, with the period $n_0$ determined by the symmetry of the lattice; (ii) when $\mod(n,n_0)=0$, the minimum threshold value $χ_{\rm min}$ appears, and the model is equivalent to site percolation with $χ_{\rm min}=p_c$; (iii) disks with $\mod(n,n_0)=m$ and $n_0-m$ ($m<n_0/2$) share the same $χ_c$ value.

cond-mat.stat-mech↗

arXiv2021

MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration

Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward tasks, as it requires quickly finding informative task-relevant experiences in both meta-training and adaptation. To address this challenge, we explicitly model an exploration policy learning problem for meta-RL, which is separated from exploitation policy learning, and introduce a novel empowerment-driven exploration objective, which aims to maximize information gain for task identification. We derive a corresponding intrinsic reward and develop a new off-policy meta-RL framework, which efficiently learns separate context-aware exploration and exploitation policies by sharing the knowledge of task inference. Experimental evaluation shows that our meta-RL method significantly outperforms state-of-the-art baselines on various sparse-reward MuJoCo locomotion tasks and more complex sparse-reward Meta-World tasks.

cs.AI↗

arXiv2021

De Sitter braneworld and gravitational waves

We study the braneworld theory constructed by multi scalar fields. The model contains a smooth and infinitely large extra dimension, allowing the background fields propagating in it. We give a de Sitter solution for the four-dimensional cosmology as a good approximation to the early universe inflation. We show that the graviton has a localizable massless mode, and a series of continuous massive modes, separated by a mass gap. There could be a normalizable massive mode, depending on the background solution. The gravitational waves of massless mode evolve the same as the four dimensional theory, while that of the massive modes evolve greatly different from the massless mode.

gr-qc↗

arXiv2021

Offline Reinforcement Learning with Value-based Episodic Memory

Offline reinforcement learning (RL) shows promise of applying RL to real-world problems by effectively utilizing previously collected data. Most existing offline RL algorithms use regularization or constraints to suppress extrapolation error for actions outside the dataset. In this paper, we adopt a different framework, which learns the V-function instead of the Q-function to naturally keep the learning procedure within the support of an offline dataset. To enable effective generalization while maintaining proper conservatism in offline learning, we propose Expectile V-Learning (EVL), which smoothly interpolates between the optimal value learning and behavior cloning. Further, we introduce implicit planning along offline trajectories to enhance learned V-values and accelerate convergence. Together, we present a new offline method called Value-based Episodic Memory (VEM). We provide theoretical analysis for the convergence properties of our proposed VEM method, and empirical results in the D4RL benchmark show that our method achieves superior performance in most tasks, particularly in sparse-reward tasks.

cs.LG↗

arXiv2021

Free-electron radiation engineering via structured environments

Free-electron radiation results from the interaction between swift electrons and the local electromagnetic environment. Recent advances in material technologies provide powerful tools to control light emission from free electrons and may facilitate many intriguing applications of free-electron radiation in particle detections, lasers, quantum information processing, etc. Here, we provide a brief overview on the recent theoretical developments and experimental observations of spontaneous free-electron radiation in various structured environments, including two-dimensional materials, metasurfaces, metamaterials, and photonic crystals. We also report on research progresses on the stimulated free-electron radiation that results from the interaction between free electrons and photonic quasi-particles induced by the external field. Moreover, we provide an outlook of potential research directions for this vigorous realm of free-electron radiation.

physics.optics↗

arXiv2021

A Brewster route to Cherenkov detectors

The Cherenkov effect enables a valuable tool, known as the Cherenkov detector, to identify high-energy particles via the measurement of the Cherenkov cone. However, the sensitivity and momentum coverage of such detectors are intrinsically limited by the refractive index of the host material. Especially, identifying particles with energy above multiple gigaelectronvolts requires host materials with a near-unity refractive index, which are often limited to large and bulky gas chambers. Overcoming this fundamental material limit is important for future particle detectors yet remains a long-standing scientific challenge. Here, we propose a different paradigm for Cherenkov detectors that utilizes the broadband angular filter made from stacks of variable one-dimensional photonic crystals. Owing to the Brewster effect, the angular filter is transparent only to Cherenkov photons from a precise incident angle, and particle identification is achieved by mapping each Cherenkov angle to the peak-intensity position of transmitted photons in the detection plane. This unique property of the angular filter is exceptionally beneficial to Cherenkov detection as it enables the realization of a non-dispersive pseudo refractive index over the entire visible spectrum. Moreover, such a pseudo refractive index can be flexibly tuned to arbitrary values, including those close to unity. Our angular-selective Brewster paradigm offers a feasible solution to implement compact and highly sensitive Cherenkov detectors especially in beam lines and it can cover a wide momentum range using readily available dielectric materials.

physics.optics↗