arXiv subjects

Rafael Rafailov

Publications and source records attributed to Rafael Rafailov.

40 records · Page 3Linked to original sources

arXiv2021

Offline Meta-Reinforcement Learning with Advantage Weighting

This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre-training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model to a new task with relatively little data. That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offline, algorithms for offline meta-RL can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. Targeting the offline meta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW), an optimization-based meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offline variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.

cs.LG↗

arXiv2020

Offline Reinforcement Learning from Images with Latent Space Models

Offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. Offline RL enables extensive use and re-use of historical datasets, while also alleviating safety concerns associated with online exploration, thereby expanding the real-world applicability of RL. Most prior work in offline RL has focused on tasks with compact state representations. However, the ability to learn directly from rich observation spaces like images is critical for real-world applications such as robotics. In this work, we build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Model-based offline RL algorithms have achieved state of the art results in state based tasks and have strong theoretical guarantees. However, they rely crucially on the ability to quantify uncertainty in the model predictions, which is particularly challenging with image observations. To overcome this challenge, we propose to learn a latent-state dynamics model, and represent the uncertainty in the latent space. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP. In experiments on a range of challenging image-based locomotion and manipulation tasks, we find that our algorithm significantly outperforms previous offline model-free RL methods as well as state-of-the-art online visual model-based RL methods. Moreover, we also find that our approach excels on an image-based drawer closing task on a real robot using a pre-existing dataset. All results including videos can be found online at https://sites.google.com/view/lompo/ .

cs.LG↗

arXiv2012

On extremums of sums of powered distances to a finite set of points

In this paper we investigate the extremal properties of the sum $$\sum_{i=1}^n|MA_i|^λ,$$ where $A_i$ are vertices of a regular simplex, a cross-polytope (orthoplex) or a cube and $M$ varies on a sphere concentric to the sphere circumscribed around one of the given polytopes. We give full characterization for which points on $Γ$ the extremal values of the sum are obtained in terms of $λ$. In the case of the regular dodecahedron and icosahedron in $\mathbb{R}^3$ we obtain results for which values of $λ$ the corresponding sum is independent of the position of $M$ on $Γ$. We use elementary analytic and purely geometric methods.

math.MG↗

arXiv2011

On the sum of powered distances to certain sets of points on the circle

In this paper we consider an extremal problem in geometry. Let $λ$ be a real number and $A$, $B$ and $C$ be arbitrary points on the unit circle $Γ$. We give full characterization of the extremal behavior of the function $f(M,λ)=MA^λ+MB^λ+MC^λ$, where $M$ is a point on the unit circle as well. We also investigate the extremal behavior of $\sum_{i=1}^nXP_i$, where $P_i, i=1,...,n$ are the vertices of a regular $n$-gon and $X$ is a point on $Γ$, concentric to the circle circumscribed around $P_1...P_n$. We use elementary analytic and purely geometric methods in the proof.

math.MG↗