arXiv subjects

Zhengyang Wan

Publications and source records attributed to Zhengyang Wan.

4 recordsLinked to original sources

arXiv2026

Beyond the Leaderboard: Counterfactual Diagnosis of End-to-End and VLA Driving Policies Under Domain Shift

End-to-end and vision-language-action (VLA) driving policies are compared by leaderboard rank, but a rank reports an outcome, not the behaviour behind it, so it predicts poorly how a policy will behave at a new site. On six released policies, rank on nuScenes open-loop error or on NAVSIM's leaderboard does not carry over to scenes with a pedestrian near the ego corridor at a new site. We propose a counterfactual check-up: a few hundred real frames, each edited two ways (pedestrian removed, or re-lit by a night-style perturbation), every edit verified by an independent detector, and the change in the planned trajectory read as a diagnosis rather than a score. From these edits two causal axes are read, and five exams built on them separate what a score merges: how far the policy plans to drive, whether seeing the pedestrian buys safety, whether that response scales with danger, whether the plan moves when nothing requires it, and how much an irrelevant lighting change moves it. On 246 NAVSIM near-pedestrian scenes, in the cells where the pedestrian lies on the planned path only 1.9% of responses are genuine avoidance, and under our open-loop protocol the median clearance change is at most 0.03 m and the median change in planned distance at most 0.08 m for every policy. In a pre-registered test from left- to right-hand drive, the exposure and specificity orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not. Read as a selection report, the profiles say which policy is safe because it plans short, which covers a human-like distance without yielding, and which is unsteady under a change that requires no reaction, and they price each verdict: most settle within a few dozen frames, hazard sensitivity needs hundreds. Code and edited frames will be released.

cs.CV↗

arXiv2026

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings. Recent vision-language models (VLMs) offer promising semantic understanding capabilities; however, their high inference latency and susceptibility to hallucination hinder direct application to real-time vehicle control. To address these limitations, this paper proposes DriveVLM-RL, a neuroscience-inspired framework that integrates VLMs into RL through a dual-pathway architecture for safe and deployable autonomous driving. Inspired by the human brain's habitual and deliberative visual processing, DriveVLM-RL decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment via CLIP-based contrasting language goals, and a Dynamic Pathway for attention-gated multi-frame semantic risk reasoning via a lightweight detection model and large VLM (LVLM). A hierarchical reward synthesis mechanism fuses these signals with vehicle state information, while an asynchronous training pipeline decouples expensive LVLM inference from environment interaction. Critically, all VLM components operate exclusively during offline training and are completely removed at deployment, eliminating inference latency at test time. Extensive experiments in the CARLA simulator demonstrate that DriveVLM-RL significantly outperforms state-of-the-art baselines in collision avoidance and task success, attaining the highest success rate while reducing collision severity from 10.09 to 1.75 km/h relative to the strongest VLM-based baseline. The demo video, code, and model checkpoints are available at: https://zilin-huang.github.io/DriveVLM-RL-website/

cs.RO↗

arXiv2026

Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving

Vision-language-model (VLM)-guided reinforcement learning (RL) has recently attracted significant attention for it, replacing brittle hand-crafted rewards with semantically grounded signals; however, deploying such simulation-trained policies on real vehicles remains a fundamental challenge, because they rely on simulator-native observations and simulator-coupled action semantics with no counterpart on physical hardware. We identify a general principle: the simulation-to-reality gap decomposes into two largely orthogonal axes, a sensing-and-dynamics domain gap and a task-and-geometry gap, the former closable without real-world policy training by re-projecting real perception and control onto the policy's training manifold. We formalize this as a transfer guarantee that bounds the deployment gap by three independently controllable error terms, and instantiate it as Sim2Real-AD, which combines a Geometric Observation Bridge, a Physics-Aware Action Mapping, a Two-Phase Progressive Training curriculum, and a Real-time Deployment Pipeline. As a proof of concept, a CARLA-trained VLM-guided RL policy is transferred zero-shot to a full-scale battery-electric Ford E-Transit van in Madison, WI, USA, and drives across car-following, obstacle-avoidance, and stop-sign scenarios using no real-world training data. To our knowledge, this is among the first zero-shot closed-loop deployments of a CARLA-trained VLM-guided RL policy on a full-scale real vehicle, and the decomposition offers a principled, broadly applicable route for moving simulation-trained, foundation-model-guided policies into the physical world, supporting energy-efficient intelligent driving on electrified transportation platforms. The demo video, code, and model checkpoint are available at: https://zilin-huang.github.io/Sim2Real-AD-website/.

cs.RO↗

arXiv2025

Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportation

Recent advances in autonomous system simulation platforms have significantly enhanced the safe and scalable testing of driving policies. However, existing simulators do not yet fully meet the needs of future transportation research-particularly in enabling effective human-AI collaboration and modeling socially-aware driving agents. This paper introduces Sky-Drive, a novel distributed multi-agent simulation platform that addresses these limitations through four key innovations: (a) a distributed architecture for synchronized simulation across multiple terminals; (b) a multi-modal human-in-the-loop framework integrating diverse sensors to collect rich behavioral data; (c) a human-AI collaboration mechanism supporting continuous and adaptive knowledge exchange; and (d) a digital twin framework for constructing high-fidelity virtual replicas of real-world transportation environments. Sky-Drive supports diverse applications such as autonomous vehicle-human road users interaction modeling, human-in-the-loop training, socially-aware reinforcement learning, personalized driving development, and customized scenario generation. Future extensions will incorporate foundation models for context-aware decision support and hardware-in-the-loop testing for real-world validation. By bridging scenario generation, data collection, algorithm training, and hardware integration, Sky-Drive has the potential to become a foundational platform for the next generation of human-centered and socially-aware autonomous transportation systems research. The demo video and code are available at:https://sky-lab-uw.github.io/Sky-Drive-website/

cs.RO↗