Reinforcement Learning for Autonomous Driving
Part 2: the research frontier
Why is closed-loop RL not yet the default for driving?
RL can greatly improve performance.
See motivation and evidence from Part 1.
But challenging to get right:
- Reward design. Difficult to capture human driving and trade-offs in decision making.
- Cost. Rendering and model inference.
- Sim-to-real gap.
- Appearance
- Traffic behaviour
- Physics, etc..
- Imitation-to-RL gap.
- No value function from pretraining
- Whole trajectories out, only the first step executed
Examples: Evidence for simpler rewards plus scale
Failure rate by scene difficulty. The gain from RL is largest where the demonstrations are rarest.
BC-SAC. One actor objective, imitation added to SAC:
\[\max_\pi\; \mathbb{E}_{s,a\sim\pi}\big[Q(s,a) + \mathcal{H}(\pi(\cdot\mid s))\big] + \lambda\, \mathbb{E}_{s,a\sim\mathcal{D}}\big[\log \pi(a\mid s)\big]\]
\[R = \min(d_{\text{coll}} - 1,\, 0) + \mathrm{clip}(-1 - d_{\text{edge}},\, -2,\, 0)\]
- The reward has two safety terms and no progress term.
- Trained on the hardest 10% of 100k miles.
Route completion times infraction penalties.
CaRL. Plain PPO, no human data, one reward:
\[r_t = RC_t \prod_i p_{t,i} - T\]
- \(RC_t\) route completion this step, \(p_{t,i} \in [0,1]\) soft penalties, \(T\) a terminal penalty. A collision ends the episode.
- Specifies outcome and not behavior.
- Reward prevents exploits and local optima.
[CaRL] Jaeger et al. CaRL: learning scalable planning policies with simple rewards, 2025
[BC-SAC] Lu et al. Imitation is not enough: robustifying imitation with reinforcement learning for challenging driving scenarios, 2023
Problem: The ego adapts to whoever drives the other cars
| Traffic Model | What breaks | Used by | |
|---|---|---|---|
| Log replay |  |
Many “false positives” (avoidable collisions). | e.g. BC-SAC; nuPlan non-reactive; the uncontrolled agents in GPUDrive and PufferDrive |
| Rule-based |  |
Very predictable, ego overfits (e.g. becomes bully). | e.g. CaRL, Roach, Implicit Affordances; Waymax ships IDM agents |
| Self-play |  |
Conventions that are effective, but alien. | e.g. GigaFlow, HR-PPO, spiced self-play, Gigapixel |
| Learned traffic model |  |
Costly and ego can still overfit. | evaluation only so far, e.g. Hagedorn et al. 2025, BehaviorBench |
[BC-SAC] Lu et al. Imitation is not enough: robustifying imitation with reinforcement learning for challenging driving scenarios, 2023
[nuPlan] Caesar et al. nuPlan: a closed-loop ML-based planning benchmark for autonomous vehicles, 2021
[GPUDrive] Kazemkhani et al. GPUDrive: data-driven, multi-agent driving simulation at 1 million FPS, 2025
[PufferDrive] Cornelisse et al. PufferDrive: a fast and friendly driving simulator for training and evaluating RL agents, 2025
[CaRL] Jaeger et al. CaRL: learning scalable planning policies with simple rewards, 2025
[Roach] Zhang et al. End-to-end urban driving by imitating a reinforcement learning coach, 2021
[Implicit Affordances] Toromanoff et al. End-to-end model-free reinforcement learning for urban driving using implicit affordances, 2020
[Waymax] Gulino et al. Waymax: an accelerated, data-driven simulator for large-scale autonomous driving research, 2023
[GigaFlow] Cusumano-Towner et al. Robust autonomy emerges from self-play, 2025
[HR-PPO] Cornelisse and Vinitsky. Human-compatible driving partners through data-regularized self-play reinforcement learning, 2024
[Spiced self-play] Cornelisse et al. Human-like autonomy emerges from self-play and a pinch of human data, 2026
[Gigapixel] Rowe et al. Scaling self-play for end-to-end driving, 2026
Hagedorn et al. When planners meet reality: how learned, reactive traffic agents shift nuPlan benchmarks, 2025
[BehaviorBench] Distelzweig et al. Beyond self-play and scale: a behavior benchmark for generalization in autonomous driving, 2026
Examples: Making self-play robust and human like
One 6M-parameter self-play policy, no human data, 1.6 billion km.
GigaFlow. Self-play with randomised reward weights per agent.
\[\begin{aligned} R &= R_{\mathrm{goal}} + \textstyle\sum_{k} \alpha_k R_k \\ \alpha_k &\sim U(\cdot) \text{ per agent and episode}, \qquad \pi(a \mid o, \alpha) \end{aligned}\]
\(k\): collision, off-road, comfort, lane, reverse and stop line.
Safe task completion against human data used. Thirty minutes as an anchor beats imitation on 52 days.
Spiced self-play. Mix self-play with thirty minutes of human logs.
\[\begin{aligned} \phi &= \arg\min_\phi \textstyle\sum_{(o, a) \in \mathcal{D}_{30\,\mathrm{min}}} -\log \tau_\phi(a \mid o) \\ \mathcal{L}(\theta) &= \mathcal{L}_{\mathrm{PPO}}(\theta) + \lambda\, \mathbb{E}_{o \sim \rho_{\pi_\theta}}\!\left[ D_{\mathrm{KL}}\!\left(\tau_\phi(\cdot \mid o) \,\|\, \pi_\theta(\cdot \mid o)\right) \right] \end{aligned}\]
Train anchor \(\tau_\phi\) on thirty minutes of human logs, then freeze it.
Very simple rewards: \(+1\) at the goal, \(-1\) on collision or off-road.
[GigaFlow] Cusumano-Towner et al. Robust autonomy emerges from self-play, 2025
[Spiced self-play] Cornelisse et al. Human-like autonomy emerges from self-play and a pinch of human data, 2026
[SMART] Wu et al. SMART: scalable multi-agent real-time motion generation via next-token prediction, 2024
Examples: Two efficient ways for close(ish) loop training
Multiple plans from a logged frame, rewards against the logged future.
Open-loop RL. Score plans on logged states.
- Start from a logged state and propose multiple plans.
- Score each plan against the logged future (no replanning!).
- Train plans with GRPO.
- e.g. ReCogDrive, Alpamayo-R1, AutoVLA, LCDrive and TrajHF.
- Trains the AR plan rollout in “closed loop”.
Distill BEV RL teacher into camera student.
Closed-loop SFT (DAgger) from an RL teacher. Roach’s recipe:
- Train RL teacher (PPO) on BEV.
- Train student with DAgger from RL teacher.
- Distillation also uses latents and values as auxiliary targets.
- Gigapixel (2026) scales this: self-play teacher, rendered-pixel student, perception adapter to real sensors.
[ReCogDrive] Li et al. ReCogDrive: a reinforced cognitive framework for end-to-end autonomous driving, 2025
[PDMS] Dauner et al. NAVSIM: data-driven non-reactive autonomous vehicle simulation and benchmarking, 2024
[Gigapixel] Rowe et al. Scaling self-play for end-to-end driving, 2026
[Roach] Zhang et al. End-to-end urban driving by imitating a reinforcement learning coach, 2021
[RoaD] Garcia-Cobo et al. RoaD: rollouts as demonstrations for closed-loop supervised fine-tuning of autonomous driving policies, 2025
[LEAD] Nguyen et al. LEAD: minimizing learner-expert asymmetry in end-to-end driving, 2026
Examples: Closed-loop RL of a pretrained sensor policy
Combining PPO training on 3968 scenes with continuous IL training.
RAD. Fine-tunes an E2E policy with PPO.
- Camera policy pretrained on 2,000h.
- Fine-tuned with PPO on 3,968 reconstructed 3DGS scenes.
- Collision ratio 0.270 → 0.089 on 337 held-out scenes.
- Co-training with imitation on human demonstrations.
- Constructs dense rewards.
Unfortunately not the official logos.
Some soon-to-be released papers from our group on fine-tuning Alpamayo in AlpaSim.
- Tricks for fine-tuning Alpamayo with open-loop RL.
- Using an RL teacher as Oracle for DAgger-style closed-loop SFT.
- Large-scale training with AlpaGym.
[RAD] Gao et al. RAD: training an end-to-end driving policy via large-scale 3DGS-based reinforcement learning, 2025
[BC-SAC] Lu et al. Imitation is not enough: robustifying imitation with reinforcement learning for challenging driving scenarios, 2023
[Spiced self-play] Cornelisse et al. Human-like autonomy emerges from self-play and a pinch of human data, 2026
[CLEAR] Shi et al. CLEAR: closed-loop reinforcement learning at scale for end-to-end autonomous driving, 2026
Noguchi and Yamamoto. Pseudo-expert regularized offline RL for end-to-end autonomous driving in photorealistic closed-loop environments, 2025
[Raw2Drive] Yang et al. Raw2Drive: reinforcement learning with aligned world models for end-to-end autonomous driving (in CARLA v2), 2025
RL on reasoning traces uses the open-loop RL
Three rewards: a large reasoning model grades the trace, a binary check that the trace’s decision matches the trajectory, and the distance to the logged trajectory as behavior reward.
| RL rewards | Reasoning grade | Reasoning-action consistency | Close encounters, log replay |
|---|---|---|---|
| SFT only | 3.1 | 0.62 | 6.9% |
| plus reasoning grade | 4.5 | 0.53 | 5.8% |
| plus consistency | 4.5 | 0.85 | 6.2% |
| plus trajectory reward | 4.4 | 0.83 | 3.7% |
[Alpamayo-R1] NVIDIA. Alpamayo-R1: bridging reasoning and action prediction for generalizable autonomous driving in the long tail, 2025