Can an RL agent steer the car as well as the controller I already wrote?
I already had a hand-coded controller that could drive the track. The question was whether a learned policy could match it. To keep the problem tractable I only let the agent control steering, one number per step, with throttle handled separately by a proportional controller. Six runs of Soft Actor-Critic later I had six collapses and a much better understanding of why they happened. That understanding is most of what this writeup is about.
First I needed something that could already drive
Random initialization in a racing environment crashes the car within a second of every episode, which makes early learning painfully slow. So before touching an RL algorithm I built a reactive controller by hand. It clusters the LiDAR point cloud in O(n) by exploiting the angular scan order of the rotating sensor, then steers off the mean lateral cone position. It completes laps. It also weaves badly on straights, because it has no idea which boundary is blue and which is yellow.
Adding a forward-facing camera fixed that. A YOLO detector trained on the FSOCO dataset classifies cone color, a pinhole ground-plane projection turns pixel detections into metric vehicle-frame coordinates, and nearest-neighbor matching puts that color label onto the LiDAR depth measurement. That perception work is its own project and I wrote it up separately in the LiDAR-camera fusion report. What matters here is that the fused stack does two jobs: it generates ground truth for the RL environment, and it acts as the demonstrator for behavior cloning.
The environment exposes a 41-dimensional observation. Six nearest blue cones and six nearest yellow cones as (x, y, color) makes 36 values, plus forward speed, lateral speed, yaw rate, the previous steering command, and target speed. Six cones a side gives roughly 15 m of look-ahead at racing speed, which is enough to see one full corner coming. The action is a single scalar steering command, so the agent only ever has to learn one thing.
ClockSpeed up to 3x and killing the render viewport got the step rate past 1,200 per second, roughly 40 times baseline. There is a ceiling on that trick: push the clock higher and the physics timestep grows enough to start breaking collision detection. Two undocumented bugs cost me several days on top of it. The car sometimes spawned with its wheels effectively locked, which a mandatory 2 second settle pause after reset fixed, and it silently ignored throttle because it spawned in neutral gear. I found both by watching the car refuse to move and working backwards.
Giving the agent something to copy
Since random exploration crashes immediately, I recorded the hand-coded controller driving laps and trained a policy to imitate it by supervised regression. That gave SAC a starting point that at least stayed on the track. Worth saying up front that this crutch is a SAC-era thing: PPO later trained from scratch on ground-truth cone positions, with none of this demonstration data. The demonstrator needed iterating of its own before it was worth imitating. A Gaussian-weighted lookahead caused phase lag at high loop rates. A geometric pure-pursuit variant ran wide at corner exits. Lowering the speed target made things worse rather than better, because it weakened the proportional correction.
What ended up working was the simplest version I tried. Take the midpoint of the three nearest cones on each side, apply a proportional correction with a softer divisor. It clips about one cone per lap. Not a great teacher, but good enough to bootstrap from.
Six SAC runs, and every one of them fell apart
With the environment validated and a warm-start policy ready, I trained with Soft Actor-Critic. It is an off-policy actor-critic method built for continuous control, so it was the obvious first choice. Every run that looked like progress turned out to have something wrong underneath it.
Two reward bugs cost real time before any of that. Throttle was hardcoded at 40 percent, which let the car reach 17 m/s before it had learned anything useful. And the centerline offset was signed, but the reward consumed it without taking an absolute value, so the agent was getting paid to drift left. That one looked exactly like a tuning problem for a while.
These were properties of the algorithm, not bad tuning
The same three problems kept showing up across tests 9 through 13, and none of them were things I could tune my way out of.
Put together, almost any SAC run was one bad batch or one reward tweak away from falling over, and no amount of tuning was going to change that.
Trading speed for runs that actually finish
After five collapses across genuinely different configurations I stopped trying to tune my way out. The problem was structural, so I switched to Proximal Policy Optimization, which avoids all three of the issues above by construction rather than by careful hyperparameters.
PPO also does without the crutch. SAC needed a behavior-cloned policy to start from, because dropping it into the environment cold meant crashing inside a second. PPO learns the track from scratch, on ground truth and the reward alone, with none of the demonstration data the SAC runs leaned on.
The tradeoff is sample efficiency. PPO wants somewhere around 300k to 500k environment interactions where SAC needed 50k to 100k, which works out to 8 or 10 hours of wall clock instead of 1 or 2 at the 12 to 15 steps per second this setup manages. I will take that trade. A SAC run that collapses leaves me with nothing to evaluate. A slow PPO run that finishes leaves me with a model I can actually measure and improve.
Three reference models and a training setup that works
[TERM] line is an episode ending and says why, mostly cone_hit or geofence, with the lateral error it held. Rolling reward sits around +820 and you can watch it save a new best checkpoint mid-run.None of the three SAC checkpoints are competition ready. The BC baseline, the Test 9.6 model and the Test 13 step-5000 checkpoint are all useful as reference points and not much more than that. The part that lasts is the infrastructure. An environment wrapper that behaves, a reward signal I have actually verified, a working speed-curriculum callback, telemetry that does not fall over mid-run. The expensive debugging is finished. The wheel-lock spawn, the gear flag, the centerline sign error, the lap counter, the corner distance metric, none of it needs doing twice.
PPO runs on this same environment, but it learns on its own. No cloned demonstrations, no warm start, just ground-truth cone positions and the reward. What I want next is a clean wall-clock-to-baseline benchmark, and I would like to know whether SAC's sample efficiency advantage costs anything in cross-track generalization. Whether residual RL on top of the reactive controller transfers cleanly is also still open, as is how much domain randomization closes the cone-detection gap between simulation and a real car.
What I would tell someone starting this
Almost all of the cost here was infrastructure correctness and picking an algorithm that fits the problem. Very little of it was model design. SAC's collapses came from off-policy replay, automatic entropy tuning, and a critic tied to one reward shape, and PPO deals with all three. I am writing up the failures next to the results because those failure signatures are the part I would have wanted to read before I started.