Skip to main content
Back to the lab

Academic Project · Reinforcement Learning

Highway-Env RL

Train an agent to drive on a 3-lane highway without crashing into other cars. We coded a DQN from scratch in PyTorch and compared it to PPO from Stable-Baselines3.

PyTorchDQNPPOStable-Baselines3GymnasiumTensorBoard

March 2026 course project with Kamelia ADMANTA. Our joint report documents the custom DQN implementation, PPO experiments, and evaluation.

hissein02/Highway-env-project
Trained DQN agent driving on the highway

Trained DQN agent weaving through traffic

The Environment

Observation

5 × 5 matrix

car + 4 nearest vehicles

Actions

5 discrete

left, stay, right, faster, slower

Training

100k steps

highway-fast-v0 (15x speed)

Evaluation

40 vehicles

aggressive, tight spacing

We used highway-v0 from Farama Foundation. The agent gets rewarded for driving fast and staying in the right lanes. If it crashes, it gets -1. Since the environment is random (vehicles spawn in different positions), running the same model twice gives different scores. We report the best results from each experiment.

DQN vs PPO

Two approaches, same task. One we built from scratch, one we used off the shelf.

DQN

Best reported

33.40

mean reward (± 8.15), 30 evaluation episodes

Coded from scratch in PyTorch with a 3-layer MLP (25 → 256 → 256 → 5). The final configuration used gamma 0.8 and gradient clipping. In a controlled experiment with the other settings held fixed, lowering gamma from 0.99 to 0.8 increased reward in this environment.

PPO

Plateaued

~27.8

mean reward (stuck after ~30k steps)

Used Stable-Baselines3 with 4 parallel environments. The report describes experiments with longer rollouts, an entropy bonus, and different learning rates. Its gamma 0.8 experiment scored 25.29 ± 10.51; the best reported PPO result remained approximately 27.8.

Our final DQN achieved the highest reward in this project: 33.40 ± 8.15, compared with approximately 27.8 for PPO. The controlled gamma experiment confirmed that reducing gamma improved DQN reward. The DQN–PPO comparison describes the configurations we tested in this highway environment.

Source: project report, sections 3.4–5

What we took away from this

Gamma matters

A controlled experiment isolated the effect of gamma: reducing it from 0.99 to 0.8 increased DQN reward while keeping the other settings fixed.

Simple model, strongest result

The final plain DQN scored 33.40 ± 8.15, ahead of the Double DQN experiment at 27.28 ± 6.77 and PPO at approximately 27.8. Adding complexity did not improve the results in our experiments.

Context matters

The racetrack bonus used PPO for continuous steering. The report describes qualitative driving behavior but gives no numerical racetrack evaluation.

Project Report

Full write-up with both implementations, hyperparameter experiments, training curves, and the final comparison.