Academic Project · Reinforcement Learning
Highway-Env RL
Train an agent to drive on a 3-lane highway without crashing into other cars. We coded a DQN from scratch in PyTorch and compared it to PPO from Stable-Baselines3.
March 2026 course project with Kamelia ADMANTA. Our joint report documents the custom DQN implementation, PPO experiments, and evaluation.
hissein02/Highway-env-project
Trained DQN agent weaving through traffic
The Environment
Observation
5 × 5 matrix
car + 4 nearest vehicles
Actions
5 discrete
left, stay, right, faster, slower
Training
100k steps
highway-fast-v0 (15x speed)
Evaluation
40 vehicles
aggressive, tight spacing
We used highway-v0 from Farama Foundation. The agent gets rewarded for driving fast and staying in the right lanes. If it crashes, it gets -1. Since the environment is random (vehicles spawn in different positions), running the same model twice gives different scores. We report the best results from each experiment.
DQN vs PPO
Two approaches, same task. One we built from scratch, one we used off the shelf.
DQN
Best reported33.40
mean reward (± 8.15), 30 evaluation episodes
Coded from scratch in PyTorch with a 3-layer MLP (25 → 256 → 256 → 5). The final configuration used gamma 0.8 and gradient clipping. In a controlled experiment with the other settings held fixed, lowering gamma from 0.99 to 0.8 increased reward in this environment.
PPO
Plateaued~27.8
mean reward (stuck after ~30k steps)
Used Stable-Baselines3 with 4 parallel environments. The report describes experiments with longer rollouts, an entropy bonus, and different learning rates. Its gamma 0.8 experiment scored 25.29 ± 10.51; the best reported PPO result remained approximately 27.8.
Our final DQN achieved the highest reward in this project: 33.40 ± 8.15, compared with approximately 27.8 for PPO. The controlled gamma experiment confirmed that reducing gamma improved DQN reward. The DQN–PPO comparison describes the configurations we tested in this highway environment.
Source: project report, sections 3.4–5What we took away from this
Gamma matters
A controlled experiment isolated the effect of gamma: reducing it from 0.99 to 0.8 increased DQN reward while keeping the other settings fixed.
Simple model, strongest result
The final plain DQN scored 33.40 ± 8.15, ahead of the Double DQN experiment at 27.28 ± 6.77 and PPO at approximately 27.8. Adding complexity did not improve the results in our experiments.
Context matters
The racetrack bonus used PPO for continuous steering. The report describes qualitative driving behavior but gives no numerical racetrack evaluation.
Project Report
Full write-up with both implementations, hyperparameter experiments, training curves, and the final comparison.