Capstone Design Project: Learning-based Autonomous Driving System for a Toyota Corolla
University of Waterloo
Final Year Engineering Capstone Design Project, 2020
Overview
For our Forth Year Capstone Engineering project, we present a learning-based autonomouse lane-keeping system deployable on a stock Toyota Corolla. Working towards the goal of developing a simple autonomous vehicle architeture, the problems of path prediciton, lane detection, path planning and vehicle interfacing are investigated. The end result is a system capable of slowly steering on various urban roads in the Kitchener-Waterloo region with a single monocular camera.
Software Design
- The ROS framework was used to run each component as its own process (node) and to enable messaging between nodes.
- Recording and playback tools from ROS were extremely useful for collecting training data and testing/debugging the stack.
- Neural network models were implemented with Tensorflow as it has extensive tooling for training and deploying models.
- The streetscape.gl toolkit was also modified to serve a 3D visualization of the car's previous and future trajectory.
Hardware and Vehicle Interfacing
The project was to be deployed to a stock 2019 Toyota Corolla Hatchback. A single RGB monocular camera with a custom windshield mount was installed in the vehicle. The system was installed on a standard GIGABYTE based desktop equipped with a NVIDIA 1080Ti graphics card. The desktop was powered by an power inverter connected to the the vehicle battery. Interacing with the vehicle was done through the CAN bus.
Left: The Toyota vehicle used for the project. Middle: The monocular camera installed to the windshield. Right: The desktop and inverter used to power it.
Perception
We used two primary networks for perception. A weakly-supervised Path Prediction network, "NavNet", outputs a cost volume associated with specific paths, while a standard Lane Segmentation network "LaneNet" outputs binary lane masks. NavNet was trained using an imitation learning like pipeline, where pose data in future frames was backprojected in to image space. LaneNet was trained with hand labeled images of our scenes. An outlier rejection algorithm is used to ignore false detections and a tracking model ensures temporal consistency.
Inference results for the two perception networks. Left: NavNet path prediction results. Right: LaneNet lane detection results.
Planner and Visualization
The project was to be deployed to a stock 2019 Toyota Corolla Hatchback. A single RGB monocular camera with a custom windshield mount was installed in the vehicle. The system was installed on a standard GIGABYTE based desktop equipped with a NVIDIA 1080Ti graphics card. The desktop was powered by an power inverter connected to the the vehicle battery. Interacing with the vehicle was done through the CAN bus.
Left: The perception output given a scene. Right: The visualization framework showing both the perception path in blue, and the planned path in green.