ROS2 Self Driving Car with Deep Learning and Computer Vision

Create sign board meshes in Blender, export as obj, and import into Gazebo as models with sdf and config files, understanding Gazebo’s physics and gravity compared to Blender.

Create gazebo models from 3d meshes by joining a sign board to a stand with a fixed joint, fix gravity, and prepare a track for simulation with traffic lights.

Create and export a blender-driven track for gazebo by modeling the circuit with curves and extrusions, adding planes and array copies, and texturing for a ready-to-simulate environment.

Organize gazebo models by creating meshes and materials folders, apply signboard 30 textures (30.png) via material scripts, and configure model.sdf to be static for testing in gazebo.

Configure gazebo model path and the ROS2 shared directory to auto load models, then launch a self-driving car world with a gazebo launch file and place traffic elements.

Diagnose ROS2 Gazebo launch path issues by tracing model and world file directories, using the package share directory, and planning automated copying of models for reliable simulations.

Configure traffic lights for a Gazebo ROS2 self-driving car simulation by modeling red, yellow, and green lamps with cylinders and a light plug-in, with independent timers via a bash script.

Create dynamic traffic light models in Gazebo with a light plugin and an SDF model. Blink green, yellow, and red lights with defined intervals and ambient, diffuse, and specular lighting.

Publish velocity commands via a twist message on the command velocity topic to control the car's linear x and angular z in a 2d environment.

Create a Python ROS2 publisher to publish a command velocity twist message on the cmd_vel topic, drive a Prius in simulation, and configure the node with setup.py and console scripts.

Integrate computer vision with a video save node to process video and publish twist velocities for driving. Set up ROS package and launch a test lane, saving outputs to data/output/new.

Explore how self-driving cars operate without human intervention across five autonomy levels, and the four-step process: mapping, scanning, anticipating, and responding, using sensors to safely reach a destination.

Explore building a self-driving car in simulation using deep neural networks and sensor fusion to follow lanes, navigate intersections, comply with speed limits, and stop at signs.

Break down self-driving process into detection and control, detailing lane lines, sign, and traffic light detection using segmentation, estimation, and data extraction, plus a rules-based controller for steering and speed.

Develop lane-follow capability by detecting lane lines through color-based segmentation, filling gaps with estimation, and training before advancing to the control stage.

Learn color-based lane segmentation by using hue in an appropriate color space to isolate lane lines, while addressing lighting variation and noise with morphology and edge extraction.

Perform color-based lane segmentation for a self-driving car using OpenCV. Convert frames to the HLS space, segment white and yellow regions, apply dilation, and extract mid and outer lane outlines.

Explore estimation techniques to transform boxed patches from middle lane segmentation into an accurate boundary representation, avoiding false positives and refining lane detection in autonomous driving.

Develop a custom OpenCV-based lane trajectory estimator by filtering small contours, merging nearby patches, and selecting the largest contour as the middle trajectory, outperforming the previous line estimate.

The lecture demonstrates a custom estimation algorithm to derive a middle trajectory from color-based segmentation by building and filtering contours, calculating centroids, and connecting contours with minimum distance using OpenCV.

Perform the cleaning stage to refine car lane detection by extracting middle and outer lane contours, correcting predictions, and extending the middle and outline to the image bottom using OpenCV.

Explore data extraction stage 4 theory in lane detection, deriving lane and car directions, computing trajectory points and displacement, and projecting the lane region for display.

We extract information from the lead to reduce testing parameters and define our first rule-based control algorithm named after the feature, to keep the car following the lead, follow live.

Implement the lane assist control system by defining a follow-lead function that estimates the required turn angle from distance and curvature, then clamps to max angles and interpolates for steering.

Object detection works in two sequential steps: localization and classification. Localization marks signs in every frame, then classification identifies the sign, such as a speed sign.

Apply the Hough circles method to localize circular road signs by analyzing edge gradients in grayscale images, using adjustable radius, distance, and vote thresholds to detect circles.

Build a custom sequential cnn with conv layers, max pooling, dropout, flattening, and dense layers multiclass classification, trained with categorical cross entropy and Adam, saved as h5 for sign detection.

Train and evaluate the CNN on the dataset, achieving ninety-five percent accuracy across six categories. Visualize the CNN structure and apply the trained model in the sign detection module.

Explore how optical flow supports tracking in ROS2 self-driving car projects, detailing sparse and dense flows, the Lucas–Kanade method, and displacement vectors across frames.

Implement an optical flow based sign tracker that transitions from detection to tracking, initializes with good features in a region of interest, and uses Lucas parameters to track across frames.

Recognize the left turn indicator and how it guides the self-driving car through a t-junction by taking a left turn. Understand that only one correct path exists for navigation.

Navigate the t-junction using sign detection and mode tracking, execute the left turn with precise speed and frozen angle handling, and verify the computer vision node in simulation.

Understand why detecting traffic lights is essential for safe intersection crossing in self-driving cars, enabling appropriate responses to traffic signals.

Explore why traffic light detection cannot reuse sign detection algorithms, highlighting localization challenges and the need for multiple features and cnn-based separate models for red, green, and yellow.

Learn the theory of haar cascades for traffic light detection, focusing on features, integral images, and cascade classifiers to achieve fast, low-cost object detection on vehicle platforms.

Build a traffic light detection system with Haar cascades: extract frames, balance positive/negative data, annotate positives, and compare annotated versus augmented training results.

Train a traffic light detector with a haar cascade on windows, from gathering training videos to annotating positives, generating samples, and training with OpenCV, then test on video.

Build a traffic light detection pipeline with Haar cascades in Linux, loading the trained cascade, detecting lights in grayscale, and highlighting them with rectangles in a driving simulation.

implements a confirmation and state retrieval mechanism for traffic lights in ROS2 self-driving car project, using color segmentation and circular-region analysis to confirm detection and determine traffic light state.

Assess if tracking suffices for traffic light detection by locating the light and extracting its state via a region of interest, aided by optical flow and mammography estimation.

Develop a traffic light tracker by extending the optical flow tracking class, adding region of interest estimation and collision imminent checks to reliably infer the light state on each frame.

Integrate a traffic light tracker into the detection stage, initialize the yellow tracker after confirmed detection, track region of interest, update traffic light state and proximity, and validate in simulation.

Review the traffic light detection process flow: frames are analyzed, a green heart cascade detects lights, confirm and track with optical flow, extract region of interest, recycle on failure.

Proposes a light-driven control algorithm for a self-driving car, using light state and proximity to switch stop and go modes with speed and angle outputs to cross the intersection.

Develop a rule-based control system for crossing intersections by using traffic light state and proximity data to decide stop, go, and timing, achieving a level three self-driving car capability.

Discover how satnav systems locate a car using an overhead view, localize with multiple satellites, map road networks, compute the shortest route, and enable motion planning for a self-driving car.

Develop a four-stage satnav for a self-driving car, featuring background subtraction localization, occupancy grid creation, graph of interest points, Dijkstra path computation, and a motion planner yielding steering and speed.

Design a satellite navigation integrated self-driving car by implementing SDC v2 with a GPS navigator, upper camera processing, and dynamic engine control, plus enhanced board localization with road network refinement.

Explore stage 1 localization of the sat nav feature in a ros2 self-driving car, covering localization module tweaks, gps navigation integration, and runtime lane segmentation debugging in simulation.

Explore stage two mapping in sat-nav by transforming an occupancy grid into a graph of areas of interest and paths, using user-selected start and destination points.

Modify the SAT NAV stage two mapping to compute start and end points from user input and refine decision points to remove false positives, using closest node and centroid methods.

Learn to fuse sat nav motion planning with self-driving car controls, using satellite imagery to estimate future poses and follow one-minute goals while managing command priorities.

Explores stage four motion planning for ROS2 self-driving cars, removing obstacle navigation, adding a rolling average filter, and implementing a sat nav enable toggle for improved control.