Traffic Forecasting with Python: LSTM & Graph Neural Network

Set up the working directory in Google Drive by downloading, unzipping, and uploading the time series forecasting folder, which includes the dataset and code.ipynb for the hands-on project.

Examine the dataset folder containing csv files for traffic speed measurements and route distances to analyze traffic flow, congestion hotspots, and predictive models with spatial dependencies.

Explore how a Jupyter notebook builds a traffic forecasting model with deep learning, focusing on temporal and spatial traffic data, with sections and step-by-step operations in upcoming lectures.

Open the uploaded project from Google Drive by right-clicking the code.ipynb, selecting Google Colaboratory, and running the notebook in Colab to follow along with the traffic forecasting project.

Open the code dot ipynb file in Google Colab and activate the GPU by changing the runtime type to GPU, enabling acceleration for deep learning models in this project.

Mount Google Drive in Google Colab to access files and large data sets, authorize the drive module, persist data across sessions, and integrate with Google Sheets, Docs, and other services.

Import essential libraries for data science and deep learning, including os, pandas, numpy, typing, and matplotlib. Configure the Keras backend to TensorFlow and import layers and Ops for neural networks.

Define the dataset directory by assigning a relative path to data dir to organize project files, improve readability and maintainability, and ensure portable code across environments for collaboration.

Read route distances and speeds data from csv files with pandas read_csv and os.path.join, then convert to numpy arrays for efficient numerical analysis.

Explore how to print and interpret the shapes of numpy arrays for route distances and speeds, using f-strings to verify data loading and aid debugging.

Select a subset of routes by using sample route indices to subset route distances and speeds, reducing data size for focused analysis, visualization, and efficient processing.

Print shapes of subset data arrays to advance data exploration, verify data integrity, and understand dimensionality, while serving as documentation for future reference and debugging in traffic forecasting with Python.

Visualize speed data from speeds array by plotting with a configured figure and legend to compare route zero and route 25, revealing trends, patterns, and anomalies for traffic analysis.

Visualizes the correlation matrix of speed data to reveal relationships between routes, identify groups with similar patterns, and detect data quality issues and outliers for traffic analysis and forecasting.

Preprocess and split data into training, validation, and test sets using train size and val size, then standardize all sets by the training mean and standard deviation.

Explore data preprocessing outcomes for speed data by using a preprocess function to standardize and split into training, validation, and test arrays, then inspect sizes to verify distribution.

Define model parameters such as batch size, input sequence length, and forecast horizon to tailor training and memory use. Enable multi-horizon forecasting or single-step prediction, and optimize performance through experimentation.

Create TensorFlow datasets from NumPy arrays for traffic forecasting by shaping input sequences to a length and forecast horizon, then batch, shuffle, and zip inputs with targets.

Create train, validation, and test TensorFlow datasets from numpy arrays using the create_tf_dataset function with input sequence length, forecast horizon, batch size, and shuffle set to false for test.

Normalize root distances by dividing by 10,000 and square them. Apply a Gaussian with sigma two, threshold at epsilon, and elementwise multiply threshold and mask matrices to remove self connections.

Define a graph info class to encapsulate graph data. Initialize it with edges and num_nodes, store them in self.edges and self.num_nodes, and promote readable, reusable code.

Compute the adjacency matrix for a graph using sigma2 and epsilon, extract edges with numpy's npoi function, and encapsulate nodes and edges in a graph info object for analysis.

Implement a graph convolutional layer as a custom layer using graph info, mean aggregation, and concat combination, with a learned weight matrix initialized by Glorot uniform.

Combine a graph convolutional layer with an LSTM layer to learn spatial and temporal dependencies in graph-based time series forecasting.

Define neural network model parameters for traffic forecasting with Python, including input features, batch size, epochs, input sequence length, LSTM units, graph conv parameters, and forecast horizon.

Initialize an LSTM GC model with 1 input feature, 10 output features, 64 LSTM units, 12-step inputs, and a 3-step forecast horizon, to learn from spatial and temporal dependencies.

Define the model's inputs and outputs with an input layer and shape, including input sequence length, graph num nodes, and node features, using St GCN–LSTM for training and inference.

Compile the Keras model by configuring the optimizer RMSprop with a learning rate of 0.0002 and mean squared error loss to train the traffic forecasting network.

Train the model using the training dataset with input-output pairs, validate after each epoch to monitor generalization and prevent overfitting with early stopping based on validation loss.

Evaluate the trained traffic forecasting model on the test dataset to assess generalization to unseen data, comparing predictions to ground truth with accuracy and error metrics to gauge deployment readiness.

Visualize actual and forecasted values from the model by plotting them on the same graph, with a legend showing actual and forecast lines to compare predictions against ground truth.

Compare mean squared errors between a baseline model and a trained graph convolutional neural network with LSTM for time series forecasting, highlighting better generalization and lower MSE.