Automating ML Pipelines for Song Recommendation System

The lecture shows how collaborative filtering uses math to compute user similarities with Euclidean, Manhattan, cosine, and Pearson metrics, then predicts ratings by weighting similar users' ratings.

Learn how mean squared error serves as a regression evaluation metric by computing the average of squared differences between actual and predicted values, with a step-by-step example.

Learn how principal component analysis reduces dimensionality by standardizing data, computing a covariance matrix, deriving eigenvalues and eigenvectors, and projecting onto the top components.

Explore classification evaluation metrics, including the confusion matrix for multi-class problems, and learn precision, recall, F1 score, and accuracy calculated from true positives, false positives, false negatives, and diagonal sums.

Set up your conda environment with Anaconda Navigator, install and launch Jupyter, create a data folder, place the dataset, and start a Python 3 ipy kernel for ML experiments.

Import required libraries for data handling, visualization, and machine learning, including pandas, numpy, matplotlib, seaborn, scipy, networkx, and scikit-learn tools like grid search cv and PCA.

Import Trax.csv with pandas read_csv, then describe its numerical features, such as danceability, keys, and loudness, across 7994 instances for a data-driven song recommendation pipeline.

Extract unique genres from the dataset and assess missing data by counting null values per column. Confirm no nulls, then perform feature extraction and visualize the features.

Explore correlation testing with Pearson correlation via a Seaborn heatmap of numerical features, with annotated values and discussion of positive and negative relationships.

Encode categorical genre data and build a user-genre interaction matrix, then apply collaborative filtering with simulated ten users and random genre selections.

Create a cosine similarity based user similarity matrix, build a network with nodes and weighted edges, apply a 0.5 threshold, and visualize a force-directed layout for song recommendations.

Implement a collaborative filtering engine to generate song recommendations by comparing user profiles with cosine similarity, selecting similar users, and returning top-n genre-based songs.

Explore k-means clustering and PCA to group songs into five clusters—high energy dance, mellow acoustic, pop hits, instrumentals, heavy speech—and visualize centroids for a song recommendation pipeline.

Master the docker compose lifecycle by defining services, networks, and volumes for MLflow, Airflow, and Postgres; build, bring up, and bring down containers to orchestrate ML pipelines.

Learn the dockerfile lifecycle, creating a python or airflow image, then build and run containers, using run, copy, expose, and cmd in a /app directory.

Define a docker compose setup to deploy mlflow and airflow microservices. Configure mlflow server on port 5000, postgres storage, volumes, and the airflow mlflow network.

Build a docker image for airflow using python 3.9 and Airflow 2.6.3, install system and library dependencies, copy requirements, install packages, and configure MLflow tracking URI.

Build and compose the microservices for the song recommendation pipeline by containerizing mlflow, airflow, and postgres, mounting dags, logs, datasets, and models, and launching with docker compose up -d.

Orchestrate airflow triggers and mlflow tracking to train, log, and deploy a music recommendation model, capturing metrics, artifacts, and dockerized dag workflows.

Create a load model function that reads a pkl file from a path, uses try and catch, logs errors with Streamlit, and returns the pickle model or none on failure.

Load the collaborative filtering model and tracks data, then build a fallback recommender in Streamlit catch resource. Apply tf-idf with English stop words to song data using fittransform, returning scores.

Build a music recommender UI in streamlit with a multi-select genre dropdown defaulting to pop, a Get recommendations button, validating at least one genre and using cosine similarity for recommendations.

Apply filtering techniques to generate top songs by genre ranked by track popularity, using a lambda to lowercase genres, then join genres and display the top five recommended tracks.

Run the streamlit app locally with streamlit run App.py to test genre-based recommendations, listen to tracks, handle warnings when no genre is selected, then create a GitHub repo and deploy.