Automating ML Pipelines for Song Recommendation System

The lecture shows how collaborative filtering uses math to compute user similarities with Euclidean, Manhattan, cosine, and Pearson metrics, then predicts ratings by weighting similar users' ratings.

Explore the k nearest neighbor algorithm for classification and regression, using euclidean distance to compute nearest neighbors, select k, then apply majority vote for classification or average for regression.

Explore the Naive Bayes classifier and Bayes theorem, deriving priors, likelihoods, and posteriors to classify data such as spam versus not spam.

Explore term frequency and inverse document frequency, learn to compute tf, idf, and tf-idf across a document corpus, and see how word distribution is measured.

Explore cosine similarity to measure the angle between two vectors and compare documents or nodes. Compute the dot product and magnitudes, then use the Euclidean norms for the final similarity.

Learn how mean squared error serves as a regression evaluation metric by computing the average of squared differences between actual and predicted values, with a step-by-step example.

Explore k-means clustering, the unsupervised method for grouping data. Initialize random centroids, assign points to the nearest centroid, recompute centroids as means, and iterate until convergence.

Learn how principal component analysis reduces dimensionality by standardizing data, computing a covariance matrix, deriving eigenvalues and eigenvectors, and projecting onto the top components.

Explore classification evaluation metrics, including the confusion matrix for multi-class problems, and learn precision, recall, F1 score, and accuracy calculated from true positives, false positives, false negatives, and diagonal sums.

Set up your conda environment with Anaconda Navigator, install and launch Jupyter, create a data folder, place the dataset, and start a Python 3 ipy kernel for ML experiments.

Import required libraries for data handling, visualization, and machine learning, including pandas, numpy, matplotlib, seaborn, scipy, networkx, and scikit-learn tools like grid search cv and PCA.

Import Trax.csv with pandas read_csv, then describe its numerical features, such as danceability, keys, and loudness, across 7994 instances for a data-driven song recommendation pipeline.

Extract unique genres from the dataset and assess missing data by counting null values per column. Confirm no nulls, then perform feature extraction and visualize the features.

Identify and extract numerical features, visualize their distributions with histograms, and interpret key song attributes—danceability, energy, loudness, speechiness, acoustic noise, instrumental noise, valence, and tempo.

Explore correlation testing with Pearson correlation via a Seaborn heatmap of numerical features, with annotated values and discussion of positive and negative relationships.

Perform a t-test comparing danceability between rock and pop genres, report the t statistic and p value, and show significantly higher danceability for pop songs.

Encode categorical genre data and build a user-genre interaction matrix, then apply collaborative filtering with simulated ten users and random genre selections.

Create a cosine similarity based user similarity matrix, build a network with nodes and weighted edges, apply a 0.5 threshold, and visualize a force-directed layout for song recommendations.

Implement a collaborative filtering engine to generate song recommendations by comparing user profiles with cosine similarity, selecting similar users, and returning top-n genre-based songs.

The lecture demonstrates building and evaluating a collaborative filtering model for song recommendations. It visualizes genre popularity distribution and a heatmap of user interactions to produce top recommendations.

Build a k-nearest neighbors and gaussian nb pipeline with a column transformer and standard scaler, performing train-test split and evaluation to tune a song recommendation model.

Tune ml pipelines by building parameter grids, evaluating neighbor and distance metrics, and using grid search cv with learning curves to compare Naive Bayes and collaborative filtering.

Explore k-means clustering and PCA to group songs into five clusters—high energy dance, mellow acoustic, pop hits, instrumentals, heavy speech—and visualize centroids for a song recommendation pipeline.

Automate the song recommendation pipeline by setting up an Airflow DAG for data splitting, generating a user-genre interaction matrix, and training and evaluating a collaborative filtering model with MLflow.

Validate dag assets by inspecting the dag directory, listing current and Airflow data sets, and verifying tracks.csv exists. Confirm mounted data and paths before building Docker images.

Master the docker compose lifecycle by defining services, networks, and volumes for MLflow, Airflow, and Postgres; build, bring up, and bring down containers to orchestrate ML pipelines.

Learn the dockerfile lifecycle, creating a python or airflow image, then build and run containers, using run, copy, expose, and cmd in a /app directory.

Define a docker compose setup to deploy mlflow and airflow microservices. Configure mlflow server on port 5000, postgres storage, volumes, and the airflow mlflow network.

Build a Python 3.9 slim docker image to run train and deploy scripts, install dependencies, copy project files, expose port 8080, and prepare an Airflow dockerfile.

Build a docker image for airflow using python 3.9 and Airflow 2.6.3, install system and library dependencies, copy requirements, install packages, and configure MLflow tracking URI.

Build and compose the microservices for the song recommendation pipeline by containerizing mlflow, airflow, and postgres, mounting dags, logs, datasets, and models, and launching with docker compose up -d.

Orchestrate airflow triggers and mlflow tracking to train, log, and deploy a music recommendation model, capturing metrics, artifacts, and dockerized dag workflows.

Create a load model function that reads a pkl file from a path, uses try and catch, logs errors with Streamlit, and returns the pickle model or none on failure.

Load the collaborative filtering model and tracks data, then build a fallback recommender in Streamlit catch resource. Apply tf-idf with English stop words to song data using fittransform, returning scores.

Access cache data to build a Get Unique Channels function, collect all genres into a set using for loop, iterate song data, and return a sorted list of unique genres.

Build a music recommender UI in streamlit with a multi-select genre dropdown defaulting to pop, a Get recommendations button, validating at least one genre and using cosine similarity for recommendations.

Apply filtering techniques to generate top songs by genre ranked by track popularity, using a lambda to lowercase genres, then join genres and display the top five recommended tracks.

Run the streamlit app locally with streamlit run App.py to test genre-based recommendations, listen to tracks, handle warnings when no genre is selected, then create a GitHub repo and deploy.

Push your ml-based song recommender system code to a public GitHub repository by initializing git, adding and committing files, switching to main, pushing to origin, and deploying a Streamlit app.

Deploy the song recommender app to streamlit cloud using github ci/cd, monitor deployment logs, resolve dependencies, and launch a working web app with live recommendations.

Build your own ML application and automate ML pipelines for song recommendations, applying practical knowledge from this course and preparing for future docker image deployment to Amazon Elastic Container Registry.