Machine Learning and Data Science with LangChain and LLMs

Explore data and analysis using Python pandas, focusing on its three data structures—series, data frames, and panels—to enable flexible, scalable data analytics.

Learn to create pandas series, a one dimensional data structure, from arrays, dictionaries, or scalars with default or custom indices, using numpy and the series constructor.

Discover how to access data in a pandas series using index positions and labels, including single, slice, and multi-element retrieval.

Analyze data series with pandas and numpy, computing mean, max, min, and standard deviation, describe quantiles, and copy series with the copy method to avoid referencing the original.

Demonstrate operations on a series, including index checks, elementwise conditions, and maths or function-based computations. Define a function to add two series and visualize results with plotting.

survey data structures with a focus on data frames, two-dimensional tabular data, and creating pandas data frames from lists, dictionaries, arrays, or other frames.

Create a pandas data frame from dictionaries or arrays, set indices and column names from dictionary keys, and handle missing values with NaN, including data frames from dictionaries of series.

Learn to update and access data frames by selecting columns with labels or indices, using df['test two'] or df1.iloc for row ranges.

Add a new column to a pandas dataframe using a pd.Series or by deriving it from existing columns, then compute a rounded average for the test values.

Delete a column in a dataframe using del or pop, and learn how shared references between df1 and df2 cause deletions to affect both dataframes.

Delete columns from a data frame using the del and pop methods, add a new column, and use the copy method to keep df1 unchanged while modifying df2.

Demonstrate pandas row selection and slicing to retrieve the second row for a student and rows two to three, using iloc and adding a new column.

Learn how to add rows to an existing pandas data frame by creating a pd.Series and appending or concatenating it, then remove rows with the drop method by label index.

Use pandas DataFrame.describe to generate statistics for numeric and object columns, handling NaN. Analyze age, salary, weight, height to show mean, max, std, and quantiles; gender excluded for string type.

Explore describing data frames with the describe method, including all attributes, numeric-only, or string-only descriptions. Examine salary statistics (mean, max, std, quantiles) and gender patterns.

Explore building a panel, a three-dimensional data structure in pandas with major and minor axes, from dictionaries of data frames and arrays, noting panel is not available in current Python.

Explore statistical analysis on a panel with two groups, using pandas describe to summarize salaries and attributes across groups; create and manipulate pd.Series data to compare groups.

This lecture teaches data analysis with pandas, building a data frame and using df.describe and statistics like mean, max, min, median, and std across numeric columns.

Analyze height and weight variables in a python dataset by computing means and summary statistics, and apply the pearson correlation to assess linear relationships while noting non-linear limitations.

Group data by city and optionally by gender to perform aggregation, transformation, and filtration, then compute counts and display city and gender groups across examples like Cairo, Delhi, Dubai, Paris.

Iterate through groups by gender, inspect the grouped data, and use the get group method to select a specific group (e.g., females) for focused analysis.

Explore aggregations using group by gender to compute mean, size, summation, and standard deviation for height and weight, applied to males and females in a dataset.

Group by gender, apply transformations and filtration on groups using a lambda to subtract group means and divide by the standard deviation, then filter cities appearing at least three times.

Explore the long chain framework for building apps with large language models, using LangChain and OpenAI to craft a question answering system with a prompt template in a Python example.

Master prompt templates in the Lang Chain framework by defining a template with curly brace placeholders, specifying input variables, and generating final prompts when running a chain.

Create and configure an OpenAI language model instance in LangChain to interact with the OpenAI API, generating text from prompts while managing API calls and parameters like model and temperature.

Create an lm chain by pairing a language model with a prompt template in the LangChain framework, then instantiate, combine, and run it to generate responses.

Execute an lm chain to generate a language model response from a specific input by preparing inputs, replacing placeholders in the prompt template, and running the chain.

Explore semantic search with LangChain by understanding the meaning and context of queries and documents. Leverage NLP, NLU, synonym recognition, and personalization to surpass keyword matching and reveal user intent.

Explore how semantic search uses natural language understanding and intent recognition to grasp queries and surface contextually relevant results, using entity recognition and embeddings like word2vec, GloVe, and BERT.

Explore synonym recognition, synonym handling, and contextual relationships in semantic search. Tailor results by user intent, personalization, past interactions, and preferences, foundational to semantic search.

Learn word embeddings and contextual models such as BERT and GPT-3 to capture semantic meanings, and enhance search and document retrieval with ner, pos tagging, knowledge graphs, and transformers.

Explore semantic search that emphasizes meaning over keywords, enabling natural language queries and contextual insights. Discuss challenges and future directions, including ambiguity, complex queries, privacy, multilingual capabilities, and model training.

Install lang chain and OpenAI. Import components like OpenAI embeddings, vector stores, character text splitter, prompt template, and retrieval QA to build a simple semantic search in Google Colab.

Initialize the OpenAI embeddings using your OpenAI API key. Learn how embeddings convert text into dense vectors in a vector space to support semantic search across documents and queries.

Explore documents and embeddings in the LangChain framework, create sample data, and apply semantic search, using Google Colab and Jupyter notebooks to prototype stable diffusion applications.

Indexing documents prepares content for efficient semantic search by preprocessing text, tokenizing, cleaning, normalizing, stemming, splitting into chunks, creating embeddings, storing vectors, and enabling similarity search with cosine distance.

Explore the retrieval QA chain that blends document retrieval with a language model to generate accurate answers by using a retriever and vector store to fetch relevant text chunks.

Run a retrieval qa chain by formulating a query, executing qa_chain.run, and printing the result, while exploring document indexing, semantic search, and overall integration.

Explore LangChain, an open source framework that links language models to external data sources for NLP applications. Discover chains and links, prompts, and callbacks for data retrieval, transformation, and monitoring.

Build a simple calculator using LangChain in Google Colab. Explore prompt management, memory management, and integration with external APIs to leverage language models like GPT.

Build a simple user interface for a calculator that accepts an operation and two numbers, using float inputs and if-else logic.

Design a simple calculator program that performs add, subtract, multiply, and divide on two numbers, and prints the result. Handle invalid operations by signaling an error.

Configure an OpenAI API key, create a prompt template, and set up the language model as a calculator to perform operations requested by the user.

Set up and connect prompts and language models using LLMChain to power a calculator that handles complex, natural language queries with an enhanced calculator function.

Integrate voice input into the calculator by capturing audio, converting speech to text with the speech recognition library, using pi audio for audio capture, and enabling text-to-speech output when needed.

Set up voice recognition using the speech recognition library to capture microphone input, adjust for ambient noise, and convert speech to text with Google recognition within a LangChain and OpenAI setup.

Leverage LangChain and an OpenAI model to enhance the calculator with voice input, using an OpenAI API key and a predefined template to parse user requests and execute operations.

Discover how to implement text-to-speech output using the Google gTTS library, play audio with playsound, recognize speech, and run a speech-enabled calculator via voice input.

Build a simple data analysis project in Google Colab using Python and lang chain, leveraging pandas, numpy, matplotlib, and seaborn for data manipulation and visualization.

Install and import essential data science libraries for LangChain workflows, including numpy, pandas, matplotlib, seaborn, and LangChain, then import them as np, pd, plt, and sns.

Load data from Kaggle or any open source by reading a CSV with pandas, using sample data and datasets like salary, Glassdoor jobs, and climate time series.

Explore the data by displaying the first rows with data.head, get basic information with data.info, check missing values via data.isnull().sum, and compute summary statistics with data.describe.

Explore data analysis with NumPy by computing mean, median, and standard deviation for arrays, filter data by threshold, and visualize results with Matplotlib and Seaborn.

Visualize data with Matplotlib and Seaborn, creating line plots, correlation heatmaps, and box plots with labeled axes and titles to reveal data insights.

Set up a simple LangChain using OpenAI LLMs to analyze data, generate insights, and prepare for advanced data analysis by integrating prompts, context, and chain usage.

Integrate LangChain with your data pipeline and large language models to generate advanced insights and automated predictions, using prompts, context, and data descriptions to analyze trends and forecast next quarter.

Explore real estate data analysis using Python libraries in Google Colab, leverage LangChain for LLM workflows, and visualize datasets with pandas, matplotlib, and seaborn.

Explore the real estate data set by loading the csv, inspecting head, checking missing values and data types, and computing basic statistics to understand distribution before moving to data cleaning.

Explore Pinecone, a vector database for storing and querying high dimensional data from text, images, and audio, enabling NLP, image recognition, recommendation systems, anomaly detection, and intelligent search.

Install core libraries numpy, pandas, matplotlib, seaborn, and pinecone client, then review their roles for numerical computing and data handling in LangChain workflows.

Explore matplotlib and seaborn to create static, animated, and interactive visualizations in Python, including simple plots such as line charts, scatter plots, and bar charts.

Explore Seaborn, built on Matplotlib, a high-level interface for attractive statistics, with iris data visualized via a hue by species pairplot.

Learn how LangChain enables model chaining and API-powered pipelines for tasks like text generation and Q&A, and how Pinecone enables semantic vector search for text data.

Discover semantic search on text data using embeddings and vector search with Pinecone, while loading data with pandas, preprocessing and tokenizing text, and visualizing statistics.

Visualize statistics of text data with matplotlib and seaborn by plotting a word-count histogram with 30 bins and kde enabled, using a 10 by 6 figure to show word-count distribution.

Set up pinecone for vector search by logging in via Google, GitHub, or email, and explore docs. Initialize with your API key, create a 768-dimensional index, and apply retrieval augmentation.

Build a language model application using LangChain and Pinecone by generating and indexing text embeddings with OpenAI, performing similarity searches to enable semantic search for text data.

Execute a real estate ML project by gathering California census data, preparing features, training and fine-tuning a model to predict district median housing prices, present, launch, monitor, and maintain pipeline.

Design and deploy robust data pipelines for supervised regression on census data to predict district median housing prices, using batch learning and a clear data flow.

Explore the root mean square error (RMSE) as a key performance measure for regression, and walk through its formula and common notations with sample data.

Explore notations for machine learning data: represent features as matrix X, define predictions with hypothesis h on x_i, and measure error using the MSE cost function.

Compare mean absolute error and rmse for regression, explain l1 and l2 norms, and show how outliers influence these distance measures.

Fetch and load housing data by managing dataset paths, directories, and csv files using Python with os, numpy, and pandas.

Parses a housing dataset using pandas, detailing ten attributes such as longitude, latitude, housing, total rooms, total bedrooms, population, households, median income, median house value, and ocean proximity.

Plot a histogram for each numerical attribute to visualize distributions and inspect how scaling and capping affect the data. Review value counts, describe statistics, and consider implications for training.

Learn how to create a test set by randomly reserving about 20% of data, and why avoiding data snooping bias helps estimate true generalization error.

Assign about 20% of the test set by hashing each instance's identifier, ensuring consistency across data refreshes. Use a crc32-based check to decide test membership and prevent leakage.

Use the row index as the housing dataset id by resetting the index, then apply train_test_split to create train and test sets, and address errors in the code.

Learn to create stable ids for records and use sklearn's train_test_split with a random_state to split the training and test sets consistently across multiple datasets, reducing sampling bias.

Stratified sampling ensures representative test sets by dividing the population into strata and sampling from each to reduce bias, then create five income categories with pd.cut and view their histogram.

Apply stratified sampling with scikit-learn's stratified shuffle split to create a train-test split by income category. Compare income category proportions in the test set and full data.

Visualize data for insights by focusing on the training set and sampling exploration set, using a longitude and latitude scatter plot with alpha 0.1 to reveal Bay Area density.

Identify high density areas such as Bay Area, Los Angeles, and Central Valley, and visualize housing prices with a scatter plot where circle size encodes population and color encodes price.

Compute Pearson's r using cor and build a correlation matrix, then interpret linear relationships such as between median income and median house value, noting non-linear patterns via scatter matrix.

Examine the strong correlation between median income and median house value via a scatter plot, noting a $500k price cap and related lines, and consider removing districts to avoid quirks.

Explore attribute combinations to identify data cleaning needs and meaningful correlations. Incorporate rooms per household and bedrooms per room against median house value to guide iterative prototype refinement.

Master data preparation for machine learning by building reusable transformation functions, separating predictors from labels, and applying transformations to new data before feeding it to algorithms.

Learn practical data cleaning by handling missing values with drop and imputation techniques, including computing medians from the training data, saving statistics, and applying Simple Imputer to numerical features.

This lecture covers converting categorical text attributes, like ocean proximity, to numbers with ordinal encoding, explains its limitations, and introduces one-hot encoding with scikit-learn's one-hot encoder.

Demonstrate using sklearn's OneHotEncoder to transform high-cardinality categorical features into sparse matrices, discuss memory efficiency, and explore replacing categories with numerical features or embeddings.

Create custom transformers for scikit-learn pipelines by implementing fit, transform, and fit_transform, using TransformerMixin and BaseEstimator to enable get_params and set_params, and demonstrate combining attributes.

The BaseEstimator, TransformerMixin based transformer computes rooms per household and population per household, and can optionally add bedrooms per room via a hyperparameter, preparing features for machine learning models.

Build end to end data transformations using scikit-learn pipelines to chain simple imputer, median strategy, standard scaler, and other steps for numerical attributes, with fit and transform methods.

Apply a scikit-learn column transformer to preprocess numerical and categorical housing features, using a numeric pipeline and a one-hot encoder, and concatenate outputs along the second axis.

Train a linear regression model from sklearn, fit it on the housing data and labels, then transform the training set and print predictions for sample instances.

Compare regression models by calculating RMSE from mean squared error on training data, illustrate underfitting with linear regression, then train a decision tree regressor to assess overfitting and test-set considerations.

learn how to evaluate a decision tree with a train-test split or ten-fold cross-validation, training on nine folds and validating on the remaining one, and compute rmse scores with cross-validation.

Compare the decision tree with linear regression using cross-validation to estimate performance and its variability, revealing overfitting in the decision tree and the promise of random forest ensemble learning.

Train and evaluate a random forest regressor in scikit-learn, assess RMSE and cross-validation scores, address overfitting by simplification or more data, and save and compare multiple shortlisted models with joblib.

Use scikit-learn's GridSearchCV to automatically explore hyperparameter combinations for a random forest regressor, using a specified param grid and five-fold cross-validation with negative mean squared error as the metric.

Tune a random forest regressor with grid search cross-validation, exploring 18 hyperparameter combinations across 90 training rounds, identifying best params and estimator, and evaluating RMSE scores.

Explore randomized search versus grid search for hyperparameter tuning, using randomized search CV to explore many parameter values, and analyze feature importances from a random forest regressor to refine models.

Evaluate the final model on the test set with the full pipeline, compare RMSE, and report a 95% confidence interval to gauge generalization against cross-validation.

Explore classification in supervised learning by examining the mnist dataset, a 70,000-digit benchmark, and learn how scikit-learn fetches openml data into x and y with descr, data, and target.

Shows how to display a digit from the 70,000 MNIST-style images with 784 features by reshaping to 28 by 28 and displaying with matplotlib imshow, using a 60k/10k train-test split.

Create a binary detector that distinguishes the digit five from not five, training an SGD classifier on the training set with a fixed random state for reproducible online learning results.

Master cross-validation for evaluating classifiers using stratified k-fold in scikit-learn, training clones on train folds, predicting on test folds, and measuring accuracy by correct predictions.

Evaluate model performance with crossvalscore using cv=3 and accuracy, illustrating a dummy classifier's predictions and explaining why accuracy misleads on skewed datasets.

Explore how the confusion matrix evaluates a classifier by counting misclassifications, using cross_val_predict for unbiased predictions, and computing the matrix with target classes and predicted labels.

Explore an illustrated confusion matrix using sklearn, reading true negatives, false positives, false negatives, and true positives. Learn precision (TP/(TP+FP)) and recall (TP/(TP+FN)) and how they relate to model performance.

Learn to measure classifier performance using precision, recall, and the F1 score with scikit-learn, and understand the precision–recall trade-off through practical examples.

Explore the precision-recall trade-off in SGD classifier decisions using the decision function and threshold, illustrating true positives, false positives, and how thresholds affect precision and recall in scikit-learn.

Explore how to use scikit-learn's decision function to obtain prediction scores, threshold them for predictions with an SGD classifier, and observe how changing thresholds impacts recall.

Explore how threshold selection affects recall and precision using precision recall curves, decision scores from cross val predict, and plotting with matplotlib in machine learning.

Learn to interpret precision recall plots, select a threshold for a target precision, and use argmax to find the threshold achieving 90% precision on the training set.

Explore the roc curve for binary classifiers, plotting true positive rate against false positive rate. Learn to compute tpr and fpr across thresholds with sklearn and visualize with matplotlib.

Explore constructing and interpreting the ROC curve and ROC AUC score, using fpr, tpr, recall, and the trade-off with PR curves when the positive class is rare.

Compare a random forest classifier with an sgd model by plotting roc curves from cross-validated predictions, using positive class probabilities as scores, and evaluating roc auc, precision, and recall.

Extend binary classifiers to multiclass tasks using one-vs-all and one-vs-one strategies, training n or n(n-1)/2 classifiers and selecting the class with the highest score.

SGD classifier uses an ovo approach with binary classifiers for multi-class tasks, then uses the decision function to produce scores and selects the highest score as the predicted class.

Implement one versus one and one versus rest multiclass classification in scikit-learn with an OVO strategy using an SGD classifier; compare to a random forest, and evaluate with cross-validation.

Perform error analysis by computing a confusion matrix from cross-validated predictions and visualizing it with matplotlib, noting that predictions lie on the main diagonal and that fives may be underrepresented.

Analyze the confusion matrix to reveal misclassifications by the SGD linear classifier, focusing on digits such as 3 and 5, and explore feature engineering like counting closed loops.

Explore multi-label classification by predicting multiple binary tags per instance, illustrated with face recognition and a multi-label k-nearest neighbors example; evaluate with f1 score and label-weighted support.

Explore multi-output classification, a generalization of multi-label tasks where each label can have multiple values, and build a denoising system that maps noisy digit images to clean pixel intensities.

Learn how linear regression predicts y hat via a vectorized theta dot x with a bias term, and how training minimizes mean squared error to optimize model parameters.

The normal equation provides a closed-form solution to minimize the cost function and compute theta hat for linear data using x transpose and the y target vector.

Use the normal equation to compute theta hat with numpy linalg and dot products, yielding theta best near four and three but not exact due to Gaussian noise.

Explore linear regression model predictions using theta hat, x nu, and y predict, and visualize results with matplotlib plots.

Apply linear regression with scikit-learn, fitting and predicting on x and y. Learn the pseudo inverse via SVD, Moore-Penrose, and the normal equation, highlighting singular matrices.

Gradient descent updates parameters iteratively in the direction of steepest descent to minimize the cost function, starting from random initialization and using a learning rate to converge to a minimum.

Explore how learning rate affects gradient descent, avoiding too-high rates that cause divergence, while recognizing that convex MSE in linear regression yields a single global minimum and reliable convergence.

Practice batch gradient descent by computing the gradient of the cost function with respect to each model parameter, using partial derivatives and the gradient vector to compute derivatives at once.

This lecture explains the gradient descent step, introduces the gradient vector of the cost function and batch gradient descent using the full training set with a learning rate.

Stochastic gradient descent speeds training by updating gradients from a single random example, enabling scalable learning, while randomness helps escape local minima with a learning rate schedule guiding convergence.

Learn to fit nonlinear data with a linear model by adding polynomial features. Use scikit-learn's polynomial features to transform data and train with gradient descent.

Explore learning curves by comparing a 300-degree polynomial model to linear and quadratic models on training data, illustrating how high degree polynomials fit the data.

Explore how learning curves reveal overfitting and underfitting, using cross-validation and train-test splits to compare models like quadratic versus linear, and visualize with mean squared error plots.

Explore learning curves to diagnose underfitting and overfitting on training data, compare linear vs polynomial regression via a sklearn pipeline, and discuss bias, variance, irreducible error, and data cleaning.

Learn how early stopping regularizes gradient descent by halting training at the validation error minimum to prevent overfitting, illustrated with batch, stochastic, and mini-batch approaches.

Apply early stopping with stochastic and mini-batch gradient descent, using a pipeline of polynomial features of degree 90, standard scaling, and a warm-started SGD regressor to minimize validation error.

Explore logistic regression, a binary classifier that estimates the probability of the positive class using a sigmoid output and a 0.5 decision threshold.

Illustrate decision boundaries by applying logistic regression to the iris dataset to classify iris virginica based on petal width.

Train a logistic regression model to estimate iris probabilities from petal width, showing 1.6 cm boundary and how predict_proba differs from predict for iris virginica.

Explore linear support vector machines, separating linearly separable classes with a maximum margin; learn how support vectors define the decision boundary and why margin matters.

Explore soft margin classification and contrast it with hard margin classification, highlighting how outliers affect the decision boundary and generalization using the iris data.

Train a linear SVC with C=1 on features scaled by StandardScaler to detect Iris virginica using hinge loss, and compare linear kernel SVC and SGDClassifier options.

Explore nonlinear svm classification via polynomial feature mapping to achieve linear separability, and implement a scikit-learn pipeline with polynomial features, standard scaler, and linear svc on the moons dataset.

Generate a moons dataset, apply a polynomial features pipeline with degree 3 and standard scalar, and train a linear SVC to build a linear SVM classifier.

Explore how the polynomial kernel enables SVMs to mimic many polynomial features via the kernel trick, balancing degree choices to control overfitting and underfitting in practical models.

Explore the Gaussian RBF kernel with SVC and SVM, showing how gamma shapes the decision boundary and acts as regularization, alongside the C parameter and feature costs.

Explore SVM regression, learning linear and non-linear regression by maximizing points on the margin while limiting margin violations, and compare models with margins 1.5 and 0.5 on linear data.

Continue from the last lecture and discuss the second graph, focusing on values around 0.5 and how the lines relate, clarifying that the value is 0.5, not 1.5.

Explore the training objective by analyzing how the slope of the decision function equals the weight vector norm, showing that smaller weight vectors yield larger margins, illustrated with w1=1.

Explore editing a diagram to illustrate smaller weight vector results, using an eraser to remove elements and highlighting changes, culminating in a diagram labeled 0.5.

Explore hard and soft margin linear SVMs, formulating the objective to minimize 1/2 w^T w with margin constraints, and introducing slack variables to handle violations.

Explore quadratic programming, where hard and soft margin problems yield convex quadratic optimization with linear constraints, and learn the general formulation and the availability of off-the-shelf solvers.

Continue the discussion on quadratic programming by defining p, f, P, and A, with dimensions NP by NP and N by NP, and explaining the A p ≤ b constraints.

Explore training a hard margin linear svm by solving a quadratic program with p, a, and bias terms, and extend to soft margin and kernel trick using qp solvers.

Explore kernelized SVM by applying a second-degree polynomial mapping to a two-dimensional dataset like the moon, transforming it into three dimensions, then train a linear SVM on the transformed data.

Demonstrates the kernel trick for a second-degree polynomial mapping in kernelized SVM, derives transformed dot products, and shows how applying the transformation to all training instances affects the dual problem.

Discover how the second degree polynomial kernel replaces the dot product with its square in kernelized SVM, using k(a,b) = (a^T b)^2.

The lecture covers common kernels - linear, polynomial, gaussian rbf, and sigmoid - along with their formulas, including k(a,b)=a^T b and the polynomial and rbf definitions.

Explore kernelized SVMs and how the kernel trick derives predictions from dual to primal forms without computing w_hat. Compute the decision function for new instances using alpha_hat, X_i, and b_hat.

Continue deriving the equations for predictions in a support vector setting by computing the dot product with only the support vectors (nonzero alpha) and including the bias term b hat.

Explore how decision trees handle classification, regression, and multi-output tasks, and visualize training on the iris dataset using Graphviz to understand predictions.

Analyze how Gini impurity quantifies node impurity using class counts from training instances in the iris decision tree, and observe how the cart algorithm builds a binary tree.

Explain how the cart algorithm trains decision trees by greedily splitting data on a feature and threshold to minimize impurity, then recurses until max depth or no improvement.

Explores regression by building a regression tree with scikit-learn's decision tree regressor, trained on a noisy quadratic dataset with max depth two on x and y.