Practical Machine Learning by Example in Python

This section for developers with experience in other languages. Experienced Python developers can jump ahead and use this section as a reference.

You will learn about basic syntax and data types.

There are two way to format strings in Python, we will learn both of them and their differences here.

PEP 498 added a simpler, more powerful way to format strings. This feature is available in Python 3.6 and later.

Python is dynamically typed, so we don't declare variable types, but values do have a type. We can convert values from one type to another. We'll learn how to do that in this lecture.

Python, like other imperative languages, supports conditional and looping statements. Whitespace is used to group statements together in Python, so that's something to aware of.

Python lists are like dynamic arrays. They can grow as needed and they are easy to use.

Dictionaries contain key value pairs, similar to a HashMap or hash table in other languages.

Functions are defined using the "def" keyword in Python. They can include any number of arguments and return any number of values.

Python supports object-oriented programming, like many other modern languages. The primary difference is the syntax, otherwise, the concepts are similar to other object-oriented languages. This is not meant to be a comprehensive lecture on OOP concepts, just a quick introduction in case you have OOP experience and you want to use those skills in Python.

Reading and writing files is very easy in Python, we will see some basic examples. We will also learn how to import modules. There will be plenty of examples throughout the course so it's not important to know everything about modules here.

If you need to connect to a database or other resource that requires a password, you don't want to display or save the password in your notebook. It's best to prompt for the password. The getpass module does just that.

You will learn how machine learning models are built and trained without any complex libraries. The goal is not to build the best model possible, it's about learning the fundamentals of machine learning and how models learn from data using a basic example. We will build on these ideas in later sections.

The process of developing a machine learning model is different from traditional software development. ML models depend heavily on data, so it's important to understand the nature of the data we are working with.

ML development is experimental, empirical, and iterative. It's important to analyze results, understand why a model is performing well or not well. You will learn this process as we go through each example.

This a relatively small data set that is widely available. We will load the data and perform some basic analysis to understand the nature of the data and how it can be used to train a machine learning model. Data analysis is an important first step when developing a machine learning model.

Logistic regression is quite straightforward. Each input feature is multiplied by a weight factor and added together along with a bias. The model will learn the weights and biases during training. The values of these weights are biases represent the "knowledge" the model has learned from training data.

We will write the code for the forward pass or prediction assuming we have the values for the weights and bias. Later we will develop the code learn the values for the weights and bias.

The loss and cost functions measure accuracy of the model during training. The loss function measures the difference between expected and predict values for one sample, whereas the cost function is an average of loss values for all samples.

How do you quickly get to the bottom of a hill? Go in the direction of steepest descent.

The fundamental equation for gradient descent is x_{k+1} = x_{k} - learning_rate*cost. Compare this with finding the minimum of a parabola.

Backpropagation is the algorithm used to train neural network models, we will use that here. Backpropagation is based on gradient descent, meaning determine which direction the weights and biases should be adjusted and adjust them slightly. This is done over and over until the model converges towards and acceptable level of accuracy.

In this lecture, we will train the model and plot the loss function to determine how well the model is training.

Once the model is trained, we can use it make predictions and also measure prediction accuracy.

It's common to split data sets into a train and a test data set. We didn't do that here for several reasons. The common approach is to split data into test and train sets to develop a model that generalizes well.

Training a production model can take a lot of time. Two ways to reduce training time are better algorithms and hardware acceleration.

What we built and trained was a single layer neural network. We can build deeper models that are more powerful. We will learn more about these deeper models throughout the course.

A quick introduction to NumPy and why it is important for machine learning and data science.

NumPy arrays can be created in many ways, we will look at a few in this video.

Some basic operations with NumPy arrays, such as element-wise arithmetic, adding arrays, reversing, and selecting rows/columns.

Linear regression is a classic method of fitting a line or curve to a set of data. This isn't exactly machine learning, but many of the ideas are used in machine learning. Most importantly, you will learn some basic linear algebra, which is heavily used in machine learning and also an important branch of mathematics.

If you are familiar with linear algebra, this would be a review. If not, don't worry, it's simpler than you might think. Feel free to review this lecture more than once to be sure you understand all of the concepts. If you still have questions, don't hesitate to ask a question or send me a private message.

We will build on ideas from the previous lecture to cases where we have more data. We will then use NumPy to fit a line with given many data points. We will also estimate the accuracy of this model.

Feel free to review these lectures as many times as you need. Don't hesitate to ask questions or send me a private message. I'm here to help.

Linear regression can applied to complex functions, long as it contained linear coefficients. This has been used extensively in the scientific community for a very long time.

Some basic statistics functions and linear algebra operations.

NumPy inter-operates with Matplotlib, which is a high quality visualization API. We will create several types of plots with just a few lines of code.

Numpy can work with images, which are represented as arrays of pixel values. This is useful when preparing or visualizing image data with a machine learning model.

Numpy arrays can be reshaped in to different dimensions if needed. Understanding the shape of a numpy array is important, so make sure you know the dimensions of any array you are using in a machine learning model.

Tensorflow is a popular, open source, high level machine learning library. Tensorflow 2 now includes Keras, so it is easy to use and supports many layer types, activation functions, and optimizers.

In this example, we will define a model to learn a mathematical function. This is simple example, so you can experiment and get results quickly.

Tensorflow supports many types of model layers. You don't need to learn all of them right away. We will learn by example.

It's important to experiment!

Activation functions perform an important function between model layers. Without activation functions, deep learning models wouldn't learn much.

In this example we will train the model defined earlier. We will evaluate the mode with test data and plot loss functions to determine how well the model trained.

During training, loss functions measure the difference between actual and expected output. There are two important loss functions we will use.

Keras includes several optimizers to efficiently train a model.

Once a model is trained, it can be used to make predictions using the model.predict(...) method.

Model weights and biases as well as the model structure can be save to a file and restored later. It's a bit like saving all of the knowledge it has gained. This is easy to do with the Keras API.

You learned a lot in this section! You can come back and review it as needed. The ideas we learned here can be applied to a more complex math problem, known as the three body problem. The three body problem does not have a closed form solution, so it must be solved numerically, which can take a lot of time.

Deep neural networks have been shown to solve this problem more efficiently. See the resources for more details.

Computer vision was at one time a very challenging problem, but that changed in 2012. We will use the MNIST data set to train a supervised image classification model to recognize hand written digits. At the end of this section, you will have learned how image recognition works and how you can build your own image recognition models using Convolutional Neural Networks (CNNs).

We will explore the MNIST data set using Google Colab.

Many models have been developed over the years. Deep learning Convolutional Neural Networks (CNNs) work best for image recognition.

Some amount of data preparation is needed. In this case, images need to reshaped and all values should be in the range of 0.0 - 1.0 rather than 0-255.

CNN models often include three types of layers:

1. Convolutional layers

2. Pooling layers

3. Dense or fully connected layers

Each of these layers require a number of parameters, e.g. number of filters, padding, kernel size etc.

CNNs are a big subject, so we will learn the basics here. Send me a message if you have specific questions.

The model is defined using the Keras API. We will learn more about the Keras API in the Keras foundations section of this course. For now, we will review the model and it's layers.

The model is trained in two steps: compile and fit. We can plot the accuracy of the model evaluate performance.

Once a model is trained, we can use it to make predictions. Prediction is much faster than training and easy to do with the Keras API.

Understanding when a model makes incorrect predictions is an important step in model development and improvement. Here, we will look at some incorrect predictions and determine the possible root cause.

Models have a number of hyperparameters that can be tuned to improve prediction performance. Some hyperparameters include:

1. Number of epochs

2. Regularization (dropout)

3. Layer specific parameters, e.g. number of filters, kernel size, etc.

In this example, we will experiment with some hyperparameters to improve accuracy and reduce variance. There are many hyperparameters we can adjust, which can be time consuming. It is possible to automate the training process to find the best set of hyperparameters.

Some common questions about image classification are answered, such as:

1. How many layers should a model have?

2. How much training data is required?

Pandas is a popular open source library for analyzing and transforming structured data.

In this example you will see how to load and inspect data using Pandas. Also, you will learn one way of importing data into Google Colab, which is a common operation.

Pandas makes it easy to select rows and columns from a dataframe.

Pandas includes easy to use methods to sort and transform data, as well as identify missing data.

Pandas can group rows together based on a column or set of columns and apply an aggregate function. This is similar to the SQL group by clause.

Pandas integrates with Matplotlib so it is possible to create high quality plots and visualizations of dataframes with just a few lines of code.

You have seen recommendations on YouTube, Amazon, and many other websites. In this example, we will learn how to train a machine learning model to make movie recommendations.

We will explore the MoviLens data set using Google Colab.

Collaborative filtering tries to optimize the weight factors of both users and products in order to make an accurate recommendation.

Some minor data preparation is needed:

• Join the ratings and movie dataframes

• Generate sequential numbers for users and movies

Keras embedding layers translates object numbers into vectors suitable for a machine learning model.

The model is defined using the Keras functional API. This is more flexible than the sequential class. The functional API supports multiple inputs, which is needed here because the collaborative filtering model requires two inputs.

This model is trained using mean squared error for a loss function and the ADAM optimizer.

Once the models is trained, it can be used to predicts ratings for a given user. The highest predicted ratings will be used to make recommendations.

To make predictions for one user, a distinct set of all movies and a vector for one user is supplied to the trained model. This will produce a set of ratings that can be sorted in descending order for find the best recommendations.

Let's look at some cases where the model did not predict well and try to understand why that's happening.

Customer comments are a great way to understand how people feel about a product or service. In this example, you will learn how to train a machine learning model to classify natural language text into two or more categories.

The data set we will use contains a number of reviews. Each review includes a label representing positive or negative sentiment.

We will use a supervised learning style using word embeddings to encode text or turn words into vectors suitable for a machine a learning model.

We will use a Tokenizer to break up comments into words and then encode each word in to a vector using GloVe embeddings. Those vectors will be concatenated to produce an embedding matrix.

We will define a 1D Convolutional Neural Network (CNN) model for this example. CNNs are commonly used for image model, but they can be useful in other applications as well. CNNs are good at learning spatial features, such as pixels in and image, or words in a sentence.

We will train our model, plot the accuracy, and look at some predictions. We will also look at cases where it predicts incorrectly to try and understand it's limitations.

You will learn how knowledge learned by pre-trained models can be used to improve the accuracy of new tasks, such as sentiment analysis.

In this example, we add a layer to a pre-trained BERT model and train it using transfer learning. We will use the same training data set as a comparison.

In this example, we will fine tune the BERT model to improve accuracy even further. Then we will make some predictions and analyze errors as before.

Fraud detection is an important problem in our modern digital world. In this example, you will learn how to detect credit card fraud using an unsupervised machine learning technique. The ideas in section can be applied to any data set to find anomalies for further review.