IPython and Jupyter Notebook In Practice: 3-in-1

This video gives you a glimpse of what this course offers to you.

To get well versed with a new tool, it is a good practice to start up with a basic tour and perform some basic and frequently used operations. This video is that quick step in using the Jupyter Notebook and IPython commands.

• Create a new Jupyter notebook using an IPython kernel

• Perform basic mathematical operations

• Learn to use magic commands

This video will give you an introduction to IPython and Jupyter for data analysis.

• Import the scientific packages: NumPy, pandas, and Matplotlib

• Use the groupby() method group the table elements by the weekday

• Plot a smoothed version of the track attendance as a function of time

NumPy is the main foundation of the scientific Python ecosystem. This library offers a specific data structure for high-performance numerical computing: the multidimensional array. This video will illustrate the basic concepts of the multidimensional array.

• Import the built-in random Python module and NumPy

• Generate two Python lists, x and y, each one containing 1 million random numbers between 0 and 1

• Compute the arithmetic distance between any pair of numbers in our two lists

Although IPython comes with a wide variety of magic commands, there are cases where we need to implement custom functionality in new magic commands. In this video, we will show how to create line and magic cells, and how to integrate them in an IPython extension.

• Create a function that accepts the contents of the line and decorate this function with @register_ line_magic

• Create %%csv cell magic that parses a CSV string and returns a pandas DataFrame object

• Create an extension module and import it into the IPython session with the %load_ext magic command

This video will let you take a step ahead and explore the architecture of Jupyter Notebook and also show you how to connect a new client (such as a Qt console) to the underlying kernel.

• Connect multiple clients to one kernel

This video will show you how to manipulate the contents of a notebook (which is just a plain-text JSON file) directly with Python, and how to convert it to other formats with nbconvert.

• Load the notebook in a string and parse it with the json module

• Count the number of Markdown and code cells

• Convert the text notebook to HTML using nbconvert and display this document in an <iframe>

The ipywidgets package provides many common user interface controls for exploring code and data interactively. These controls can be assembled and customized to create complex graphical user interfaces. In this video, we introduce the various ways we can create user interfaces with ipywidgets.

• Create a simple user interface for controlling four parameters of a function that displays a plot

• Create a slider for selecting pairs of numbers

• Create the Tab instance, set the titles of the tabs, and add the plot button below the tabs

The ipywidgets packages provides many built-in control widgets to interact with code and data in the Jupyter Notebook. This video will walk you through those steps which will let you build a custom interactive widget from scratch, using Python on the kernel side, and HTML/JavaScript on the client side (frontend).

• Create a CounterWidget class deriving from DOMWidget

• Display the widget

Many aspects of the Jupyter Notebook can be configured. In this video, we will show you how to configure the Jupyter application and the Jupyter Notebook frontend.

• Check whether the Jupyter Notebook configuration file already exists

• Inspect the contents of the notebook configuration the JSON file

• Get and change the frontend options from Python

The %timeit magic and the %%timeit cell magic allow us to quickly evaluate the time taken by one or several Python statements. Let us take a step ahead to see the methods for more extensive profiling. In this video, we will estimate the time taken to calculate the sum of the inverse squares of all positive integer numbers up to a given n.

• Define a variable and time the computation in pure Python

• Use the %%timeit cell magic to time the same computation

• Time the NumPy version of this computation

How could you break down the execution time into the contributions of all called functions? cProfiler is a solution to this problem. This video will walk you through simple step to implement this profiler.

• Create a function generating random +1 and -1 values in an array

• Write simulation code with %%prun

Python's native cProfile module and the corresponding %prun magic break down the execution time of code function by function. Sometimes, we may need an even more finegrained analysis of code performance with a line-by-line report. Let’s see how to do this.

• Import NumPy and the line_profiler IPython extension module

• Write the code in a Python script using the %%writefile cell magic

• Execute the function under the control of the line profiler

In this video, we will look at a simple memory profiler unsurprisingly named memory_profiler. Its usage is very similar to line_profiler, and it can be conveniently used from IPython.

• Load the memory_profiler IPython extension and define a function that allocates big objects

• Run the code under the control of the memory profiler

• Display the result

This video, we will show you how to avoid unnecessary array copies in order to save memory. In that respect, we will need to dig into the internals of NumPy.

• Check whether two arrays share the same underlying data buffer in memory

• Use the flatten() and the ravel() methods

Sometimes, we need to deal with NumPy arrays that are too big to fit in the system memory. A common solution is to use memory mapping and implement out-of-core computations. Let’s see how to implement these in our code.

• Create a memory-mapped array in write mode

• Feed the array with random values

• Save the last column of the array and flush memory changes to disk

The first way to make Python code run faster is to know all features of the language. This video will show you how badly-written Python code can be significantly improved when using all the features of the language.

• Define a list of normally-distributed random variables, using the random built-in module

• Write a function that computes the sum of all numbers in that list.

• Write a slightly improved version of this code

This video will show you how to accelerate pure Python code generating a Mandelbrot fractal. Let’s go ahead and do it right now!

• Create a function that generates a fractal in pure Python

• Run the simulation and display the fractal

• Accelerate this function using Numba

NumExpr evaluates algebraic expressions involving arrays, parses them, compiles them, and finally executes them, possibly on multiple processors. We will see how that works in this recipe.

• Import NumPy and NumExpr and generate three large vectors

• Evaluate the time taken by NumPy to calculate a complex algebraic expression involving our vectors

• Perform the same calculation with NumExpr

Performance gains are most significant in CPU-bound programs, notably in tight Python loops. By contrast, I/O bound programs are not expected to benefit much from a Cython implementation. In this video, we will see how to accelerate the Mandelbrot code example with Cython.

• Import the Cython Jupyter extension

• Add the %%cython magic before the definition of the mandelbrot() function

• Add type information using typed memory views for NumPy arrays

With Cython, we have a way to release the GIL temporarily in a portion of the code in order to enable multi-core computing. This is done with OpenMP, a multiprocessing API that is supported by most C compilers. In this video, we will see how to parallelize the previous code on multiple cores.

• Import the prange() function

• Add nogil after each function definition in order to remove the GIL

• Run a loop in parallel over the cores with OpenMP, using prange()

Let’s take the next step to implement the embarrassingly parallel computation of the Mandelbrot fractal in CUDA using Numba.

• Import the packages and check whether Numba correctly identified our GPU

• Execute the GPU function, passing the empty array

• Send the NumPy array to the GPU with the cuda.to_device() function

This video will walk you through the usage of ipyparallel, which offers an even simpler interface that brings powerful parallel computing features in an interactive environment.

• Launch four IPython engines in separate processes

• Create a client that will act as a proxy to the IPython engines and check the number of running engines

This video will show you how to interact with asynchronous tasks running in parallel with ipyparallel.

• Create a client and a load-balanced view on the IPython engines

• Create a simple progress bar for our asynchronous tasks

How can we have data structures resembling NumPy arrays (dask.array) and Pandas DataFrames (dask.dataframe) that efficiently scale to huge datasets. How can you split a large array into smaller arrays (chunks). This video will provide a solution to this problems.

• Initialize a large 10,000 x 10,000 array with random values using dask

• Profile the memory usage and time of the same computation using either NumPy or dask.array

• Use multiple cores to perform computations on large Arrays and create a client using dask.distributed

Recent versions of Matplotlib have significantly improved the default style of its figures. Today, Matplotlib comes with a set of high-quality predefined styles along with a styling system that lets one customize all aspects of these styles. This video will let you explore these styles.

• Import the libraries and check the list of all available styles

• Create a plot and set a style with mpl.style.use()

• Change the style for a given plot using the context manager syntax

Seaborn is a library that builds on top of Matplotlib and Pandas to provide easy-to-use statistical plotting routines. In this video, we give a few examples, adapted from the official documentation, of the types of statistical plot that can be created with seaborn.

• Import NumPy, Matplotlib, and seaborn

• Plot the histogram, Kernel Density Estimation (KDE), and a gamma distribution fit for the dataset

• Display a bar plot, a violin plot, and a swarm plot that show an increasing amount of details

In this video, we will give a few short examples of interactive Bokeh figures in the Jupyter Notebook. We will also introduce HoloViews, which provides a high-level API for Bokeh and other plotting libraries.

• Import the packages NumPy, Bokeh, and HoloViews

• Create a scatter plot of random data

• Use Pandas to plot the hourly average temperature

This video will show you the use of Vega, which is a declarative format for designing static and interactive visualizations. Along with we will learn to use Altair that provides a simple API to define and display Vega-Lite Visualizations.

• Load the flights-5k dataset

• Create a scatter plot showing the delay as a function of the date

• Create a bar plot with the average delay of all flights departing from Los Angeles

This video will provide you a walk through the entire course, acknowledging the key take-away and a glimpse at every section.

What could be the first step after getting access to the data? This video, will provide you with that first step where we will use pandas to load the data very easily, explore the variables, and make basic plots with Matplotlib.

• Import NumPy, pandas, and Matplotlib

• Use head(), tail() and describe() methods to view the beginning, end, and summary statistics

• Use the groupby() method

This video will let you take a look at the same tennis dataset as in the previous video. Following a frequentist approach, we will estimate the correlation between the number of aces and the proportion of points won by a tennis player.

• Import the packages and load the dataset

• Find the proportion of points won, and the (relative) number of aces

• Compute the chi-squared test statistic and the associated p-value

In this video, we apply the maximum likelihood method on a dataset of survival times after heart transplant (1967-1974 study).

• Plot the raw survival data and the histogram

• Generate linearly spaced values for the x axis (days)

• Rescale the theoretical distribution to the histogram

How can we use a Kernel Density Estimation (KDE) to estimate the density of probability of a spatial distribution? Let’s look at the geographical locations of tropical cyclones from 1848 to 2013, based on data provided by the NOAA, the US' National Oceanic and Atmospheric Administration.

• Import the packages and open the data with pandas

• Display the storms on a map with Cartopy

• Display the estimated density with imshow()

In this video, we will show how to interface R with Python in the Jupyter Notebook, and we will illustrate the most basic capabilities of R with a simple data analysis example.

• Load the longley dataset with the statsmodels package

• Define x and y as the exogeneous (independent) and endogenous (dependent) variables

• Pass the data to R and fit a linear model to the data

This video will walk you through a basic example of linear regression in the context of curve fitting. This toy example will allow you to illustrate key concepts such as linear models, overfitting, underfitting, regularization, and cross-validation.

• Define a deterministic nonlinear function underlying our generative model

• Use scikit-learn to fit a linear model to the data

• Plot the result of the trained linear model

In this video, we will see how to recognize handwritten digits with a K-nearest neighbors (K-NN) classifier. This classifier is a simple but powerful model, well-adapted to complex, highly nonlinear datasets such as images.

• Import the packages and load the digits dataset

• Fit a K-NN classifier on the data

• Evaluate the score of the trained classifier on the test dataset

In this video, we will introduce support vector machines, or SVMs. These models can be used for classification and regression. Here, we illustrate how to use linear and nonlinear SVMs on a simple classification task.

• Generate 2D points and assign a binary label according to a linear operation on the coordinates

• Fit a linear Support Vector Classifier (SVC)

• Modify the labels with an XOR function

In this video, we will find the most influential features of Boston house prices using a classic dataset that contains a range of diverse indicators about the houses' neighborhood.

• Create a RandomForestRegressor model

• Get the samples and the target values from this dataset

• Display a graphic representation of the trees, using the Graphviz package

A large part of unsupervised learning is devoted to the clustering problem. There are many clustering algorithms. We will see a few of them in this video, applied to a toy example.

• Generate a random dataset with three clusters

• Relabel and display the results of the clustering algorithms

• Cluster the dataset with the K-means algorithm, a classic and simple clustering algorithm

In this short video, we will see how to use SciPy to find the root of a simple mathematical function of a single real variable.

• Import NumPy, SciPy, scipy.optimize, and matplotlib

• Visualize the root on the plot

• Use the brentq() method

This video will demonstrate a few usage examples of the function minimization algorithms implemented in SciPy.

• Define a simple mathematical function

• Plot this function on the interval [-20,20]

• Define the Lévi function

In this video, we will show an application of numerical optimization to nonlinear least squares curve fitting. The goal is to fit a function, depending on several parameters, to data points. In contrast to the linear least squares method, this function does not have to be linear in those parameters.

• Define a logistic function with four parameters

• Generate random data points by using the sigmoid function and adding a bit of noise

• Display the fitted sigmoid curve

In this video, we will give an application example of the function minimization algorithms described earlier. We will try to numerically find the equilibrium state of a physical system by minimizing its potential energy.

• Define a few constants in the International System of Units and the initial positions of the masses

• Get the indices of the spring connections

• Minimize the potential energy with a function minimization method

In this video, we will illustrate several aspects of the Fourier transform. We will apply this tool to weather data spanning 20 years in France obtained from the US National Climatic Data Center.

• Use the goupby() method and remove any N/A value with dropna()

• Cut out frequencies higher than the fundamental frequency

• Perform an inverse FFT to convert the modified Fourier transform back to the temporal domain

Linear filters play a fundamental role in signal processing. With a linear filter, one can extract meaningful information from a digital signal. In this video, we will show two examples using stock market data (the NASDAQ stock exchange).

• Smooth out a very noisy signal with a low-pass filter to extract its slow variations.

• Apply a high-pass filter to the original time series to extract the fast variations

How can you know about the repeating patterns or serial correlation? How can you learn about the timescale of the fluctuations? Let’s try to find out the solution to these problems.

• Import the essential packages and download the Babies dataset

• Write functions to retrieve the frequencies of baby names as a function of the name, gender, and birth year

• Create a function that displays the evolution of a baby name as well as its (normalized) autocorrelation

The exposure of an image tells us whether the image is too dark, too light, or balanced. It can be measured with a histogram of the intensity values of all pixels. Improving the exposure of an image is a basic image-editing operation. As we will see in this video, it can be done easily with scikit-image.

• Create a function that displays the image along with its histogram of the intensity values

• Rescale the intensity of the image using scikit-image's rescale_intensity function

• Use Contrast Limited Adaptive Histogram Equalization (CLAHE)

This video will walk you through few simple and easy to implement steps for applying filters on an image for various purposes: blurring, denoising, and edge detection.

• Create a function that displays a grayscale image

• Load the Astronaut image and convert it to a grayscale image with rgb2gray()

• Apply a Sobel filter that enhances the edges in the image

The scikit-image provides several segmentation methods. In this video, we will demonstrate how to segment an image containing different objects.

• Find an intensity threshold separating the bright from the dark background

• Clean the binary image by smoothing the elements and removing the border

• Use the regionprops() function to retrieve specific properties of the components

In this video, we will find points of interest in an image with scikit-image. This will allow us to crop an image around the subject of the picture, even when this subject is not in the center of the image.

• Find salient points in the image with the Harris corner method

• Detect corners from this measure image, using the corner_peaks() function

• Create a box around the median position of the corner points to define our region of interest

In this video, we will show how to play sounds in the Notebook. We will also illustrate the effect of simple digital filters on speech sounds.

• Create a Python function that loads an MP3 sound and returns a NumPy array with the raw sound data

• Create a function that plays a sound in the Notebook, using IPython's Audio class

• Create a simple widget to quickly test the effect of a high-pass filter with an arbitrary cutoff frequency

In this video, we will create a small electronic piano in the Notebook. We will synthesize sinusoidal sounds with NumPy instead of using recorded tones.

• Define the sampling rate and the duration of the notes

• Use NumPy and IPython's audio class to generate and play the sound of a note

• Create the piano with the Notebook widgets

Let’s proceed ahead and learn how to simulate a famous chaotic system: the logistic map. We will draw the system's bifurcation diagram, which shows the possible long-term behaviors as a function of the system's parameter. We will also compute an approximation of the system's Lyapunov exponent, characterizing the model's sensitivity to initial conditions.

• Simulate a few iterations of the discrete dynamical system with two different values

• Use 1000 iterations of the logistic map and keep the last 100 iterations

• Simulate the system and plot the bifurcation diagram

How can you model real-world phenomena such as car traffic, chemical reactions, propagation of fire in a forest, epidemic propagations, and much more? Let’s step ahead and learn this right now!

• Import the essential packages

• Write a function that performs an iteration on the grid, updating all cells at once

• Write a function that simulates any elementary cellular automaton

In this video, we will simulate a simple linear second-order autonomous ODE, describing the evolution of a particle in the air subject to gravity and viscous resistance. Although this equation could be solved analytically, here we will use SciPy to simulate it numerically.

• Import the packages and define a few parameters appearing in our model

• Create a Python function that takes the current vector and time as arguments

• Use the SciPy odeint() function to simulate the system

Partial Differential Equations (PDEs) are hard to solve analytically. Therefore, PDEs are often studied via numerical simulations. 

In this video, we will illustrate how to simulate a reaction-diffusion system described by a PDE called the FitzHugh–Nagumo equation.

• Define a function that computes the discrete Laplace operator

• Simulate the system of equations using the finite difference method

• Show the state of the system at the end of the simulation

Markov chains are relatively easy to study mathematically and to simulate numerically. In this video, we will simulate a simple Markov chain modeling the evolution of a population.

• Import the packages and define the birth and death rates

• Generate random vectors, and update the population sizes with vector operations

• Create histograms of the population size at different times

How can you represent random occurrences of instantaneous events? The Poisson process is a solution to this problem. In this video, we will show different methods to simulate a homogeneous stationary Poisson process.

• Specify the average number of events per second

• Draw a vertical line for each event and use the cumsum() function

• Get the process by cumulatively summing all of the intervals

Let’s see how to model a continuous-time random walk, where a particle evolves in space by making independent random steps in all directions.

• Simulate Brownian motions with 5000 time steps

• Simulate two independent one-dimensional Brownian processes

• Use a gradient of color to illustrate the progression of the motion in time

In this video, we will show you, how you could simulate an Ornstein-Uhlenbeck process, which is a solution of the Langevin equation.

• Define a few simulation parameters and renormalized variables

• Simulate the process with the Euler-Maruyama method

• Display the time evolution of the distribution of the process

In this video, we will show you how to create, manipulate, and visualize graphs with NetworkX.

• Import NumPy, NetworkX, and matplotlib

• Check the list of nodes and edges of the graph, and its adjacency matrix

• Use an automatic layout algorithm

In this video, we load and visualize a dataset containing many flight routes and airports around the world.

• Load the first dataset containing many flight routes

• Create the networkX graph from the edges array

• Use Cartopy to project the points on the map

In this video, we will show an application of a well-known graph algorithm: topological sorting.

• Download the dataset and load it with the NetworkX function read_graphml()

• Perform the topological sort

• Draw the graph with a shell layout algorithm, and display the dependence order using the node colors

In this video, we will show an application of graph theory in image processing. We will compute connected components in an image. This method will allow us to label contiguous regions of an image, similar to the bucket fil tool of paint programs.

• Find all contiguous dark blue regions containing more than three pixels

• Use the connected_components() function of NetworkX to find the components

• Assign different color to each component

In this video, we will show how to load and display geographical data in the Shapefile format. Specifically, we will use data from Natural Earth to display a choropleth map.

• Write a function that draws the borders of Africa

• Write a function that displays the countries of Africa with a color that depends on a specific attribute

• Display two choropleth maps with the population and GDP of all African countries