Web Scraping In Python: Master The Fundamentals

Here we will make our first HTTP request and cover the possible outcomes.

In this tutorial we will look at how we can read the text response that we get when we contact a website.

Here we're going to cover how we can start the use the text response to parse out the data that we're looking for.

Here we will look deeper into the exception cases and see how we should adapt our code to incorporate them.

Having considered the straightforward as well as the exception cases, we can now complete the data parse for one company.

Here we will see where we can get more ticker symbols from, and start by identifying and selecting the range of data that is of interest for us.

Here we will start our process of parsing out the ticker symbols, based on identifying patterns that we see in the website code.

Here we will finish up our method of scraping out company ticker symbols, so that we have a complete, and much larger, set of company ticker symbols to scrape data for.

Now that we have a complete set of symbols to scrap with, we can modify our code from before to incorporate these new companies.

Here we will ensure that we have data for all companies and put them into well formatted pandas dataframe data structure.

Here we will quickly cover the goal of this section as well as the extra libraries we're going to need to install.

We'll go into a short review of what a recursive function is, using the Fibonacci sequence as our example.

Updated lecture due to deprecated PhantomJS

We'll learn how to create a browser instance as well as the basic navigation that happens within it.

Here we'll start to look at and become familiar how the content of the website looks like.

Here we'll learn how we can use elements and XPath to navigate our response data.

Here we'll use what we've learned so far to start parsing out the relevant data.

Now we're going to apply what we've learned so far to identify the direct path to our data.

Now we will use the path that we've identified to navigate through the HTML to our data.

Here we will continue on to get the data out now after we've navigated to it.

We'll combine everything we've produced before to get out our data efficiently and into a nice format.

APIs are the other way of getting data from the web, and make it a lot easier since the data is formatted for us nicely, and all we really have to do is ask for the right data. APIs are usually easier to get data from than web scraping, as we don't need to identify patterns and deal with exception cases to extract valuable data.