BB&N HacKnight - Web Scraping with Beautiful Soup

• Designed for readability, so it is similar to the English language

• Uses indentation to indicate the block of code → will create an error if not done properly

• Comments start with a #

• print(“Hello!”) → Hello!

Variables

• Used to store data values under a reference “name”

• Do not need to declare variables beforehand

  
  

Conditional Statements

• Python supports the following logical conditions: ==, !=, >, <, <=, >=

• The inside of an “if” statement executes if it evaluates to True, it does not if it is False

• Indentation is important!

Functions

• A block of code that can be referred to throughout the program for specific actions and runs only when it’s called

• Functions take data (parameters) and return data (results)

• They are defined using the def keyword

Loops

• There are two types of loops: while loops and for loops

• While loops execute a statement as long as the condition is True

  • If “i” is not incremented, the loop will likely go on forever

• For loops are used to iterate over a sequence (like lists)

Beautiful Soup Library and Basic Web Scraping

Installing Libraries

Open the terminal in PyCharm

Installing Beautiful Soup: pip install beautifulsoup4

Installing the parser method: pip install lxml

Import Libraries

--> from bs4 import BeautifulSoup

Opening File Objects

“With open” → Allows you to open a file and read its content

Parameters → The file’s name, whether you are reading or writing the file

“As” keyword → Specify the variable name

.read() method → Reading the file’s content

--> with open("home.html", "r") as htmlfile:

--> content = htmlfile.read()

Creating an instance of the Beautiful Soup library

Arguments → HTML file you will scrape, the parser method (lxml)

--> soup = BeautifulSoup(content, "lxml")

Finding all the HTML tags that are defined as <h5> … </h5>

.find() → searches for elements with the specific HTML tag

Only returns the first element and terminates

.find_all() → returns all the elements with the specific HTML tag

--> courses_html_tags = soup.find_all("h5")

Inspect tool

<div> … </div> represent all of the individual “cards” of courses

Grabbing prices

<a> … </a> is responsible for the price button

Filter out <div> tags using a commonality → class_ is “card” for all of them

--> course_cards = soup.find_all("div", class_="card")

Iterate over list

Use a for-loop

.text → used to access the text in the card

.split() → used to access the last element, use [-1]

F string → to print a dynamic sentence for each course

--> for course in course_cards:

--> course_name = course.h5.text

--> course_prices = course.a.text.split()[-1]

--> print(f"{course_name} costs {course_prices} dollars")

Final Project: Web Scraping on a Real Website

Website to scrape - TimesJobs [Search on Search Engine of choice]

Enter skill “Python” into search bar

Goal: get jobs that are “Posted a few days ago” text

1. Install requests library - pip install requests

• Requests information from a specific website

• Use the .get() function

• Copy paste URL into the function

200 → means the request was done successfully

.text → to access the HTML text only

--> from bs4 import BeautifulSoup

--> import requests

--> html_text = requests.get("[insert copied link]").text

--> print(html_text)

1. Create a BeautifulSoup instance

--> soup = BeautifulSoup(html_text, "lxml")

1. Access classes

Think of the white boxes as a list of elements → use inspect

<li> inside of a <ul> tag → an unordered list

Copy the name of the class → clearfix job-bx wht-shd-bx

Use the find() method → job = soup.find("li", class_ = "clearfix job-bx wht-shd-bx")

[The page is paginated, so this will return only the results visible on the first page]

1. Search for the header of the first job

Use .find() on only the “job” variable → doesn’t make sense to search the whole page

--> company_name = job.find("h3", class_ = "joblist-comp-name").text

Example output = east india securities ltd.

1. Remove white spaces

.replace() method → company_name = job.find("h3", class_ = "joblist-comp-name").text.replace(" ", "")

1. Get skill requirements

Inspect skills and get the class name ( srp-skills)

--> skills = job.find("span", class_ = "srp-skills").text.replace(" ", "")

1. User-friendly output

print(f'''

Company Name: {company_name}

Required Skills: {skills}

''')

1. Filtering out jobs “Posted a few days ago”

Inspect time and get the class name (sim-posted)

This time there is a <span> inside a <span> → need to search inside one <span>

--> published_date = job.find("span", class_ = "sim-posted").span.text

1. Use .find_all() for “job”

Change variable name to “jobs” for clarity

1. Using a for loop

--> jobs = soup.find_all("li", class_ = "clearfix job-bx wht-shd-bx")

for job in jobs:

  company_name = job.find("h3", class_ = "joblist-comp-name").text.replace(" ", "")

  skills = job.find("span", class_ = "srp-skills").text.replace(" ", "")

  published_date = job.find("span", class_ = "sim-posted").span.text

1. Filter out jobs without the word “Few”

Place the published_date code as the first line inside the loop

--> if "few" in published_date:

Adding Functionalities

Prettifying

1. Make two separate formatted strings to remove indentation spaces

--> print(f"Company Name: {company_name}")

--> print(f"Required skills: {skills}")

1. Use the .strip() method to get rid of the spaces

1. Add the direct link of the job post

<li> → <header> → <h2> → link about the job

Follow this same pattern for all of the jobs

--> more_info = job.header.h2.a

1. Call the href attribute for more_info

--> more_info = job.header.h2.a["href"]

Filtering out skill requirements

1. Add a user input component

--> print("Put a skill you are unfamiliar with:")

--> unfamiliar_skill = input(">")

--> print(f"Filtering out {unfamiliar_skill}")

1. Filter out unfamiliar skills

--> if unfamiliar_skill not in skills:

Saving Each Job Post as a File

1. Wrap the program into a function

--> def find_jobs():

1. Logic of “__name__ == ’__main__’ “

This conditional statement tells the Python interpreter under what conditions the main method should be executed

This function will only be executed if the file is ran directly

Guards the print statement

--> if __name__ == '__main__':

1. Repeating the project in a time interval

time.sleep () →allows the program to wait a certain amount of time, argument is in seconds

--> import time

--> time.sleep(600)

(Program runs every 10 minutes)

1. Making the program more dynamic

--> time_wait = 10

--> print(f"Waiting {time_wait} minutes...")

--> time.sleep(time_wait*60)

Writing Information in a Separated Directory

1. Create “New Directory” named posts

   
   

2. Using the enumerate() function instead of a for loop

Allows us to iterate as much as the index of the jobs list and the job content itself

Index → like a counter for the job variable

--> for index, job in enumerate(jobs):

1. Add functionality to create text files

--> with open(f"posts/{index}.txt", "w") as f:

Posts → directory name

{index}.txt → name of the file we are creating

“W” → defines the extent to which the user accesses the file (permission level → writing)

As statement and f variable → writing to a file with the f variable

1. Modify print statements

Need this so that the information is being outputted differently

--> f.write(f"Company Name: {company_name.strip()} \n")

--> f.write(f"Required skills: {skills.strip()} \n")

--> f.write(f"More Info: {more_info} \n")

--> print(f"File saved: {index}")

Checking our Program!

Optional Challenge → Create a scraper that filters out more than one unfamiliar skill