Web scraping and data analysis with Python

Name: Web scraping and data analysis with Python
Rating: 5.0 (5 reviews)

A practical introduction applied to ESG rating and stock market data

Created byDimitrios Koulialias PhD

Last updated 7/2024

English

English [Auto],

What you'll learn

Get an insight on data acquisition and data analysis processes using financial data from yahoo finance as a test-bed
Get an introduction into HTML and its main tags/elements for understanding a webpage structure
Obtain a first understanding of BeautifulSoup as a means for scraping data from a webpage
Preprocess data in Python using Pandas
Manage and extract further information out of the scraped ESG rating score data
Get an introduction into Markowitz portfolio theory for data analysis based on a (constrained) optimization approach
Incorporate the ESG rating score data into Markowitz' portfolio theory and assess the effect on the risk and return

Course content

5 sections • 20 lectures • 3h 22m total length

Course introduction5:22
Explore combining web scraping with data analysis in Python using BeautifulSoup and pandas to fetch ESG scores from Yahoo Finance and apply Markowitz portfolio optimization to the Swiss market index.
Installation / setup of the development environment0:18

General considerations3:16
Trace web scraping's evolution from the World Wide Web's early days to modern Python tools—beautifulsoup, scrapy, and selenium—highlighting its indispensable role for data professionals.
Introduction to HTML - I6:05
Explore the basics of HTML, its structure and tags, including doctype, html, head, and body, plus header and paragraph elements, and saving and running HTML files locally.
Introduction to HTML - II6:46
Explore HTML basics by working with headers, paragraphs, links, and divisions, and apply CSS to style a red bordered, center-aligned section using a class.
Introduction to HTML - III8:58
Master the structure of HTML lists and tables, including ul, ol, li, thead, tbody, tr, th, and td with practical examples and borders.
Introduction to BeautifulSoup3:56
Explore how BeautifulSoup fetches a page, stores the response as a soup object, parses HTML, and uses find or find_all with a second-parameter dictionary to select specific tags.
Scraping the 'test.html' file13:31
Apply BeautifulSoup to a local test.html file using Jupyter notebook, install and import requests, bs4, and pandas, and extract paragraphs, links, and table data with find, find_all, and text.
Scraping a webpage16:26
Learn to scrape a webpage with BeautifulSoup: fetch html, parse with bs4, extract titles and paragraphs from five divisions, and export to csv.

General considerations8:14
Define ESG as environmental, social, and governance measures guiding sustainable investing. Examine ESG rating scores, data provider differences, and how Markowitz portfolio theory can integrate ESG.
Scraping an ESG score from one stock of the Swiss Market Index (SMI)7:30
Scrape ESG scores from Yahoo Finance for the Swiss Market Index using a stock. Use a user-agent header and BeautifulSoup to extract the sustainability score and prepare a pandas dataframe.
Scraping all ESG scores of the SMI stocks12:00
generalize scraping esg scores for all SMI stocks with a reusable Python function, parsing ticker URLs and handling missing data by assigning 100, then build a pandas dataframe.
ESG score distribution across company sectors19:32
Retrieve company industry sectors from Yahoo Finance and merge them with ESG scores to analyze and visualize ESG score distribution by sector using Python.

Portfolio selection - The Markowitz model6:56
Explore classic and extended portfolio selection theory by Markowitz, focusing on the minimum variance portfolio, efficient frontier, and Sharpe ratio to balance return and risk.
Example - Minimum variance portfolio of two stocks19:19
Learn to build a minimum variance portfolio for two stocks by estimating returns and the covariance matrix. Use SciPy to optimize weights and obtain the 69/31 allocation with 5.7% volatility.
Implementation: Minimum variance portfolio of SMI20:24
Apply Markowitz theory to build a minimum-variance portfolio for the Swiss market index using Yahoo Finance data, compute covariances, and optimize weights under constraints.
Markowitz goes green - The portfolio selection model extended with ESG1:47
Explore the Markowitz model extended with ESG, minimizing portfolio volatility under constraints for maximum return and value, with theta_p equal to the weighted sum of theta_i and a trade-off.
Implementation: Minimum variance portfolio of SMI with ESG26:39
Construct a minimum variance portfolio for the SMI with ESG, incorporating ESG constraints into the MVP, optimizing allocations, and visualizing how ESG scores influence risk, return, and diversification.
Efficient frontier calculation as function of ESG score11:47
Demonstrates calculating the efficient frontier by minimizing portfolio volatility under a constant return and ESG constraint, plotted as a risk–return frontier.

Requirements

Some basic knowledge in Python as well as finance would be helpful, but is not mandatory

Description

Have you ever wondered how web scraping works and what added value you can gain from the extracted data? Plus the opportunity to get more information by analyzing data in an end-to-end project in Python? Then this is the right course for you!

In this course “Web Scraping and Data Analysis with Python – A Practical Introduction Applied to ESG Rating and Stock Market Data,” I will provide you a comprehensive introduction on how to apply web scraping and data analysis approaches in Python using financial data from Yahoo Finance as a test-bed. After a brief insight into HTML, we will cover the essentials of web scraping using BeautifulSoup, a well-known library in Python, in order to extract a series of ESG (Environmental, Social, and Governance) scores along with their underlying stock prices.

In turn, we will move to the data analysis part, where we will conduct portfolio optimizations based on the Markowitz model. There, you will get familiar with the approaches on how to perform a classic portfolio optimization and then enhance it by incorporating the ESG scores obtained from the web scraping part. This will help you gaining an understanding on how to make data-driven decisions in finance, and to assess the effect between the risk/return profile and sustainability factors.

By the end of this course, you will obtain the skills to build your own web scraping and data analysis projects in Python, enabling you to extract valuable information from the web and turn it into actionable insights.

Who this course is for:

This course is for anyone who is interested in applying python for web scraping and data analysis associated with financial, i.e, stock market and ESG rating score data

Web scraping and data analysis with Python

What you'll learn

Explore related topics

Course content

Introduction2 lectures • 6min

Web Scraping with Python7 lectures • 59min

ESG Data4 lectures • 47min

Data analysis using the ESG scores6 lectures • 1hr 27min

Summary and outlook1 lecture • 3min

Requirements

Description

Who this course is for: