ETL using Python: from MySQL to BigQuery

This is a direct and to the point course that will get you quickly ETL'ing data from MySQL to BigQuery.

The lessons in this course are broken out into short How-Tos. Therefore you can take this course over the weekend and be ready to show off your skills on Monday morning!

Things that we will cover:

• Setup

  • Setting up a GCP Account

  • Credential and Authentication for security

  • Python Environment Setup

• Extract

  • Use Python to connect to MySQL

  • Use Python's pandas to export data

  • Python library usage for saving files to file paths

• Transform

  • Use Python functions to transform data

  • Use Python pandas to transform data

  • Use inline SQL during Extract for data transformation

• Load

  • Use the BigQuery Python library

  • Connect to BigQuery

  • Load data to BigQuery

  • Incremental Loads vs Truncate and Load

  • Other data handling options during Load

After taking this course, you'll be comfortable with the following pretty cool things:

• Connect to MySQL using Python

• Learn how to obscure your database credentials so you're not exposing them in your code

• Usage of the os module for the purpose of saving files and hard coding fewer things.

• Use both Python and the pandas library to transform data on the fly during the Transformation phase of your ETL

• Learn how to use GBQ's modules/libraries to make the loading of the data a very easy, straightforward task

Have fun, enjoy and keep growing!