Transform Data into Insights with Dagster and Deepnote

In this video, I show you how to use the Deepnote platform as a setup-free development environment. We cover the different types of machines available, including GPU options for working with large amounts of data. I also demonstrate how to select from various programming environments, including different versions of Python and TensorFlow optimized versions. Additionally, I show you how to customize your environment with a Dockerfile and how to enable and disable connections to the machine running your notebook. I also discuss the Linux terminal feature and the integrations available, such as connecting to Google Drive, Amazon S3, and various databases. Overall, Deepnote makes it easy to connect to different data sources and focus on your work, rather than setting up infrastructure.

Learn how to use Dagster, a cloud-native orchestrator, to build and manage your ETL pipelines. In this video, I will explain what an orchestrator is, why you need it, and how Dagster can help you monitor your jobs, schedule them, and ensure they are running smoothly. I will also show you some of Dagster key features, such as the task-based workflow and the single pane of glass data platform. Additionally, I will share tips on how to use Dagster Slack community for support.

I'll also be mentioning that there are two versions of Dagster: the open-source version that you can run on your own servers and manage yourself, or the cloud version which is a fully managed service. The cloud version is a good option if you don't want to spend time on deploying and maintaining infrastructure, and it also has a free trial. Additionally, I'll be talking about the documentation for Dagster, which is comprehensive and includes tutorials for getting started and deploying the open-source version.

In this video, we will be discussing the use of Metabase as a business intelligence tool. It allows us to connect to different data sources and in this case, we will be connecting to a Postgres database which will already contain pre-processed data. The main goal of the data pipeline is to process, transform and make the data accessible and useful for the database and easy to visualize. The great thing about Metabase is that it is a low-code tool, which means non-technical people can use it to create their own visualizations and do their own data aspiration.

In this video, I will be discussing the use of Google Cloud specifically for the Cloud SQL for PostgreSQL. I will mention that signing up for an account will give you $300 in free credits which will be used for the course. I will also mention Tableplus, a useful application that I use as a database explorer that can connect to cloud or local databases and can be downloaded for different operating systems.

In this section of the course, I will be showing you how to build a solution architecture for our data pipeline. We will be using a Postgres database and Google Drive as input sources. The focus of this course is on building the pipeline and developing its structure, rather than on DevOps. We will be breaking down the pipeline into three stages: extract, transform, and load. The extract stage involves getting data from the source, the transform stage involves applying any necessary transformations such as normalizing or cleaning data, and the load stage involves saving the data to another database. We will also be visualizing the data using Metabase. The development process is divided into three sections: Business Logic Development, Dagster development, and Production. The reason for separating the development environment is to improve the speed of execution. We will be using Deepnote for fast development of the business and transformation logic.

Everything is connected to the same data source, even in the early stages of development, to ensure that the functions will run on all data sets and prevent errors. We will be using Dagster ops and jobs for local development, and deploying them to the cloud for production. It is important to use separate databases for development and production to prevent interference. We will be using a local Postgres database for speed and simplicity.

To begin, we will create a trial account on the Google Cloud platform. You will need a credit or debit card, but don't worry, you will start with $300 of free credits and your card will not be charged if you exceed that amount. After signing up, you can either create a new project or use the default project provided. Next, we will search for the Cloud SQL API to create our database. After creating the instance, we will specify a few settings such as the instance ID, password, and development version. We will also select a region that is closest to us and has the low CO2 option. Lastly, we will enable the public IP and add the network to connect to the database. We will also set up Table Plus to connect and verify the database connection.

In this video, we will continue building our data pipeline by setting up Deepnote and creating a new project in our account. We will also configure the Postgres connection and Google Drive integration, which we will use later on in the course to generate data for a hypothetical company that grows and sells berries. They sell their produce in-person at markets, through a main shop on their farm, and even accept crypto as a form of payment. Our data pipeline will collect and normalize all this transaction data for easy use and visualization.

In this video, I explain in detail how the dummy data has been generated by the provided Jupyter notebook.