AWS Data Engineering Project: End-to-End RDS to S3 with Glue

STEP 1: Install VS Code

1. Search: VS Code download

2. Download and install

During install:

• Add to PATH

• Add "Open with Code" options

1. Open VS Code once to verify

STEP 1: Install WSL

1. Open PowerShell as Administrator

Run:

wsl --install -d Ubuntu-22.04

1. Restart system if prompted

STEP 2: Setup Ubuntu

1. Open Ubuntu from Start menu

2. Create username: unix

3. Create password: unix

NOTE:
Password will NOT be visible while typing

STEP 3: Update System

Run:

sudo apt update

sudo apt upgrade -y

STEP 4: Install Python

Run:

sudo apt install python3 python3-pip -y

STEP 5: Verify Python Installation

Run:

python3 --version

STEP 6: Verify Python Location

Run:

which python3

Expected:
Python should be located at:
/usr/bin/python3

STEP 1: Install Extensions in VS Code

Open VS Code

Install:

• Python (Microsoft)

• WSL (Microsoft)

STEP 2: Open Project in WSL

Open Ubuntu terminal

Run:

cd ~

mkdir -p sample

cd sample

To open VS Code in WSL:

code .

STEP 3: Verify WSL Connection

Terminal should look like:

unix@machine:~/sample$

STEP 4: Set Default Terminal (if needed)

In VS Code:

Ctrl + Shift + P

Search:

Terminal: Select Default Profile

Choose:

• WSL / Ubuntu

STEP 5: Verify Python

Run:

python3 --version

which python3

STEP 1: Prepare Working Directory

Open Ubuntu

Run:

cd ~

mkdir -p data0to1_pipelines

cd data0to1_pipelines

mkdir azure

cd azure

STEP 2: Copy Project Code

If your zip file is in Windows Downloads:

• Unzip it

• Copy the contents into the above WSL path using File Explorer

Then run:

code .

This will open VS Code from the azure (or respective cloud) folder.

STEP 3: Give Execute Permission

Run:

find . -type f -name "*.sh" -exec chmod 755 {} \;

STEP 4: Install Required Utilities

Run:

sudo apt install unzip dos2unix -y

STEP 5: Convert Shell Script Format

Run:

find . -type f -name "*.sh" -exec dos2unix {} \;

STEP 6: Going Forward – Open Project in VS Code

Run:

cd ~/data0to1_pipelines//

code .

Please keep an eye on the billing cost so that we don't overrun into huge billing costs.