Master AWS Lambda Functions for Data Engineers using Python

Learn to set up a Windows development environment for building end-to-end ETL pipelines with Python and AWS, including WSL Ubuntu VM, Docker Desktop, and Visual Studio Code remote development.

Master the basics of PowerShell on Windows 10 and 11, from launching and customizing its settings to using SSH for remote access and essential commands like dir and mkdir.

Set up Ubuntu on Windows using WSL, launching PowerShell to install the Ubuntu-based VM, choosing Ubuntu 20.04, and rebooting to complete installation with optional GUI support.

Install ubuntu 20.04 on windows via wsl, set up a user, and connect with wsl to explore the linux home directory and run commands like ls, cd, and pwd.

Install and configure docker desktop on Windows, including enabling WSL2 and validating Docker via PowerShell, for data engineers using Python.

Validate docker on Windows using PowerShell by running a hello world container. Pull images, list containers, and clean up with docker rm and docker rmi.

Review docker desktop resource configurations on Windows using WSL2, validate memory usage across containers, and plan cloud-based fallback via AWS Cloud9 while cloning the repository to set up the environment.

Install visual studio code on windows by downloading the executable, running the installer, choosing user installation and adding to path, then launching and opening a folder as workspace.

Install the remote development extension kit in Visual Studio Code on Windows via the extensions wizard, enabling remote SSH, WSL, and containers, and access files remotely.

Install Python 3.9 on Ubuntu under WSL, verify availability, set up a Python 3.9 virtual environment with distutils, and activate it for AWS Lambda development.

Configure a Windows-based environment for Python and AWS using WSL with Ubuntu, Docker Desktop, Visual Studio Code with the remote development extension pack, Python 3.9, and distutils.

Set up the ghactivity-aws project folder in Visual Studio Code, install the Python extension, run a hello world to validate, and prepare a Python 3.9 virtual environment.

Validate that your machine uses Python 3.9 for the project via the integrated terminal in vscode, check available versions (3.6, 3.8, 3.9), and install 3.9 if needed.

Create a python 3.9 virtual environment named ga-venv in the project folder using python 3.9 -m venv, validating python 3.9 and automatically activating in visual studio code.

Set up a Python 3.9 virtual environment and manage project dependencies with two files, requirements_dev.txt and requirements.txt, to build AWS Lambda data pipelines using boto3, requests, pandas, pyarrow, and s3fs.

Install and validate aws cli in your environment, either globally or in a virtual environment, then use aws cli commands to interact with aws services and verify outcomes.

Use your personal AWS account to take this course to avoid permission issues and ensure smooth learning, with costs kept low and a clean, closed account afterward.

Create and verify an AWS account, add a credit card, sign in to the AWS console, and review services and authentication steps including multi-factor authentication.

Set up an AWS IAM user with administrator permissions by creating a group, applying administrator access, adding users, and backing up credentials for AWS CLI and Python boto3.

Configure the aws cli with a dedicated ghactivity profile, input access key id and secret key from a csv, set us-east-1, and validate access by listing s3 resources.

Configure the aws cli to use a default profile by exporting aws_profile, enabling commands like aws s3 ls and mb without --profile and ensuring access to the correct account.

Learn to use the datetime module and timedelta to increment an hourly file name in the format YYYY-MM-DD-HH.json.gz, converting between strings and datetime with strftime and strptime.

Validate boto3 setup in Python to interact with AWS services and read S3 buckets using a profile. Demonstrate switching between ghactivity and default accounts by toggling the aws_profile environment variable.

Set up a Python 3.9 project in Visual Studio Code, configure AWS CLI and boto3 access to S3, and establish a Jupyter-based notebooks environment for training.

Explore gharchive.org github activity data, learn to download hourly and daily json.gz files with wget, and read them with pandas for aws lambda data engineering projects.

Download gharchive data for a specific hour using Python requests and save it as 2022-06-05-0.json.gz in a local data folder. Prepare for data review with pandas in the next lecture.

Review ghactivity data with pandas by reading a gzipped json file using read_json with lines and orient=records. Inspect the dataframe's shape, columns, dtypes, and nested fields (repo, payload, org).

Learn to manage s3 with python boto3 in a jupyter environment, including setting aws_profile, listing buckets, and creating new buckets with a unique prefix.

Create a DynamoDB table named jobs to store job details, using boto3 to manage the table with a job_id hash key and operations like put_item, get_item, and scan.

Create a DynamoDB table named job_run_details to store each job run with a composite key: job_id (hash) and job_run_time (range), enabling ordered access and quick retrieval.

Use Python date time objects and time delta to increment by one hour and generate the next file name for the JSON data ingestion workflow.

Fetch first run details from DynamoDB with boto3, compute baseline days, and incrementally download ghactivity data files to s3 using a three-day catch-up strategy and date-based file naming.

Implement incremental load logic for the next file using boto3, AWS Lambda, and Python. Track last run with DynamoDB and compute file names via date arithmetic.

Explore S3 concepts, including buckets and objects, and learn to create buckets with the AWS CLI and upload files using cp with recursive listings.

Explore how to upload files to s3 as objects using Python boto3, including configuring aws_profile, creating an s3 client, and using upload_file with bucket and key.

Learn to write data into S3 as objects with boto3 put_object, using byte streams, bucket, and key. You will also read the object with get_object and decode.

Learn to save gh activity data to AWS S3 by uploading byte content with a boto3 s3 client using put_object, then validate with pandas read_json from S3.

Convert a datetime to an integer Unix epoch using Python's time and datetime modules, via time.mktime on a time tuple, yielding an integer timestamp for use in job run details.

Save each pipeline run to the DynamoDB table job run details, recording job id, start time, and bookmark details after uploading the file to S3.

Validate that the uploaded file resides in s3, inspect the job run details, and use pandas to verify the data content and the captured last run file name and status.

Validate end-to-end core logic for fetching files from gharchive and uploading to S3, updating dynamodb job run and bookmark details. Prepare the modularized app for deployment as a lambda function.

Learn to start with AWS Lambda and deploy your first Python 3.9 function to ingest data from gharchive to S3. Understand passing arguments, custom handlers, and resource settings.

Learn how AWS Lambda enables serverless, event-driven code execution and create a function from scratch in Python 3.9, configure a basic role, and test with a Hello World demo event.

Pass json payloads to AWS Lambda, interpreted as Python dicts in the handler, and test the event in logs and the function response.

Apply custom module and handler names in AWS Lambda by editing runtime settings; rename lambda_function to app and lambda_handler to lambda_main, deploy, and test to ensure proper import.

Learn how to interact with AWS services from Lambda by creating an S3 client with boto3, listing buckets, and configuring IAM permissions for read-only S3 access.

Explore how AWS IAM roles grant service-to-service permissions, and how users and user groups manage external access with least-privilege policies, illustrated by Lambda and S3 access.

Pass the bucket name as a lambda event argument to list objects in an S3 bucket, extract object names from contents, and return object_names after deploying and testing.

Define and read environment variables in AWS Lambda with Python and the os module, retrieving the bucket name via os.environ.get('bucket_name').

Review and customize AWS lambda resources from general configuration, including memory, storage, and timeout, with CPU allocated to memory. Deploy and test changes while managing dependencies through development and packaging.

Set up a local development environment for AWS Lambda using Python 3.9, create a virtual environment, install requests and boto3, and validate end-to-end deployment flow.

Develop and deploy a python-based AWS lambda function that uploads content from gharchive.org to S3 using requests and boto3, with an environment variable bucket name and proper event-context handling.

Build a clean AWS Lambda deployment zip by including app.py and dependencies, excluding the ald-venv folder, and verify with unzip -t before deployment as Lambda function.

Learn to deploy a Python 3.9 AWS Lambda function from a zip file, configure S3 permissions and environment variables, test in the console, and note Docker images for large code.

Develop a Python-based lambda for downloading gharchive data using wget and requests, access response.content, and upload to S3, while setting up and validating a Windows development environment for AWS deployment.

Set up a Visual Studio Code project in WSL with Python 3.9 for AWS lambda development, installing boto3 and requests and creating app.py for local testing.

Learn how to fetch data from gharchive.org and upload it to aws s3 using python boto3, with a bytes payload, aigithub bucket, and a lambda-ready core workflow.

Update app.py to deploy as an AWS Lambda function with a Lambda handler, importing requests and boto3 to fetch from gharchive into s3, using an environment variable for the bucket.

Build a zip package for a python app to deploy as an aws lambda function by installing requests in the current folder, including app.py and dependencies while excluding boto3.

Deploy a python-based application to AWS Lambda using a zip file, S3 upload, and console configuration. Learn environment variables, IAM roles, and testing to validate Lambda deployment.

Review how to set up a Python development environment for AWS on Windows with WSL and Visual Studio Code, deploy to Lambda, validate functionality, and rate with feedback on Itversity.

Learn to build, test, and deploy a Python Lambda function that ingests data from gharchive to AWS S3, with modular code, a custom Lambda handler, zip deployment, permissions, and monitoring.

Modularize ingestion workflow into a lambda-ready project, organizing code under app and util with bookmark.py and ghactivity_ingest.py, using env vars for bucket and folder to ingest gharchive files to S3.

Create and activate a Python virtual environment and select a version for validation. Use Python -c, import, and driver programs to invoke functions such as greet from modules like app.hw.

Validate the data ingestion workflow by running a Python CLI to upload files from gharchive to S3, using environment variables bucket_name and folder and an aws profile.

Develop a lambda handler to ingest data to s3, invoking ghactivity ingest to s3 and returning job_run_details with status 200; validate via cli and s3 upload checks.

Build and bundle a python lambda deployment by creating a dependencies folder, installing requirements.txt, and packaging the app and dependencies into ghactivity-aws.zip for AWS Lambda deployment.

Create and deploy a python 3.9 aws lambda function from a zip file uploaded to s3, then configure it via the console and test the ghactivity ingester workflow.

Configure a custom lambda handler by using app.lambda_ingest as entry point and set environment variables, including python path, bucket_name, and folder, for zip deployment and Dynamo DB and S3 permissions.

Attach Amazon s3 full access and Amazon Dynamo DB full access policies to the lambda function's role, or create an inline policy for restricted access, then test.

Increase AWS Lambda timeout and memory to prevent task timeouts; adjust the general configuration, set memory to 1 GB and timeout to 30 seconds, then retest to confirm successful execution.

Review how to monitor and troubleshoot lambda functions using CloudWatch, inspect log groups and streams, diagnose permissions, memory, and timeout issues, and verify deployment details.

Explore the 256 MB zip file size limit for deploying AWS Lambda functions and learn to use custom Docker images by building one with your app and dependencies.

Automate building and packaging AWS Lambda functions using a shell script that creates a dependencies folder, runs pip install -r requirements.txt, zips the code, and prepares deployment via CI/CD.

Master AWS lambda functions with custom docker images, exceeding zip size limits, by building a python3 runtime image, validating locally with docker and curl, then deploying via ECR.

Create a custom docker image for AWS lambda by writing a Dockerfile, copying requirements.txt, installing dependencies with python 3.9, copying the app, and setting the lambda handler app.lambda_ingest.

Launch a terminal, validate the docker file, and build a custom docker image named ghactivity-aws with the python 3 run-time and dependencies from requirements.txt.

Create and validate a custom docker image ghactivity-aws for lambda by running a container, then verify requirements.txt, app folder, and installed dependencies.

Run the lambda_ingest validation inside a docker container using python -c, configure aws_profile and other environment variables, copy the .aws folder, and upload test data to s3.

Run a custom docker image to validate a python aws lambda workflow by mounting the host .aws credentials and configuring aws_profile, bucket name, and folder, then verify s3 operations.

Validate AWS Lambda functions locally with Docker and curl by port forwarding 8080 to 9080 and posting to the Lambda invocation endpoint.

Create a private AWS Elastic Container Registry repository named ghactivity-aws and push a custom docker image, enabling lambda functions to run from ECR-hosted images.

Learn to push a custom Docker image to an AWS ECR repository named ghactivity-aws, including login, tagging, and pushing steps, preparing for a Lambda deployment.

Create a lambda function from a custom docker image stored in AWS ECR, configure environment variables and permissions, and set 1 GB memory with a 30-second timeout.

Run and validate a Lambda function built with a custom Docker image, fix Mac M1 architecture, redeploy, and verify results via S3, CloudWatch logs, and sanity checks.

Automate building and pushing a custom Docker image for AWS Lambda, clean up old images, log in to ECR, tag, and deploy the latest image via AWS CLI.

Automate updating a lambda function to the latest custom docker image in ecr by listing images, retrieving function details, and running a script to rebuild, push, and reconfigure.

Configure s3 event notifications and lambda triggers to auto invoke when json files land in s3, enabling csv-to-json conversion and future parquet storage.

Switch between ghactivity and analytics aws profiles to validate s3 access and illustrate aws_profile usage for seamless account switching.

Set up a python project to explore AWS Lambda triggers and S3 event notifications, creating a virtual environment, installing boto3, and testing S3 access from VS Code.

Set up datasets to explore triggering aws lambda on s3 events, using s3 notifications or lambda triggers. Learn bucket management, dataset cloning from GitHub, and recursive data copy to s3.

Learn to list AWS S3 buckets and objects with Python using boto3, creating an S3 client, calling list_buckets and list_objects, and extracting bucket names and object keys.

Learn to filter S3 objects with boto3 list_objects using a prefix, iterate through contents and object keys, and understand bucket, object, and prefix concepts for S3 events and event triggers.

Configure AWS S3 event notifications to trigger AWS Lambda when objects are placed in the airetail bucket, enabling processing of new data via a Lambda function.

Create a simple AWS Lambda function for s3 event notifications, grant s3 full access via its IAM role, and configure triggers using add trigger or the notification wizard.

Configure an s3 trigger for a lambda on put events in airetail with retail_db prefix; test with s3 cp and view CloudWatch logs for event structure.

Master how to retrieve S3 object key from a lambda event triggered by S3, using Python to navigate event records and S3 object details, and verify results in CloudWatch logs.

Learn to read csv data from s3, convert records to json using the json module and boto3, and write the results as json files within a lambda function.

Trigger an AWS Lambda function from S3 to read CSV content, convert each record into a JSON string, and write the results back to S3.

Convert CSV data to a JSON document and upload it to S3 as a JSON file with UTF-8 encoding. Validate Lambda trigger by checking CloudWatch logs and S3 event notifications.

Configure s3 event notifications to invoke a lambda that converts csv to json when files arrive in the airetail retail_db prefix. Validate results via cloud watch logs and s3 triggers.

Deploy a Python lambda that reads json data from S3, transforms it to parquet, writes back to S3 via a custom Docker image, and validate end-to-end with events bridge.

Convert json data from S3 to parquet with pandas by reading json lines and dropping payload, then write snappy parquet back to S3; plan chunking for lambda deployment.

Learn to convert json to parquet in chunks using pandas, processing 10,000-record batches to overcome memory limits in aws lambda, and partitioning outputs by year, month, day, and hour.

Create a DynamoDB jobs entry for ghactivity_transform, set the environment profile, and add a job item with id, description, and is_active to enable json to parquet processing.

builds core logic to transform json to parquet in ghactivity_transform.py via transform_to_parquet, reading from s3, chunking 10,000 records, writing to tgt_folder, returning last_run_source_file_name and last_run_tgt_file_pattern.

Create a ghactivity transform to parquet wrapper to deploy the json-to-parquet transformation as a lambda function, wiring file_name, bucket_name, and tgt_folder, and saving job_run_details.

Validate a wrapper function that converts json to parquet by setting environment variables, authenticating with AWS, invoking ghactivity_transform_to_parquet, and preparing a lambda handler with logging and a 200 status.

Define lambda_transformer to extract last_run_file_name from event, invoke ghactivity_transform_to_parquet to transform json to parquet, return status 200 with job_run_details, and deploy to aws lambda using docker image for curl validation.

Validate Lambda handler locally by transforming json to parquet via curl, using a Docker workflow that builds, runs, and tests ingestion and transformation in a container, then pushes to ECR.

Push and tag the rebuilt docker image to AWS ECR, log in, and export the AWS profile, then update the ghactivity-ingestor lambda to use app.lambda_ingest via image configuration, and test.

Deploy the ghactivity transformer lambda using a container image, set S3 and DynamoDB permissions, and configure 4 GB memory with a 3-minute timeout for processing json to parquet with snappy.

Validate an AWS Lambda data transformer end-to-end by checking S3 uploads, parquet data via pandas, and DynamoDB tables before building an orchestrated pipeline to transform data.