Learn Data Engineering with Databricks on AWS Cloud

Explore amazon s3, a scalable, durable cloud object storage service with buckets and objects, integrated with aws for website hosting and software apps.

Explore the Glacier storage class in S3 and compare pricing and latency with standard storage. Learn to set Glacier or Glacier Deep Archive for objects via lifecycle or replication rules.

Create an S3 bucket on AWS, set up a retail_db folder structure, and upload diverse objects using drag-and-drop.

Enable bucket versioning in AWS S3 and set lifecycle rules to manage older versions, scoped by a prefix like retail_db, to prevent data loss and enable recovery.

Enable cross-region replication for an S3 bucket to achieve fault tolerance by creating a replica in another region, configuring an IAM role, and applying a replication rule with versioning.

Set up and configure AWS S3 using IAM user credentials, create an IAM user with administrator access, generate access keys, and configure the CLI to manage S3 buckets.

Learn to manage S3 bucket objects with the AWS CLI, listing, recursively exploring contents, deleting folders, copying directories across buckets, excluding files, and verifying results.

Discover how identity and access management enables databricks on aws by managing users, groups, roles, and permissions, with secure temporary access via sts.

Create an IAM user, enable console and programmatic access, attach administrator policies, and generate access keys for CLI use.

Learn to log in to the AWS management console using an IAM user, using the console sign-in URL to auto-fill the account ID and sign in as the IAM user.

Validate AWS IAM user programmatic access by configuring the AWS CLI with access key id and secret key, then verify access by listing S3 buckets and their contents.

Learn how AWS IAM identity-based policies work, including predefined and custom policies, attaching policies to users, groups, or roles, and the core elements of policy: effect, action, and resource.

Learn how to create and manage AWS IAM user groups, attach administrator and S3 read-only policies, and inherit group permissions for new users.

Learn to create AWS IAM custom policies with JSON, granting a user granular read, write, list, and delete access to a specific S3 bucket folder by policy assignment.

Discover how to integrate Databricks on AWS with S3, Glue Catalog, and IAM roles by creating groups, adding users, and configuring instance profiles to enable access to these services.

Create an IAM user group for Databricks developers on AWS, name it DB dev, and prepare to assign permissions and add users so they inherit group policies.

Create IAM users and add them to the DB dev group to inherit group permissions; enable console and programmatic access, auto-generated passwords, and download credentials.

Create an AWS s3 bucket for Databricks developers with admin-only access, configure permissions, and verify that non-admin users cannot access or list the bucket.

Learn how to grant inline policy permissions on an AWS S3 bucket for users in a group, and validate access by simulating sign-in and bucket operations.

Attach the AWS Glue Console Full Access policy to the DB dev user group to grant Databricks on AWS access to AWS Glue.

Upload json datasets to S3, organize into folders in the demo bucket, and enable the AWS Glue crawler to crawl through the files and folders.

Create an AWS Glue service role for Glue crawlers and attach S3 permissions so a dev user can run crawlers that catalog data into tables.

Learn how to create and run an AWS Glue crawler to create multiple Glue catalog tables from an S3 bucket, producing a database with tables derived from bucket folders.

Discover how Databricks on AWS provisions a single-node cluster that creates an EC2 instance and an instance profile to access S3 and the Glue Catalog for catalog tables.

Create an AWS IAM role on EC2 to generate an instance profile, attach permissions, and register it with Databricks so the cluster can access S3 and the Glue catalog.

Register the AWS IAM instance profile in the Databricks admin settings, attach it to the cluster, and ensure the cluster can access AWS services.

Attach an AWS IAM instance profile to a Databricks cluster to access S3, AWS Glue, and EC2, with options for new, running, or terminated clusters.

Grant S3 bucket permissions to the Databricks cluster via the instance profile by creating an inline IAM policy that allows list, get, put, delete, and put object acl on bucket.

Integrate a Databricks cluster with the AWS Glue data catalog by attaching an instance profile, granting Glue permissions, and enabling the Glue metastore.

Learn to set up a single node Databricks cluster for a local environment using Databricks Connect, including prerequisites, creating compute, selecting runtime 13.2, and choosing an m6 large node.

Install Databricks Connect on your local system by creating and activating a Python virtual environment. Then use pip to install the Databricks Connect client.

Configure Databricks Connect on your local system by generating a personal access token, collecting workspace details and cluster ID, then run the Databricks Connect configuration to finalize connectivity.

Install the Databricks command line interface with pip, generate a token, and configure the host in the Databricks cfg to connect to a Databricks workspace on AWS.

Learn how to interact with the dbfs using the Databricks CLI, exploring commands like ls, ls -l, cat, mkdir, cp, and rm to manage files and folders.

Learn to manage Databricks clusters with the Databricks CLI: list clusters, get details by cluster ID, view Spark versions and node types, and start, resize, or edit clusters.

Modularize notebooks in Databricks by creating macros and using the %run command to call functions across notebooks, such as app and second app.

Learn to run modularized notebooks as Databricks jobs on AWS by creating a notebook job, selecting the path, and running on a small cluster, to see hello world output.

Refactor your application into Databricks notebooks, organizing five files (write.py, util.py, read.py, process.py, app.py) into a workspace folder and splitting code into cells with environment variables and cluster setup.

Run multiple Databricks notebooks on a development cluster to validate, set environment variables, and transform json to parquet in DBFS.