Data Mastery: Unifying Big Data and Cloud Technologies

MongoDB is a NoSQL database with strong demand in industry and freelancing, serving as an alternative to SQL for web and mobile apps, and enabling data-driven projects.

Explore MongoDB as a flexible, schema-less database using documents and collections for mobile, web, games, analytics, and data science, with durability.

Meet Mohammad Ahmed, a cloud and big data engineer who shares real-world experience with databases, migration and ETLs, cloud infrastructure, and DevOps, guiding you through a fruitful MongoDB journey.

Compare SQL databases and NoSQL Mongo database, perform CRUD in MongoDB with Node.js and Python, and build a simple Django app using MongoDB plus a Spark ETL pipeline.

Engage with modular lectures that teach MongoDB operators and techniques, followed by hands-on quizzes and detailed solution videos, culminating in projects to apply your data mastery.

Learn to unify big data and cloud technologies by building Django with Mongo CRUD apps and a Spark ETL pipeline that reads CSVs into MongoDB, with mini projects and quizzes.

Explore how SQL databases work, their limitations, and how NoSQL solutions like MongoDB address them, illustrated by an employee schema and dependent-child table example.

Highlight how SQL schemas rely on joins and table normalization, then explain NoSQL stores data as key-value pairs with no dedicated schema for flexible, single-row access and no memory leakage.

Install MongoDB on your local machine, set it as a service with default data and log directories, and explore the Mongo shell and Compass GUI.

Finish installation and restart the system, then set the environment variable for MongoDB by adding the bin directory to the path to access Mongo from the terminal as a service.

Explore how SQL terms map to MongoDB concepts, comparing databases, tables vs collections, rows vs documents, and columns vs fields, to build intuition for both systems.

Learn to perform basic MongoDB operations by using the mongo shell to create and switch databases, view existing databases with show dbs, and run use commands.

Master MongoDB commands: create and switch databases, and list them with show dbs. See that a database appears only after adding data, then drop it after verifying the current database.

Explore MongoDB collections and database operations by creating a school database, adding collections for teachers, students, staff, and lectures, and using show dbs, show collections, and drop commands.

Master the create operation in MongoDB by learning single and batch insertions into a collection, reducing requests and improving efficiency, with quizzes to reinforce concepts.

Create documents in a MongoDB collection by inserting JSON objects with db.collection.insert, showing no fixed schema. Use find to view results and compare NoSQL flexibility with SQL schemas.

Learn how to create multiple documents in a collection using insert many, compare it with single insert, and observe IDs and acknowledged responses in MongoDB for efficient bulk insertion.

Perform a hands-on quiz on create document operations by creating a database and a collection named create quiz, then insert student marks across quizzes for math, English, science, history, French.

Create quiz collection in a Mongo database and insert multiple student documents with fields for student name and marks in maths, english, science, history, and french.

Practice inserting documents individually into a collection, one at a time, by writing the query for the create operation. Rewrite json objects to reinforce hands-on practice before the next video.

Insert documents into the quiz collection using JSON objects to record student names and marks in maths, English, science, history, and French, then verify with a find query.

Master basic create operations by inserting single or multiple documents into a collection, complete with quizzes, and prepare for advanced crud operations and operators in upcoming modules.

Explore update operations in a NoSQL context, focusing on MongoDB documents, handling missing or extra fields, updating field values, and the role of update operators to be covered later.

Explore MongoDB's schema-free updates: filter-based edits update only selected fields, with upsert inserting a new document when no match exists.

Update documents in a collection by applying multiple criteria, crafting queries that modify fields such as name, gender, and age, and verify changes with a find operation.

Perform an update operation to add English, Science, and French marks for student Mark using data from the Create Quiz Collection, preparing for the forthcoming solution discussion.

Update a student’s record in the create quiz collection by applying English, science, and French marks using the object id, then verify with find and pretty.

Practice update operations by writing queries to set quantity to 100 for skew a, b, c, and update orders to 50 where quantity is ten and rating is 4.5.

Update the quantity to 100 for the document with sku ABC 123 using a db.update query, illustrating replacing the entire document and the future use of set operators.

Demonstrate the basic update operation by filtering with find and updating documents to 50 where quantity is ten and rating is 4.5, noting ratings inside a matrix.

Recap the basics of update operators, how to set filtering conditions for documents, and the current limitation on updating a single field, with more to come in future modules.

Explore the basic read operation by learning how to set reading conditions, read the entire collection, and apply simple filtering to match exact criteria and extract useful information.

Learn to read data with criteria in MongoDB by applying filters to find documents, name Ahmed or age 20, and see how an empty filter returns all records.

Perform read document operations to retrieve and print data from the create quiz collection, with simple and formatted outputs, and print Flavin's marks while filtering by French and history scores.

Explore read operations on the quiz collection, using find queries, pretty formatting, and specific filters for student names and subject marks to retrieve targeted documents.

practice basic read operations by querying a documents collection from the repo file, printing all documents, formatting the results, and filtering for quantity ten with a rating of 3.5.

Execute a read operation to fetch all documents, print them plainly or in a formatted view, and filter results where quantity is ten and matrix ratings equal 3.5.

Master the basics of crud operations with MongoDB, focusing on the basic read operation and syntax, and prepare for advanced crud topics in upcoming modules.

Explore the delete operation within the crud cycle by selecting specific documents and executing deletion, laying a basic foundation for more detailed practice in later modules.

This lecture shows deleting documents by replacing find with remove, using an empty query to delete all, or filtering by name, age, or ObjectID.

Practice delete document operations by removing records with nine marks in French, then delete documents named John, and finally clear the remaining data using the Create Quiz dataset.

Perform delete operations in MongoDB to remove documents from the create quiz collection with nine marks in French and with name John, then clear the collection with an empty query.

Load data from delete quiz.txt into a collection and write queries to delete documents with quantity ten. Then remove all remaining documents and review the solution in the next video.

Delete documents in MongoDB by using a remove query to target quantity equals ten, then delete all remaining documents and verify with find.

Strengthen your understanding of CRUD basics and MongoDB operators through the basic delete operation. Look ahead to the next module where actual MongoDB is explored and drawbacks are overcome.

Explore query and projection operators in this module introduction, building on basic MongoDB CRUD and showing how to query data and shape results using comparison operators.

Learn how the eq operator in MongoDB performs equality checks, querying embedded fields like item.name and arrays like tags to filter inventory documents.

Master the gt operator to query documents with quantities greater than a value in MongoDB, and apply gt and gte for 20 and above in inventory.

Master the less than operator in MongoDB queries by filtering inventory documents by quantity with $lt and $lte, using the MongoDB shell's db.inventory.find syntax.

Explore the in operator to search for elements within a list and simplify queries when filtering documents by quantities like 15, 20, or 25.

Apply the $ne operator to filter inventory documents whose quantity is not 20, using db.inventory.find, to illustrate not equal to comparisons in the query and projection operators.

Explore the not in ($nin) operator in Mongo queries, filtering documents where fields like qty are not in a specified list, returning items such as 25 and 30.

Explore the and operator for multi-condition filtering, applying conditions like quantity not equal to 20 and tags containing b, with examples using db.inventory.find.

Apply the MongoDB or operator to filter documents by multiple conditions, such as item name equals AB or quantity equals 20, using DB.inventory.find.

Explore the not operator in query logic to negate conditions, such as retrieving documents whose quantity is not 20 by wrapping the equal condition in not and brackets.

Explore the $exists operator in MongoDB to filter documents by field presence, combine with value checks (quantity equals 20) and tag matches, using and logic.

Use the type operator to verify field types and filter out anomalies, returning records where quantity is string, double, or integer in MongoDB.

Explore the evaluation query operators with the $expr operator to filter documents by comparing spent and budget using gt, lt, eq, and elt in a monthly budget collection.

Explore how the mod operator filters documents by remainder, enabling even/odd grouping and divisibility checks in field queries.

Explore how the MongoDB $text operator enables flexible text search by indexing a field before querying, using multi-word terms, exact phrases, negation, and case sensitivity.

Master the $all operator for array fields, requiring all specified elements to be present and using it to filter documents by array content with examples like appliance, school, and book.

Apply the $elemMatch operator to query documents with arrays of objects, drilling into nested elements to find any element meeting criteria like size and number within quantity.

Learn how the $size operator filters documents by the number of elements in an array, using fields like qty and tags in a find query to include matching documents.

Learn the dollar projection operator to limit array data and return the first matching element, applying semester 1 and grades.mean filters, with MongoDB indexing guiding the results.

Use the slice operator to limit the number of elements returned, unlike the dollar operator which returns the first match. Learn applying slice to grades, semesters, and nested elements.

Explore query and projection operators through a quiz on equal checks, filtering documents by quantity 15, code 000, and tags containing C, with solutions discussed in the next video.

Create a MongoDB collection, insert records, and use the $eq operator to filter documents by quantity 15, code 000, and tags containing c.

master query skills with the greater-than operator by filtering a new collection to find documents with quantity over 15 and size over 12, where size resides in the item field.

Demonstrates using the $gt operator to filter documents by quantity greater than 15 and by nested item.size greater than 12, including creating and querying a collection.

Explore the gte quiz on query and projection operators by fetching documents with quantity at least 15, size at least 12, size equal to 2, and sizes at least 10.

Create a collection, load data, and use $gte to fetch documents with quantity at least 15, size at least 12, sizes containing 2, and sizes at least 10.

Solve the in operator quiz by filtering documents with quantity 15, 25, or 35; sizes 2 or 22; codes 1, 23, or 000; and tags a, b, or d.

Use the in operator in MongoDB to filter documents by quantity, sizes, code, and tags, including nested fields like item.size and item.code.

Explore the less than operator through a practical quiz using quiz.txt, selecting documents with quantity less than 15 and size less than 12, and the next video discusses the solution.

Create the light quiz collection, insert data, and run find queries to retrieve documents with qty less than 15 and item.size less than 12.

Practice the less than or equal to operator via a quiz with the LTE quiz.txt file, filtering documents by quantity at most 15 and size at most 12.

Learn to query a MongoDB collection using the $lte operator to extract documents with quantity at most 15 and size at most 12, including nested fields like item.size.

Demonstrate using the $lte operator to filter documents by an array field, checking item.sizes for any value less than or equal to 10 so only matching documents appear.

Practice not equal to checks in a quiz dataset using the $ne operator, retrieving documents where quantity != 20, size != 10, and sizes do not contain 5.

Apply the $ne operator to filter quiz documents by quantity not equal to 20, size not equal to 10, and items whose sizes do not include five.

Solve a quiz on query and projection operators using $nin to filter documents by quantities, sizes, codes, tags, and names not matching given values.

Create the nin quiz collection, insert documents, and query with the $nin operator on the qty field to return documents whose quantity is not 15, 25, or 35.

Apply the not in ($nin) operator to filter documents by size not equal to 2 or 22 and by tags not containing A, B, or D.

Apply the $nin operator in the mongo shell to filter documents by excluding the name values x, y, and ab, and retrieve all others from the quiz collection.

practice the and operator through a quiz on logical operators, filtering documents where quantity is not 15 and size is 15, and where tags contains D and name is AB.

Practice constructing MongoDB queries with the and operator to filter by quantity not equal to 15, size 15, and quantity greater than 15, with tags contain a.

Practice the or operator to filter documents by conditions such as name ab, quantity not 15, or size 15. Explore or-based queries on tags and codes.

Explore data mastery by using MongoDB find queries with the $or operator to filter documents by nested fields like item name, size, quantity, and tags.

Demonstrates writing a query using the $or operator to return documents where (quantity equals 15 and name equals Abby) or (quantity equals 20 and name equals CD).

Tackle a quiz on the not operator in query and projection tasks, solving with not equal, not in, and not contain conditions to filter documents.

Explore how to build a MongoDB collection, insert data, and use the not operator with or conditions to filter documents by name, quantity, or size.

Learn to use not, not in, and or operators to filter documents by tags and name, negating conditions such as tags not containing d or c and name not ab.

Explore how to filter documents using the not operator, combining conditions with or: code not equal to 123, quantity greater than 15, and tags not containing a or d.

Explore the exists operator through a practical quiz, filtering documents where qty or quantity is present, where code is not present, and where quantity is 20.

Create and populate a collection, then use the $exists operator to find documents with the qty field present and filter for items where code is missing and qty equals 20.

Practice the expr and projection operators by writing queries that compare budget to spend, filter by food or drinks, and prepare for the solution discussion in this quiz.

Explore solving quiz queries in MongoDB using $expr with operators such as eq, lt, gt, and in, to filter by budget and spent and by category.

Explore the mod operator by querying mod quiz.txt to retrieve documents with even quantities, then filter for odd quantities, and anticipate the solution discussion in the next video.

Learn how to use the $mod operator in MongoDB to filter even and odd quantities, including collection creation, data insertion, and using remainder checks and negation for accuracy.

Master the text operator with a quiz that filters documents by subject keywords, views, and exact case-sensitive coffee matches, including cafe con leche with XYZ or ABC.

Learn how to create a MongoDB collection, index a text field, and use the $text operator in queries to filter by subject, author, and views.

Master the all operator by writing queries on sample data to filter documents by tags such as school or book, and by quantity and colors brown and orange.

Unify big data and cloud technologies by exploring query and projection operators, including $in and $all, to filter documents by tag values.

Learn to query MongoDB using and operator and $all to find documents where quantity size is greater than six and colors are brown and orange, reflecting a dynamic schema.

Explore query and projection operators with $elemMatch in a quiz using El match quiz dot txt to filter results by numeric ranges and color blue or green.

Learn to query MongoDB arrays with $elemMatch, filtering for large size, elements greater than 30 and less than 80, and colors blue or green.

Use lm match operator to filter quantity by size, number, and color, ensuring size is large, number is greater than or equal to 90, and color is blue or green.

Use the size operator to filter documents by tag and quantity counts: identify two-tag documents and three-quantity documents, paving the way for the solution discussion in the next video.

Create a MongoDB collection, insert data, then query documents with two tags using the $size operator, and query documents with three quantities in the array.

Explore update operators, especially the current date field operator, which sets the last modified field to the current timestamp for documents meeting criteria.

Leverage the $inc update operator to increment fields like quantity and metrics inside the object across many documents that meet a condition, such as quantity thresholds.

Explore how the MongoDB $inc operator increases or decreases numeric fields with positive or negative values. Apply updates to decrement quantity and orders by five for documents with id 2.

Learn how min operator assigns a smaller value to a field only when the new value is less than the current one, demonstrated on id two high score.

Explore the max update operator in MongoDB, learn how it updates a field only when the new value exceeds the current one, and see parallels with the min operator.

Explore the mul update operator, which multiplies a field by a constant. Use it to adjust price or quantity, with or without conditions, via update many.

Explore the rename update operator to rename fields across documents with update many, turning old names into new ones in a NoSQL context, and learn how missing fields are ignored.

Explore how the $set operator updates documents, including setting quantity, updating nested details (model, make) and replacing arrays like tags, with new values.

Discover how to use the $set operator with update many to modify documents and nested array elements via dot notation, updating tags and ratings.

Learn how the unset operator removes fields from MongoDB documents, including nested fields, via update many, with examples that drop the mobile field, set values to empty, or remove name.first.

Demonstrate the addToSet operator to treat arrays as sets in an update, ensuring unique elements are added and duplicates are avoided.

Master the MongoDB $pop update operator to remove the first or last element from an array, not arbitrary elements, with practical examples and usage notes.

Learn to apply the pull operator to remove array elements by criteria across documents, using document-level and array-level conditions, with practical examples of fruits, vegetables, and the results field.

Learn how the MongoDB push operator adds elements to an array across all documents using update many, updating the scores field with new values like ten.

Learn how the each operator, used with push in update operations for MongoDB, adds multiple elements to an array as individual entries rather than a single combined one.

Master the position operator in update workflows, using the each and push operators to insert array elements at a chosen index, including starting from index zero.

Apply the sort operator to an update flow, using ascending (1) or descending (-1) to sort array data after push updates, and observe sorted results with find.

Explore update operators through a quiz, writing queries to increment quantities for items with over 20 orders, double orders by rating above 4.2, and modify or delete specific SKUs.

Load the data and dump it into a MongoDB collection, then use updateMany to increment the quantity by two for documents with more than 20 orders.

Explore update operators by using updateMany with the $set operator to zero out quantity for documents matching the school value ABC 123123.

Apply update operators to remove the matrix field from documents with skew equal to ABC123 using $unset, demonstrating how the same data outcome can be achieved through different approaches.

Tackle an update operators quiz by using a file to add school to all documents' tags, set chair for id three, and add bottle, cable, and mic.

Apply the add to set update operator to add a school tag to every document's tags array, avoiding duplicates, and observe resulting documents now containing book bag, appliance, and school.

Find the document where id equals three, then apply update operators in data mastery by using update many with the push operator to add chair to the tags array.

Learn to use push and add to set to add bottle, cable, and mic to all documents with update many, and why add to set may do nothing if present.

Install Node.js on your local machine, configure the environment PATH, then connect MongoDB with Node.js and perform basic CRUD operations.

Install and verify Node.js, confirm the version with node -v and path; install VS Code, create a project folder, and run node demo.js to see Hello world.

Install node.js and configure VSCode, then spin up a MongoDB cluster and connect to it, optionally using Atlas to configure databases and connect Node.js to read and write data.

Create and connect to a MongoDB Atlas cluster on AWS (North Virginia) with the free tier, then build databases and documents in the cluster from Node.js with CRUD operations.

Explore MongoDB Atlas cluster management, from free tier limits to dedicated clusters. Create database users with authenticated access and roles, and connect via application or shell using the Atlas URL.

Control database access in MongoDB Atlas by configuring network access, using cidr notation or 0.0.0.0/0 to limit ip addresses. Keep the url unchanged and require credentials for all connections.

Connect to a MongoDB Atlas cluster, browse collections, and create databases and collections manually or via scripts, then insert, find, and delete documents within collections.

Connect node with MongoDB by installing the MongoDB driver for Node.js, forming the Atlas URI, and using an async main function to connect, list databases, and close the client.

Connect Node.js to MongoDB and list databases from the admin client, iterating over the results to print each database name.

Explore inserting single and multiple documents into a MongoDB collection from Node.js, using insertOne and insertMany, and review the resulting acknowledged IDs.

Learn to read documents from a MongoDB collection using Node.js, employing find with a cursor, then iterate or convert to an array and print the results.

Update MongoDB data with Node.js using updateMany and the increment operator. Apply a simple condition to increment a field by five and verify the updated documents.

Learn how to delete documents in MongoDB with Node.js using deleteMany, set a condition to remove multiple documents, and verify the deletion counts.

Connect MongoDB with Python using Atlas and PyCharm, creating a Python db and a dummy collection in Atlas, then set up PyCharm to work with Python MongoDB.

Configure mongodb atlas with pycharm and establish a python script connection using pymongo's MongoClient. Connect to the python db database and dummy collection Rthy after installing pymongo and dns.

Learn to insert data into MongoDB using python by performing insert many with documents via a client-connected collection, and understand when to use insert one as an alternative.

Learn to read documents from MongoDB using Python by defining a read_docs function, querying with find using a condition, and iterating results for display.

Explore updating MongoDB documents with Python using the update operator. The tutorial updates documents where doc one two value is three, setting the value to Apple, with implementation details.

Learn to delete MongoDB documents with Python by deleting documents matching the condition doc one two equals to Apple, using delete_many or delete_one on a collection.

Install Django with pip, set up a PyCharm project, and configure it to use MongoDB with the Django driver. Create Django project and app, preparing MongoDB integration in next video.

Learn to create a Django project and a Django crud app in PyCharm, run the server, and verify the project structure for MongoDB integration.

Configure Django to work with MongoDB Atlas by defining a posts model and updating settings, then run makemigrations and migrate to create the collection in MongoDB.

Learn how Django migrations integrate with MongoDB Atlas, using a JSON field to store data while recognizing the role of schema, and why a structured approach is preferred.

Explore how Django maps URLs to views, creating add, read, update, and delete post functions, and wiring them in the main URL configuration to handle browser requests.

Demonstrates the create operation in Django using MongoDB, showing the add post view in PyCharm, returning an HTTP response, and testing data insertion with Postman into MongoDB.

Send data from Postman to a Django view via post and access request data, with CSRF exempt for demonstration, leading to MongoDB storage.

Create a post object with a title and description in a Django view, save it to MongoDB via the configured settings, and verify the insertion in MongoDB Atlas.

Read data from MongoDB in a Django view and return a JSON response for postman. Use the posts model, fetch all documents, and expose title and description with dot notation.

Exempt the request from CSRF tokens and retrieve the id and new title from the request, then update the MongoDB document via Django and return an HTTP response.

Delete a MongoDB document using Django by exempting CSRF, retrieving by id, deleting the post, and returning a deleted message, while noting Django's limitations with MongoDB.

Explore spark with Mongo by building a simple ETL pipeline that extracts data from the CSV, transforms it, and loads it into MongoDB, using Databricks community edition and a cluster.

Learn how to set up a Databricks cluster and install essential libraries to connect PySpark with MongoDB using the Mongo spark connector and Spark XML for data parsing.

Explore loading a csv file in a Databricks environment, read it with spark.read.csv, display the dataframe, and prepare to dump the data into MongoDB.

Create a configured spark session, read data from a file, and write a dataframe to a MongoDB cluster using the spark connector, demonstrating overwrite and append options.

Explore data mastery through a practical journey across AWS core services, IAM, S3, Lambda, Glue, CloudFormation, CLI, VPC, RDS, DynamoDB, Redshift, and tools like CloudWatch, SQS, SNS, and Step Functions.

Explore how Amazon Web Services enables scalable, serverless computing by letting you focus on your business while AWS handles servers, storage, and backups.

Learn how to create your AWS account, including prerequisites and email verification, 12 months free quota, sign-in, and troubleshooting payment issues to access your own AWS console.

Explore the AWS management console, switch regions, access the cloud shell, and locate services like S3. Review billing details and prepare for analytics, compute, and databases categories.

Explore how AWS services are categorized into analytics, streaming, ingestion, database, and networks, illustrated with an IoT solar panel use case that streams sensor data to storage for real-time analysis.

Learn how AWS glue, lambda, and athena transform daily to monthly sales data in s3, enabling cross-city analysis and targeted queries.

Learn how AWS IAM policies control access to resources like an S3 bucket by defining groups such as HR and developers, and attaching policies to grant or restrict permissions.

Explore how to manage AWS access with IAM policies by creating users, assigning policies, and granting bucket-level permissions to an S3 resource.

Explore how Amazon S3 stores files as objects in buckets, enables versioning for restores, and uses storage classes like standard, infrequent access, and intelligent tiering to balance access and cost.

Discover how to enable and create an s3 bucket, manage storage classes and lifecycle policies, and use versioning to restore previous data with show version, variant IDs, and delete markers.

Explore how S3 lifecycle policies optimize costs by automatically moving objects between storage classes from standard to infrequent access and archive, and by managing versioning, deletions, and intelligent-tiering.

Explore how S3 event triggers automate Lambda, SNS, or SQS actions on bucket uploads, changes, or deletions, with examples of processing files and notifying users.

Learn how AWS Lambda runs code in response to events with serverless compute that auto-scales, triggered by S3 and other services, with configurable memory, storage, and timeouts.

Trigger a lambda from an S3 event and parse the event payload in Python to extract the file name and bucket name, using CloudWatch logs to view event data.

Discover how lambda layers enable sharing libraries like pandas and pi mysql across multiple functions by packaging dependencies into a zip and attaching a layer for s3 data transformations.

Create a lambda layer for Python dependencies, package Pi MySQL into a zip via cloud shell, upload to S3, and attach the layer to a lambda function.

Learn to create and attach a compatible Python 3.7 runtime layer to an AWS Lambda function, add a Pi MySQL layer, and import libraries via the function.

Explore AWS Glue, an integration service that moves data from multiple sources, runs etl jobs to transform and load data, and uses crawlers to infer schemas and catalog data.

Learn to create and run a visual AWS Glue ETL job that ingests CSV from S3, infers schema, maps data types, and outputs parquet to S3.

Learn how AWS Glue crawlers infer schema and metadata, create Glue catalog and tables from an S3 data source, and enable SQL queries in Athena.

Explore how simple CloudFormation templates deploy AWS services in one go, automating resource creation and linking, with ready templates like WordPress on a single EC2 instance.

Learn to deploy an S3 bucket via CloudFormation by uploading a template, enabling versioning, and deploying a stack that creates the bucket and multiple resources.

Explore infrastructure as code and automation with AWS CLI, from installing and configuring the tool to listing S3 buckets and deploying AWS Glue jobs via the command line.

Explore virtual private cloud (VPC) concepts in AWS, including private networks, subnets, availability zones, route tables, security groups, and network access control lists for secure data deployment.

Create a VPC with public and private subnets, configure route tables and internet gateways, and add gateway endpoints for S3 and DynamoDB to enable private access.

Explore Amazon RDS, a popular relational database that simplifies setup, operation, and scaling in the cloud. See how tables organize products, customers, and orders with security, backups, and durability.

Create an AWS RDS database via the console by configuring a subnet group and VPC, selecting MySQL and the free tier, setting credentials, and enabling automated backups.

Access an RDS instance with AWS Glue by updating inbound security group rules. Use a Python shell job with PyMySQL to connect, create a table, insert rows, verify via CloudWatch.

Explore how AWS Redshift serves as a data warehousing solution for large-scale analytics, enabling real-time queries, dashboards, and machine learning integrations with data from diverse sources.

Create a Redshift cluster in the console with a dc2 large, one node, load sample data, and run queries in the editor, including joins between category and events.

Discover how AWS DynamoDB stores NoSQL data with encryption, offers global availability for disaster recovery, and streams to S3 and AWS Glue for media and text analysis.

Create and access a DynamoDB table by defining a partition key, adding items with id and attributes, and using scan or query with filters; build an orders table.

Discover AWS security and the shared responsibility model, including KMS, Secrets Manager, and IAM, plus SQS for secure, encrypted producer and consumer messaging.

Learn to use AWS KMS and Secrets Manager in the console to encrypt data across Redshift, RDS, and S3, manage keys, securely store credentials, and enable secret rotation.

Create an AWS SQS queue in the console, choosing standard or FIFO, configure visibility timeout, max 256 KB, 14-day retention, enable SSE-SQS, then send and poll messages.

Learn how AWS Simple Notification Service (SNS) topics notify multiple users via alerts when an S3 object creation triggers a pipeline failure, enabling data engineers to act quickly.

Create a standard SNS topic, subscribe an email endpoint, confirm the subscription, and send emails via publish message with a subject and body.

Build an AWS based ETL workflow that ingests daily S3 CSV files, creates year/month/day partitions in the output bucket, and uses Athena to query inserted, updated, and deleted records.

Execute a lambda triggered by s3 uploads to partition data by upload day, write outputs to a destination bucket, and email the uploader when csv schema mismatches, enabling s3 queries.

Show how to build a serverless data pipeline using S3 buckets, a Python lambda, SNS notifications, and a pandas layer to partition output by year, month, and day.

Configure lambda with S3 and SNS permissions, test the pipeline by uploading files to input and verifying outputs in the warehouse output bucket, while handling schema validation and email alerts.

Query S3 data with Athena after building a Glue crawler, creating a database and ETL_assignment table, and filtering orders by date, user, and status to identify top returning users.

Discover how cloud computing enables storing and analyzing massive data and learn how Azure enables turning data into insights for business success.

Leverage cloud computing to scale data processing on demand, paying only for resources used. Access centralized, available data securely with RBAC, enabling innovation and agility while focusing on core competencies.

Explore Azure services across infrastructure as a service, platform as a service, and software as a service, with storage accounts and databases powering cloud solutions.

Learn to create a Microsoft Azure account, choosing between student free access and a general free option with card verification, then complete signup steps.

Learn how to create a resource group in Azure, understand subscription selection, set a region, and prepare your environment for deploying services in this course module.

Master Databricks as a big data analytics platform, learn storage accounts, and use Data Factory for data transformation and integration, while securing keys with Key Vault across cloud services.

Explore what Databricks is, its key features, and why it powers big data analysis. Learn to create and manage clusters, store and read data, and build visualizations to ignite insights.

Enable collaboration with Databricks, the cloud-based big data analytics platform on Azure, uniting data engineers, data scientists, and analysts with multi-language support and seamless integration with Azure services.

Learn to create an Azure Databricks service, select subscription and resource group, choose a region and standard pricing, then launch the workspace and explore core features.

Explore how a Databricks cluster provides essential compute resources—RAM, cores, and scalability—to enable data engineering, ETL pipelines, streaming and batch processing, and analytics on the cloud.

Learn to create a Databricks computing cluster on Azure Databricks, choosing single-node or multi-node configurations, and configuring driver and worker types, memory, cores, and auto-scaling with termination settings.

Explore Azure Databricks storage options by comparing DBFS and Hive Metastore, learn to upload files to DBFS, and create tables in Hive Metastore with a default database.

Explore Azure Databricks workspace features, including workspace, repos, shared folders, and user management, then read data from DBFS and Hive Metastore to transform in notebooks.

Read data from dbfs into a data frame using spark.read, with csv options like infer schema and header, then create a temp view and visualize the average salary by department.

Read data from Hive Metastore using SQL, locate the table in the default database, then visualize age versus average salary by department and gender in Databricks.

Write data into Databricks internal storage (dbfs or hive metastore) from notebooks by converting pandas data frames to spark data frames and using overwrite, append, or ignore modes.

Master Azure Databricks by implementing advanced data processing through scheduled workflows that run notebooks, ingest data from URL, and manage clusters and libraries.

Explore what an Azure storage account is and how it secures and accelerates data access. Compare standard and premium types, redundancy options, and when to store structured versus unstructured data.

Explore what Microsoft Azure Storage is and how it enables modern data storage scenarios, storing images, audios, videos, log files, and configuration files, with millisecond access for analysis.

Discover why Azure Storage is a popular cloud storage solution, offering durability, high availability, redundancy, security with RBAC and encryption, scalability, and accessible tools like Storage Explorer and http/https access.

Learn how storage account performance differs between standard and premium tiers, comparing HDD and SSD storage, costs, latency, and when to choose each based on data access patterns.

Master Azure storage redundancy by storing multiple copies across primary and secondary regions, including locally redundant storage, redundant storage, geo redundant storage, and geo zone redundant storage.

Compare locally redundant storage, which creates three copies in the same data center for a low-cost option, with zone redundant storage that copies data across region zones for higher availability.

Mastering Azure storage accounts explains geo redundant and zone redundant storage. Primary region holds three copies in one data center; secondary region holds three, delivering higher availability than primary redundancy.

Learn how Azure blob storage stores unstructured data as binary large objects in containers, with scalable sizes and metadata for managing images, videos, and text.

Explore Azure blob storage access tiers hot, cool, and archive and their cost implications, lifecycle rules, and tier conversion to optimize data usage, latency, and retention.

Master Azure file storage to share application files across remote teams, organizing file shares, directories, and tools within a storage account for secure access to logs and media.

Master Azure queue storage to store millions of messages in scalable queues, secured by role-based access control and http/https access, with 64 kb per queue limits.

Explore azure table storage as a scalable, NoSQL table in a storage account, where each row has a different schema and data is stored as key-value pairs for non-relational data.

Learn to create and configure azure storage accounts for databricks, choosing subscription, resource group, region, and storage options like standard lrs, hot tier, and blob storage containers.

Master Azure Data Factory fundamentals by exploring its core features, ETL concepts, integration runtime types, linked services, datasets, pipelines, activities, data flows, and triggers for end-to-end data management.

Explore how Azure Data Factory, a robust and scalable cloud-based data integration service, calls Databricks, blob storage, SQL Server, Functions, and Key Vault to move, transform, and schedule data pipelines.

Mastering Azure Data Factory shows how ADF is an ETL tool by extracting data from diverse sources, transforming with drag-and-drop no-code blocks, and loading into databases and storage.

Learn to create an Azure data factory by selecting a subscription and resource group, naming the factory, choosing a region and version 2, and configuring Git integration before launching Studio.

Explore how integration runtime controls how Azure Data Factory connects to data sources and destinations, with self-hosted runtime for non-Azure platforms and Azure integration runtime for Azure services.

Learn to create a new integration runtime in Azure Data Factory, choose runtime types, name it, and configure data flow runtime compute from small to custom with a time-to-live.

Learn how linked services bridge Azure Data Factory with external data sources like Databricks and Blob Storage, enabling data access via the integration runtime.

Create and manage datasets in Azure Data Factory by linking to Databricks or Blob Storage, where a dataset references a table or file through a linked service and integration runtime.

Explore Azure Data Factory activities as the basic units of data tasks, covering data movement, transformation, analysis, and common pipeline actions like copy data and data flow.

Master Azure Data Factory pipelines, a unit of ETL tasks that group extract, transform, and load activities to move data from Databricks notebooks to storage accounts.

Build an Azure Data Factory pipeline that runs a Databricks notebook to read data from a URL, stores it in hive metastore, and copies it to blob storage as CSV.

Discover how Azure Data Factory triggers automate end-to-end data pipelines by scheduling runs without human intervention, using schedule, tumbling window, and storage event triggers.

Mastering Azure Data Factory teaches how schedule triggers fire pipelines at future times with daily, weekly, or monthly frequencies, linking multiple pipelines to a single trigger and vice versa.

Explore Azure Data Factory's tumbling window trigger, which runs pipelines at a defined duration and start-end window for real-time data, with one-to-one pipeline mapping and configurable retries.

Mastering Azure Data Factory event based triggers respond to blob storage events, auto-trigger pipelines via Event Grid, enabling many-to-many relationships between files, datasets, and pipelines.

Learn to create a schedule trigger in Azure Data Factory, configure tumbling windows, concurrency, delays, and retries, and attach the trigger to pipelines.

Master data flows in Azure Data Factory with visual drag-and-drop blocks that generate code executed on a Databricks cluster, enabling column selection, joins, and filters from source to sink.

Master Azure Key Vault to secure secrets, keys, and certificates, and integrate it with data factory, storage accounts, and data bricks to enable encryption and avoid direct key exposure.

Discover how key vault functions as a cloud service to securely store secrets like api keys, passwords, certificates, and cryptographic keys, and access them with encryption.

Centralize your secrets, keys, and certificates with Azure Key Vault, enforcing industry-level encryption, role-based access control via access policies, and detailed change logging.

Create a secure Azure Key Vault by selecting subscription, resource group, name, region, and pricing tier, then enable soft delete. Review vault details and understand storing keys, secrets, and certificates.

Master Azure Key Vault by creating and managing secrets for storage accounts and Databricks, using access policies and vault access control to grant read and write permissions.

Learn to create a linked service to access storage accounts via Azure Key Vault, securing secrets and permissions in Data Factory across subscriptions, and publish changes.