AWS DynamoDB - Complete Guide (incl Schema Designing)

Here I explain why DynamoDB is recognized as a fully managed serverless database. Here you get to know the Primary key, Secondary key and Secondary Indexes for the first time in this course.

I walk you through with DynamoDB web console to show you what features are available there. Here I will create a table, add items to the table, create GSI table and etc. This will demystify the DynamoDB.

DynamoDB team released the NoSQL Workbench tool very recently. This is a great tool for beginner to play with DynamoDB. Even DynamoDB experts use this tool to design the initial schema. You can install this as a standalone application on your Mac, Windows machine. You even don't need a AWS account to use this tool.

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. I will show you how to download and configure an AWS account to this. After that will create a table and do some basic DynamoDB operations using CLI.

Achieving best results with a NoSQL database such as DynamoDB requires a shift in thinking from the typical relational database. Use the above best practices when modeling data with DynamoDB.

DynamoDB has two capacity modes and those come with specific billing options for processing reads and writes on your tables: on-demand and provisioned. Watch this video to learn more about the billing options for each capacity mode.

All of your data is stored on solid-state disks (SSDs) and is automatically replicated across multiple Availability Zones in an AWS Region, providing built-in high availability and data durability. We will discuss these concepts in this video.

DynamoDB needs to deploy enough RCU and WCU to handle your traffic to database. Having good understanding on this is vital because this highly influence your AWS Bill. 

DynamoDB pricing depends on your

• Throughput requirements

• Storage

• Backups

• Streams

and few other factors. When using DynamoDB each decision we take we take carefully after assessing its affect to performance and cost.

Taking into everything we learned in this section, I will explain a comprehensive pricing calculation. Here we cover how to calculate price when you have enabled DynamoDB auto scaling as well.

This is a good example to get started with DynamoDB. Here we have three, one:to many relationships and we model these in our schema three different ways. We will list down all the access patterns in our order management application and we will implement these access patterns in the rest of the sections.

This is your first encounter to modeling a database schema using NoSQL Workbench from start to end. After completing this video you will gain so much confidence about understanding a DynamoDB schema.

We will create a Inverted Index to query some of the one:to many relationships. To filter out data we  will create a Sparse Index. Also we will create a composite sort key here. Watch until he last section to learn how this composite sort key going to make our life easier.

This is the favorite feature of Workbench tool. I will show you how to generate code for any complex query for

• NodeJs

• Python

• Java

Clean and nice UI!

You would use a "where" condition to support this sort of query in relational db. How you can design your DynamoDB to support this sort of query without scanning the entire table or GSI?

Here we are going to design a multi player game which can be played on a social media platform. The challenge is schema should be scalable to millions of users.
As the general practice, the first step is to brainstorm and identify all the access patterns. Then we will design the key schema for our base table.

This example we use Python boto3 library. boto3 is the official AWS SDK for Python. Here we setup a Cloud9 IDE so that you don't need to download and configure AWS CLI or boto3 library on your PC.

I mentioned that DynamoDB does not have joins that a relational database has. Instead, you design your table to allow for join-like behavior in your requests. Here we design our schema to mimic a join behavior so that you will be able to query both user and his games in one request.

Not all the games are in open state. When a player won the game, that particular game cannot be started again. So how a player can find open games to join? That's exactly what I am going to show in this lecture.'

You may not understand  transactional writes well, if I explain it just on power point slides. The problem we are going to solve on this video is a perfect candidate for transactional writes. If you were to solve this without transactional writes, your logic would be ridiculously complex.

Instead handling the conditions on your code, you can write them on your query itself. DynamoDB can handle this conditions well. I will teach you how to apply percentiles here.

If there is a many:to many or one to:many relationship in your data, that suggest you would need a secondary index. Because GSI helps to query your data in the other way around.

ex : - Find all the games a user played, and find all the users who played a particular game

Remember to clean up your Cloud9 IDE. You are running a EC2 instance here and you will be billed hourly if you left it alone.

We are going to build a schema for a social application.

Imagine you are building a mobile application to upload photos. Users can upload photos and his/her followers can view the photos. Users can follow other users.

Users will be able to browse all photos for a particular user, starting with the most recent photo.

Finally, users can react to a photo with one of four emojis — a heart, a smiley face, a thumbs up, or a pair of sunglasses.

Here we do the key schema design for the base entities in our application. These are Users, Photos, Reactions, Friendships.
We will create a Cloud9 environment to execute our python scripts. In the next section we will create the db and load dump data.

We will create the base table and bulk load a data dump to the db. You can download all the scripts needs to complete this tutorial in the downloadable section.
I will explain the Python models we created, so it is easy to cast our db responses to cast into these models. This is a powerful feature compared to NodeJS, because there you have to work with third party libraries to work with models.
Also I will show the schema which was created on Workbench tool. It is always easy to understand a schema when it is created on Workbench tool.

Here we retrieve multiple entities of data in a one single request. This is how you mimic join queries on DynamoDB. There is a misconception that there is no relationships in DynamoDB. It is wrong. DynamoDB is all about relationships :)

You can't implement this access pattern using our base table. When you want to query from an attribute which is not happen to be the partition key of the base table, you want to create a GSI table.

One core tenant of relational model is normalization of data in which you avoid duplicating data in multiple places.
But with DynamoDB, you often want to denormalize your data. However, there are times when denormalization can complicate your data model.

In this video, you will see how to use partial normalization and the BatchGetItem API call to handle this situation.

You would use a transaction to make sure the changes are applied to the balances of both customers rather than just one.

When a one user follows another user, we need to create a new Friendship item and at the same time we need to increase the followers and following count of respective users.

How we can use Transactional writes to save our life?

Another scenario where we will use transactions is when a user add a reaction to someone elses photo. We need to create a new Reaction item and need to update the reaction type count in the particular image.
I am sure you will be figure out where you can use Transactional writes on your DynamoDB schema after completing this section.

Sort keys can add more power to the querying capabilities of DynamoDB. However, you must use some forethought.
Remember to employ string templates that take advantage of the begins_with operator and to order the string by least specific data to most specific.

We explore real-life examples of optimizing tables for data retrieval with sort keys. You use sort keys to not only group and organize data, but also to provide additional means for querying against items in a table.

DynamoDB by default paginates the results from Query operations. It doesn't have first class support to implement normal pagination we see on a relational database.
As an example there is no by default way to know the total number of pages. Many online examples only talks about fetching the next page. Here I do a follow along with you and implement pagination using a React app, where you can go back and forth. You can apply this same logic to any front end say Angular, Vue and etc.

Here we create a lambda function with granular security permissions. You can download the sample code attached. No more handling connection pools or freeing up resources. Just API calls ;)

This is a bonus video. There is purely on how to fix CORS issues on AWS API Gateway. This is a pain point for many beginners, where they get feel frustrate. Here I discuss many important things which, sometimes experience AWS developers are not aware of.

This is a complete example how you implement a meaningful pagination using React. Core idea here is to storing the LastEvaluatedKey inside an array. This would be helpful when you go back. Full react code you can download here.

In this example, we'll show how to maintain a global leaderboard with a dataset in DynamoDB. A leaderboard is a common need for data applications.
In this walkthrough you will understand how to use write sharding combined with a scatter-gather query to satisfy the leaderboard use case.

AWS provides a leaderboard example using game scores in the DynamoDB docs. However, that example uses leaderboards within multiple different games, rather than a global leaderboard as we have here.

The previous solution has a one problem. Now we have to avoid hot partitions. This is not a big problem to solve if you identify it before hand.