Introduction to Microservices with Python (with Next.JS)

This lecture will layout what you will learn in this whole section.

This includes all the topics that we will cover in this section, as well as some pointers about how to get the most out of the material and what approach to take when studying.

Students will learn about what makes the Micro-services Architecture so special and so useful.

In this lecture we will look  at the full developer and user benefits that come with Micro-services Architecture.
For the developer it will be the ease of independent development workflow as well as ease of scalable deployment with fault tolerance.
For the user it will be the added reliability as well as great responsiveness of the system even when the demand is high.

A quick summary on the Benefits of the Microservices Architecture.

In this lecture we will learn about how to start our FastAPI server which will host our microservices and we will write and run our first simple microservice.

NOTE: the full source code for this lecture has been attached here.

You first Exercise which will help you with making sure your environment is set and that you are able to write your first micro-service and then add some simple functionality to it.

NOTE: solution and explanation is provided in the next lecture.

This is a solution and explanation for the [exercise Set#1]

NOTE: The source code has been attached to this lecture

In this lecture we will look at an overview of how Microservices communicate and we will look at simple examples of RESTful protocol.

In this lecture we will look at an overview of how Microservices communicate and we look in more detail of how the RESTful protocol works.
We will learn the distinction beteen the body of the request as it pertains to input data for POST request methods as well as how we can add input parameters to the request URL for GET request methods.

We will also briefly look at JSON Objects used in the our API.

In this exercise you will practice using query variables in your GET requests. Proper endpoint design and good API implementation is what Microservicesis all about.
You can use either the classical "?name1=value1&name2=value2" syntax for the endpoint URL or the "/{name1}/{name2}" syntax (or even better use both)

NOTE: solution and explanation is provided in the next lecture.

This is a solution and explanation for the [Exercise Set#2]

NOTE: The source code has been attached to this lecture

In this lecture we will quickly look at how to process JSON Objects in Python using the Pydantic model.

This will help us parse JSON entities in a simplified and Object Oriented manner.

In this lecture we will explore what we need to know to design and write a Microservice.
We will look into RESTFul API design including familarity with HTTP succes and error codes.
We will also look into CRUD operations and how they map into HTTP operations/methods.

In this lecture we will write our first full microservice which will cover the CRUD functionality for a User.
This is a starting point for exploration of the different elements that go into writing robust Microservices such as proper endpoint creation, proper data object definition and of course covering of the requirements.

NOTE: the full code for the service has been attached to this lecture.

In this lecture we go over testing our User Microservice's endpoints using cURL.

In this lecture we go over testing our User Microservice's endpoints using POSTMan.

In this lecture we go over testing our User Microservice's endpoints using PYTest.
We go further with creating a testsuite which is a collection of test methods which get executed one after the other to create a testing context and provide a test report at the end of the test run.

The main advantage of using a testsuite is that we can encaposulate a number of tests which build on top of each other to create a collection of tests that with a single execution can target and test all of our endpoints together.

NOTE: the full code for the service has been attached to this lecture.

This is a Next.js Reactive UI tutorial which will give you an introduction to Next.JS which we will use in some of our exercises and solutions to demonstrate UI-based clients to our Web-Services.

If you do know Next.JS then this is a gentle introduction to this topic which should be sufficient to learn it, and then understand and follow all of our UI-based examples and solutions.

NOTE: the full code for the service has been attached to this lecture.

In this exercise you will practice working with the User Microservice by adding new endpoints to enhance the service as well as creating additional test suite methods to test the new endpoints.

You will use both query parameters for searching as well as use JSON body that is sent to the server to create new functionality.

Bascially the idea is to have teh student be fully capable to designing and writing service methods for any occasion.

NOTE: solution and explanation is provided in the next lecture.

This is a solution and explanation for exercise #1 from [Exercise Set#3]

We cover here a bit more about swagger as well as go into meta-thinking when trying to code a solution.

NOTE: The source code has been attached to this lecture

This is a solution and explanation for exercise #2 from [Exercise Set#3]

We cover here the pytest fixture used for test-suites to provision test data.
We look into the fixture written for this test suite and also look at the tests in the test-suite.

NOTE: The source code has been attached to this lecture

This is a solution and explanation for exercise #3 from [Exercise Set#3]

We look at the CRUD and UI separation and go over what pages we will need to create.

1. We look at what kind of navigation we will have within our pages
2. We go over how to define and configure the Microservice Server Gateway
3. We create the application scaffolding by creating all of our page stubs

At the end of this lecture we will have all of our pages defined and static navigation wired.

NOTE: The full source code for Exercise Set#3 has been attached to this lecture.

This is a solution and explanation for exercise #3 from [Exercise Set#3]

In this Part we will build a fully functional and reactive Add New User Page and wire it to our Microservices API

1. We look at how to properly develop a Next.JS client page
2. We go over how to properly handle communication errors and show them to the user
3. We learn about the Router to programmatically control page transitions

At the end of this lecture we will have a fully functional Add New User Next.JS page.

This is a solution and explanation for exercise #3 from [Exercise Set#3]

In this Part we will build a fully functional and reactive List Users Page and wire it to our Microservices API

1. We look at how to properly develop a Next.JS client page
2. We go over how to pre-load data in the page using a DOM hook
3. We will hook up the abliity to delete users from the page

At the end of this lecture we will have a fully functional List Users Next.JS page.

This is a solution and explanation for exercise #3 from [Exercise Set#3]

In this Part we will build a fully functional and reactive Edit User Page and wire it to our Microservices API

1. We look at how to properly develop a Next.JS client page
2. We go over how to pre-load data in the page using a DOM hook

At the end of this lecture we will have a fully functional Edit User Next.JS page.

In this lecture we look into what scalability mean in terms of I/O requests and responses frmo multiple clients.

1. We will look into the concepts of Non-Blocking IO and why we need to have this in our Microservices.
2. We will also look into worker-threads and how scaling of multiple tasks can be achieved.

In this lecture we will go into details how async/await pattern works within the Python eco-system. We will learn about the following:

1. Coroutines
2. Python Event Loop
3. Worker-Threads
4. Efficient non-blocking I/O
5. asyncio Python library.

Note: We have also attached a PDF with a summary reference of some of the most useful asyncio functions.

In this lecture we will look at three (3) examples of asycn/await (non-blocking I/O) code to help us better understand how to make code more scalable and efficient:

1. Async based File I/O which outputs progress to the terminal - this is a great example of spaning multiple concurrent tasks in a single-process environment.
2. Async based Microservices to helps us with making our microservices more efficient and scalable.

Once you understand those examples, you are ready to make your microservices efficient and professional.

NOTE: The source code has been attached to this lecture

In this lecture we will look at another example of the async/await pattern but this time in a more realistic micro-service setting.

We simulate here a slow database connection and scale a solution with a mock client test that can run N threads to test and benchmark the scaled (better) solution.

Note: The code for this lecture has been attached.

In this exercise set the student will scrutinize the User Service code with an eye for code areas that should be changed to take advantage of the async/await pattern.

Additionally, the student will look critcially at the way our in-memory database code is structured and start thinking about how to abstract it away from the service.

Note: we have attached the latest User Service code for student's convenience.

In this exercise the students will write code to test and benchmark as multi-end point microservice.
We will create an aggregator micsroservice which will acts as a proxy to another set of end-points in a separate microservice.
The student will:
1. Write the aggregator (the proxy)
2. Write the slow service end-points (the endpoints being bench-marked)
3. Modify the pervious mock client to accept a --path cli argument to target specific end-points with our client bench-mark tool.

Please note that this is an **extra** exercise which is similar to what we did in lecture #34. So this is a slight modification of that code but based on a new set of starter code.

NOTE: the starter code has been attached to this exercise.

In this section introduction the students will look at the need to abstract persistence away from a microservice.

The idea introduced here is that we need proper separation of concerns and this will not only improve our archiectture and performance but will also put us on a path of flexible solution scaling.

Simple warmup exercise for this section.
We will asses the User Service from perspective of persistence abstraction.

In this lecture we will explore how good the current User Microservice is with regards to thread-safety. This is bascially a solution/discussion based on [Exercise Set #6].

Here we explore the issue, we test it, and we then provide one possible fix for the thread-safety issue we have found in the User Microservice.

NOTE: The code for this lecture has been attached for your convenience.

In this lecture we experiment with our thread-safe solution from last lecture to see how it fairs with 4 worker processes when running the uvicorn server

In this lecture we look back at our 'thread-safe' solution from lecture#39 and find a better way to address the critical section issue within the context of a fastAPI (uvicorn) server.

After this lecture we are ready to start abstracting our our repo into a standalone persietence solution for our microservices.

In this lecture we will look closer at the User Microservice we have so far  and why we absolutely need to create a proper Database abstraction from perspective of Separation of Concerns, and Pluggability/Interoperability.

We also tackle the isssue of Non-Blocking I/O and thread-safety in our contract interface.

In this set of exercises the student will explore proper architectural appoach to writing DB code.
We will concentrate on proper interface and contract abstraction as well as creation of specific implementations such as simple in-memory implementation and RDIS implementation as well.

NOTE: the interface code has been attached with this lecture

In this part of the solution we enhance the interface Contract for our user data repo.
We add error definitions as well as better documentation for the contract.

NOTE: The Enhanced Contract Interface has been attached with this lecture.

In this part of the solution we use the new Interface Contract to implement In-Memory Database

We also refactor the user microservice to use the new exernal database implementation.

NOTE: The Code for both the DB implementation and the refractored User Service has been attached with this lecture.

In this lecture we will look into REDIS - a super fast in-memory database solution that is a leading industry standard.
We will learn about REDIS before we do the [Solution#1 Exercise Set#7 - Part #3] in which we will implementa a REDIS version of our data base repo.

In this lecture we go over what ACID is and how it applies to REDIS (and databases in General)
We bascially control proper transactionality of our requests.

NOTE: We have attached a summary of REDIS commands as well as ACID Transactions Summary

In this lecture the student will learn about the keyspace anatomy of the REDIS server.

We will explore how to map sets into specific keys and then how to map elements of sets into individual objects belonging to that set.

REDIS gives us great flexibility and allows us to create our own sets of keys thus defining specific data domains such as Users or Orders etc...

We will need to understand this before we write the code for our User data.

In this lecture we go over the REDIS implementation of our user repo.
In this code we pay specific attention to proper transactionality for some of the methods to ensure proper ACID transactionality when we have more than 1 atomic operation (2 in fact) and we need to ensure that both operations follow ACID rules.

NOTE: The code for this lecture has been attached. We have also attached the REDIS commands summary in PDF format for the student's  convenience.

In this solution (to Exercise #1 part d - Exercise Set #7) we are looking at a simple but naive solution to creating a configurable Repo Factory which will allow the User Service to load a configured User Repo implementation in an abstract manner.

I call this naive as this is not the best and not the most generic way of doing this. We will look at a refined version a bit later.

NOTE: Code for this lecture has been attached.

In this lecture we will learn how to run Docker and pull REDIS image into it so that we could run it as a server.
This is a simple and quick introduction so that we can run our solution with REDIS backend (which we will do in the next lecture)

NOTE: I have attached a PDF document which will help you install and test out your Docker installation for your specific machine.

In this lecture we will run the REDIS abstraction against a Docker container whcih will be running REDIS server.
We willl demonstrate the factory pattern when loading different repo implementations including the REIDS repo implementation.

We also added some simple logging to showcase the factory inner-workings.

NOTE: All the code showcased in this lecture has been attached with this lecture.

In this exercise set the student will create a new implementation for the UserRepository which will be a PostgreSQL implementation as well as adding a new Factory implementation so that the User Service can seemlessly plug in the new implementation with no code change.

We also learn a bit more about Docker by introducing the concept of a valume.

This is the culmination of the course material and is a perfect project to take all that students have learneed and then progressively create a fully functional blogging site application.

In this project the students will:

1. Analise the provided template (Next.JS and Tailwind) and document the user workflow

2. Design the data needs of the application and create the database

3. Design the API for the backend and create the microservices

4. Write test suites for the microservices

5. Secure the API

Enjoy the application you have created!

NOTE: the Next.js template information and details about the project have been attached to this lecture.