AWS Serverless Design for IoT [Retired]

In this lecture we describe the baic sketch which we will base our ESP8266 and ESP32 device implementation on to talk to AWS IoT.  There is a link to the original sketches which allow a single page application with embedded certificates to connect to AWS IoT Core and transfer data.  Also we will discuss installation directions for the boards manager and firmware packages as well as the PubSub library for MQTT.

In this lecture we continue to work on our ESP8266/ESP32 single page sketch to communicate with AWS IoT Core.  We discuss how to configure our personal AWS endpoint as well as replace our devices private certificate, and client certificate with our own device certificates downloaded from AWS IoT Core (See appendix/YouTube video link where I explain this if you don't already know how to generate device certificates).   We can use our own x.509 CA root certificate or use the one provided by the sketch.  We also discuss making things easier by using the EOF technique to avoid meticulous certificate formatting issues.

Now that we have configured the Arduino sketch for our account and endpoint with our device certificates we can now connect our device an speak to AWS IoT Core directly from our ESP8266/ESP32 device.

In this lecture we discuss how to publish from the cloud to a subscription topic on the device.  This built in benefit of MQTT allows inherent duplex communication but using a topic to publish from the device but also another topic to publish from the cloud.  This feature is useful if we ever need to change device state based on cloud events.  Also I demonstrate a method to generate fake data from a virtual device on AWS IoT Core if you don't have a physical device available.

An Introduction to AWS IoT Analytics which is a managed AWS serverless solution using underlying AWS services including AWS IoT Core, Kinesis, DynamoDB, and S3.

Creating a AWS IoT Analytics data-flow configuring our Channel, Pipeline, and Datastore.

We discuss altering our hello world sketch from the last lecture to now send a data packet to AWS IoT Core that provides temperature and humidity via a snprintf data buffer.

A deeper discussion into AWS Lambda and how it functions when dealing with IoT data.  Specifically how we can use Lambda data enhancement in the AWS IoT Analytics pipeline to process, filter, compute, or enhance our IoT data.  An important overview of lambda in general is discussed as it is the key component of the AWS serverless model.

In this lecture we test a Lambda function in Node.js with an incoming payload from the test function within lambda.  We go over explicitly extracting both the incoming 'event' object as well as the 'context' object from an incoming data payload.   

Inthis lecture we note the difference between a batch of data coming in from the pipeline in AWS IoT Analytics and what will come in directly from AWS IoT Core later in the course.  The batched JSON from the underlying AWS Kinesis service has an additional nesting in the JSON structure.  We also show how to set our Lambda permissions through the AWS CLI.  Finally we run our simPub1 Arduino script on our device to produce out data set for using in AWS QuickSight and then AWS SageMaker.

In this lecture we use AWS QuickSight with the built in connector to our AWS IoT Analytics service to ingest the data for a customized visualization.  We use a simple line chart but many visualizations for BI are available for you to manipulate the IoT date from your Data Set in IoT Analytics.

Now that we produced our data set from our device sending IoT data and having that data enhanced with Lambda we had a chance to use AWS QuickSight for some visualizations.  In this lecture we can use a simple Python program to provide addition visualizations with minor statistical analysis on our devices IoT data by invoking the Data Set we created previously and utilizing the Pandas and Matplotlib librarys

In this lecture we discuss the pros and ons of both utilzing an invoking our Data Set directly from IoT Analytics as opposed to creating a data lake repository of our data held in a AWS S3 bucket.  Many more AWS services can work on a S3 bucket as well as additional advantages discussed in this lecture.

In this lecture we now duplicate our IoT data, not only in our Data Store and Data set for consumption by other AWS services, but we also send our IoT data in CSV format to a bucket repository in S3 which will function as a data lake.  Our data in S3 is now set to be consumed by any outside or internal service we like as long as bucket permissions are set.

In this video I show you how to create a completely open and accessible bucket on AWS S3 for 2022  with a static IP.   Once we create our accessible public bucket we can take one extra step to make it serve as a static web host as well as a IoT data bucket. We will set the Bucket Policy to 'read only' and then design an accessible CORS rule to allow cross-origin access with the appropriate headers.

In this lecture we set up our S3 bucket which holds our IoT data to be public to anyone who wants to read it.  This is fine as long as our data is not confidential as we are only given read permission not delete or write permission. 

We demonstrate a Google Chart hosted on a remote testing environment to ensure it can consumes our IoT data from another S3 bucket and can graph the IoT data  given the serverside timestamp on the x-axis.

We demonstrate a Google Chart hosted in a public S3 bucket which consumes our IoT data from another bucket holding our IoT data and graphs it over the serverside timestamp on the x-axis enhanced in our lambda function. The temperature and humidity are charted on the Y-Axis.

An introduction to our advanced example on AWS IoT Analytics.  Here I discuss the design flow en toto, and how we will be using various AWS services to transmit, enhance, store, retrieve, and visualize out IoT Data.

In this lecture I discuss the first of our two Lambda functions written in Node.js.  This Lambda function will enhance and decode our incoming GPS data to give use the current weather and city/state for our incoming GPS coordinates.  To accomplish this we will be 'rolling up' the Weather-JS and Cities Node modules with NPM to provide the needed functionality 

In this lecture we will first test our Lambda function from Lambda itself, and then test it in from within  the AWS IoT Analytics design flow.  Once in the design flow, we will give AWS IoT Analytics permission to execute our Lambda by using the AWS CLI to grant permission.

In this lecture we go over our Arduino Sketch, and show how to create a data package within our Arduino sketch which transmits both internal pseudo-weather and pseudo-GPS data to AWS IoT Core via MQTT.  Once our IoT data is in IoT Core, it will be connected to AWS IoT Analytics via the 'Channel' in IoT Analytics that we created previously

We focus in further on our design flow for the last few parts of our IoT serverless architecture in which we create our second lambda, API Gateway endpoint, and static website hosted on S3.

In this lecture we design our second lambda function which allows us to extract IoT data from our private data bucket held in S3.  This lambda will first extract the CSV data using the AWS-SDK with a S3 object, and then  transform it to JSON format for better consumption on our static website

In this lecture we create a URL endpoint that we link to our extraction Lambda.  Once our GET request is invoked from our API Gateway created endpoint, we can then ingest the transformed data from our Lambda function to our browser, Postman, or webhost.

In this lecture we design our static web page in Html/JavaScript and invoke our URL endpoint previously created with API Gateway.  Once the URL is invoked we can retrieve our data via lambda and then place it in our Highcahrts visualization.  We will visualize both the internal and external temperature and humidity,  as well as generate maps with google given our coordinates sent  from our device and stored in our data lake on S3.

A quick preview for adding security by adding an API Key.  This material is covered in depth toward the end of the course, however you can also get the secured website for preview here.

In this video we create an Action and Rule in AWS IoT Core which sends data to a S3 bucket we are using as a static web host. No storage is needed for this simple design.  This is a Highcharts visualization hosted on S3 that works with an IoT Core action that saves data to S3. The JavaScript code in the S3 webhost fetches the IoT data in the S3 bucket and synchronously charts the resultant data points for each fetch interval. The cost should be a few pennies if you are off the free tier.

In this video we upload our HTML and JavaScript code to our webhost in S3.  We then test the code from the MQTT test client in AWS IoT Core to insure it is producing our Highchart visualizations.

In this video we use our new AWS IoT Core Action/Rule in AWS IoT Core which sends data to our S3 bucket. We sent test data with a customized bash script provided by AWS to automatically produce data with editable iterations and delays between IoT JSON payloads.

A discussion of different levels of data ingestion and consumption techniques.  After going over simple "level" one design utilizing lambda ingestion from AWS IoT Core.  We can also implement IoT Data ingestion that originates from API Gateway.

Brief overview of serverless methods for IoT Data.  In this section we focus on a simple IoT data dispatch to s3 utilizing a Lambda function written in Node.js.  Specifically using Lambda to dispatch our incoming event object from AWS IoT Core to S3 for a data Lake as an IoT data repository.

Writing a Node.js function in Lambda to choose a bucket and file for our IoT data object.  We will have one data object per folder as a first example.

Testing our Lambda function and assigning appropriate IAM permissions to write from Lambda to S3.  We send a fake data package from AWS IoT Core but attached is the script for both your ESP8266 and ESP32 to send the data package I demonstrate from AWS IoT Core on the MQTT broker.   Note the topic name in the sketch we publish from we will need to subscribe to on the device.

Now we change or Lambda function to create multiple timestamped events and place them into the same we created earlier.

Making our Static website public and setting our permissions, bucket policy, and CORS.  Then using three files (index.html, main.js, and visualizer.js)  with Chart.js we can now  visualize our IoT data using the format generated by our modified Lambda function in the third lecture of this section.  If you like you can use your own webhost and do your own website based ouround your own IoT data.

Introduction to Kinesis Data Firehose and using this service with IoT

We cover how to configure Kinesis Data Firehose for IoT data sent from our device, and then we look at the specific schema that is saved to our S3 data lake as a repository for our Kinesis data.  Notice the JSON is saved in batched format similar to IoT Analytics when it was enhanced in Lambda.  This is because Kinesis Firehose was the underlying service in the data pipeline.

Now that we have figured out what our data schema looks like on the S3 repository we can design a Chart.js program that gives us a Bar chart and line graph for basic visualization.

We introduce our project for level two IoT Serverless design.  We base our project on the Guitar Room environment sensor project.  This project is based around a practical implementation of the AWS JavaScript SDK in the browser.  We add Lambda enhancement as well as utilize DynamoDB, IoT Core, Cognito, and a S3 static Web Host.

In this lecture we format our database in DynamoDB to have the Schema reflect the data that will be passed on from Lambda. 

We develop a Lambda function to add a properly formatted ISO 8601 timestamp.  This function takes the environmental JSON event object from AWS IoT Core and modifys it to pass to our DynamoDB table in the proper format.  We need to insure our data is formatted correctly by testing a JSON test package directly from the Lambda function.

In this lecture we connect our Lambda fuction to AWS IoT cCore to accept our  enviro data from our ESP8266/ESP32 device.  This new action from IoT Core will send the JSON payload as an event to be enhanced by our Lambda function from the previous lecture.

In this lecture we create a simple AWS Cognito  Identity in JavaScript which we can insert directly into our static web host.   This now provides permissions which will allow our static web host to read our IoT data lake from S3.   This unauthorized Cognito Identity will then be provisioned in IAM with an attached S3 read-only policy.

In this lecture we upload our index.html and refresh.js as static web pages held in a public S3 bucket.  Once these pages are uploaded we will further examine our refresh.js page and discuss how the DynamoDB query is handled by using the AWS JavaScript SDK invoked directly from the browser.  We will also discuss refresh timing, as well as macthing our variable names with our DynamoDB table and Lambda function.

We introduce level three IoT Serverless design utilizing Lambda to invoke AWS services to retrieve IoT data from our data repositories.  I explain the pro's and con's of using this method over our previous level two and level three methods to design a serverless IoT process on AWS.

Here we set up out Lambda function to retrieve our IoT JSON data objects from our data repository held in S3.

Using API Gateway with Lambda integration we create an accessible URL endpoint, tied to our Lambda function, that we can use in our static web host.

Here we create a static webhost in JavaScript and HTML and utilize our API Gateway endpoint to invoke our Lambda function and retrieve our IoT data from S3 so that we can visualize the data with Highcharts

In this lecture we clean our IoT data and in turn demonstrate how we can add GetObject and ListObject permissions in our Lambda role so that we can now use a private bucket to host our IoT data in S3 to demonstrate the ability to keep our data private as needed.

In this Lambda example we invoke the AWS-SDK in Node.js to ingest an incoming IoT data payload from IoT Core, and then republish that JSON data to another topic on a different IoT device. The receiving device simply needs to subscribe to a matching MQTT topic sent from lambda to receive the data payload on the device.