Build Realtime Data Dashboard With AWS,Python,Kafka,Grafana

Learn how to create an AWS account, verify email and phone, set up a password and payment method, enable MFA with Google Authenticator, and sign in as root user.

Launch EC2 in AWS, name it crypto project, select Ubuntu on free plan, enable a public IP, skip the key pair, and connect via the browse-based terminal to set up.

Stop idle AWS EC2 instances to pause billing when not in use. Then start the instance and reconnect via the browser-based terminal to resume work.

Learn essential linux commands like pwd, mkdir, cd, touch, ls, and vi, plus encoding fixes, to create and run a hello world python script for a data dashboard project.

Set up a Python virtual environment, update and upgrade system packages, install crypto price data and streaming libraries, and verify installations with Linux commands and a hello word script.

Build a real-time crypto price collector in Python using the free Crypto Compare API to fetch Bitcoin and Ethereum prices, printing timestamped records every 120 seconds.

Fetch live Bitcoin and Ethereum prices via a crypto API, set up a Python virtual environment, write and run a script, then extend it to Kafka for real-time processing.

Kafka acts as a middle layer that decouples producers and consumers, using topics to route real-time data between systems, ensuring reliable delivery and automatic recovery when producers or consumers fail.

Understand how Zookeeper coordinates a distributed Kafka cluster by tracking brokers, enabling leader election and metadata management, with partitioned data and leader-follower replication for reliability.

Start an instance, install Java 11, download Kafka 3.7.2, extract it, rename the folder to Kafka, start ZooKeeper on port 2181, and then start Kafka on port 9092.

Kafka topics act as mailboxes for data streams, with producers sending messages and consumers reading them. It demonstrates creating a crypto price topic and verifying flow with producer-consumer scripts.

Run zookeeper, jukiper, and kafka in the background using nohup and & so they keep running after you close the terminal. Track the process id and output file.

Run Kafka and Jukiper in the background with nohup, log outputs to a file, and verify producer and consumer stay active on EC2 for a smooth real-time dashboard workflow.

Create a Python crypto producer to fetch real-time Bitcoin and Ethereum prices from the crypto compare API and publish them to a Kafka topic as JSON messages.

Collect Bitcoin and Ethereum price data from a crypto API and push it to Kafka to demonstrate end-to-end data ingestion and real-time streaming with a Python producer and consumer.

Set up a Postgrade SQL database named CryptoDB with a Crypto user and two tables, crypto price and crypto prediction, to store Kafka price data and Python-scripted ML predictions.

Start the instance, run Kafka and producer, then deploy a Python consumer to store real-time crypto price data in PostgreSQL. Verify storage by querying crypto DB with a select.

Master basic sql commands—select, from, where, max, as, order by, and group by—to query the crypto price table in a postgres database and extract meaningful insights.

Connect to CryptoDB, display price and timestamp from the crypto price table, filter BTC, show the max price with an alias, and group by symbol for the highest price.

Collect and store crypto price data from Kafka into PostgreSQL, then generate 60-second price predictions using linear regression and save results back to the database.

Build a real-time crypto price predictor by creating a Python script that forecasts one minute ahead and stores it in the PostgreSQL crypto prediction table, validating the end-to-end pipeline.

Explore Grafana as a real-time data visualization tool, connect to a database like postgrade SQL, and set up Grafana on Ubuntu to display real-time crypto prices via port 3000.

Install Grafana on an EC2 instance to visualize data on a dashboard, add Grafana repository and GPG key, update packages, start and enable service, and open port 3000 for login.