
Set up frontend to create new configs with react router and forms. Query min and max timestamps, set the date picker, and store config data in a string-array grid DB.
Create a styled user config page using tailwind, build a dynamic config form with react hook form, and integrate react select dropdowns for column name and type.
Outline the machine learning portion by selecting a Kaggle dataset for intrusion detection and evaluating an Actian log dataset. Prepare to ingest data into the database using Python and SQL.
Create a dedicated machine learning docker service in the project, using docker compose, a persistent volume, and an alpine image with java and python that depends on grid db server.
Prepare training data by cloning the Actian source, merging benign and exploit logs into one proxy-format file, labeling them, and setting up a Node.js wash script for grid db ingestion.
Ingest cleaned data into gridlock by configuring logs and updating docker compose, enabling data ingestion and training.
Connect to grid DB via SQL with Python by starting the JVM with gpipe, using a SQLAlchemy fork for a JDBC connection, and constructing the grid store JDBC URL.
Learn how to transform log data into features, train a Python random forest on exploit and benign samples, and save a pickle model for inference.
Set up a Python inference module in the processor to read incoming GridLog logs, convert them to features, run predictions on malicious or benign, and feed results back to Java.
Let's put everything together and test our ML model to make sure it works as expected!
Learn to integrate machine learning into your existing web applications and understand how machine learning can impact future projects.
This course is part 3 in our series, teaching you how to build a full-stack Java application, from nothing to fully functioning! In this course, we will continue the work from part 2, changing the way the configuration works from just raw .json files in the file system, to a full config page on the frontend. This entails using html forms, sending and handling complex data structures to the backend, and saving these data into a database. We are also introducing more TypeScript, so we will be creating TypeScript types to ensure the data in the form is formed correctly.
We will then also tackle Machine Learning. We will go over what it is, how we use it in this project, and how to implement it yourself. The flow of the ML in the course is as follows:
1. GridLog reads raw HTTP logs from the host
2. GridLog saves raw logs
3. GridLog reads raw logs from DB and parses into searchable columns
4. While saving the parsed logs, if GridLog detects these are HTTP logs, it will run Machine Learning inference on the logs to try and predict if the logs are malicious or benign
5. If malicious, save the DB entry as possible intrusion attempt
6. Mark attempt in Log Viewer
To get the above working, we will need to use free Machine Learning libraries to do supervised training on a dataset provided to you. Once trained, we can run inference on any new incoming HTTP logs.
So for this course, you will learning how to implement all of this into an already working by simply adding in a new Docker container to your working docker orchestration file (Docker compose in our case)
Source Code for this code can be found on our GitHub page which is found in the resources section of our Introduction lecture.