Log analysis - Detecting Web Attacks.

In this first module of the course, we will discuss about introduction and importance of log analysis where we discuss about what is log analysis and try to understand its importance. We will be looking into the basics of logs and how a typical log looks like by giving some examples. We will also look into what information is contained in logs and how it does look in real-life logs.

We try to ask a question - why do we need to analyse and monitor logs? and understand its benefits.

In this lecture, we look into typical use cases where log data holds immense value. We discuss some use cases where we talk about the data that the log file can contain and its importance.

This lecture discusses about a generic web application architecture and related details on where logging takes place.

In this lecture, we discuss about the various HTTP response codes. These HTTP response codes helps us to understand the response the application gave with reference to the query sent by the client. Also, we take a quick recap on the topics discussed in this module and proceed further with the next module.

In this lecture, we discuss about the contents & learning goals of this module. This module is focused on web servers - apache, nginx & Microsoft IIS and we also discuss about the log format of SSH and FTP servers.

In this lecture, we talk about apache web server, its logging format and logging levels, the default location of logs and default log format. We take some sample log entries and explain the details about the fields that we see in a log entry for Apache web server.

Here, in this lecture, we show you hands-on logging location and customizing the log entries for apache web server which is installed on an ubuntu server.

Here, we see a practical demonstration on logging location & configuration for apache web server installed on a Fedora server.

In this lecture, we take a quick look on the theoretical part and getting ourselves introduced with Nginx web server. We take a look at some sample log entries and break them down to understand the fields that are being logged by this server. We also look at the common log location where the nginx logs are stored.

In this quick demonstration, we look at the logging location & configuration file location for nginx web server installed on an Ubuntu server.

In this demonstration, we look at logging location & configuration settings for Nginx web server installed on Fedora server.

In this lecture, we take a look at Microsoft IIS web server, its introduction and how it logs the events. We take an example of a sample log entry and break it down field by field and understand the logging fields and the location where the logs are stored.

In this demonstration, we take a look at Microsoft IIS web server's logging location.

In this lecture, we take a look at the logging formats of SSH & FTP server. We inspect some sample entries and understand the details on how FTP & SSH servers logs the event details. After this, we move on the the recap of the module and end the module.

In this lecture, we start by setting up our test environment. This introduction shows about the parts of the module and knowledge accomplishments.

In this lecture, we demonstrate the installation and setting up of metasploitable linux VM in virtualbox. This vulnerable VM will server as our victim machine, we will be firing all our attacks on this VM.

In this demonstration we show how to download and import kali linux VM and set it up on our Virtualbox. This Kali linux VM will serve as attacker machine that will simulate attacks to our metasploitable VM.

In this lecture, we quickly discuss about the available networking modes in Virtualbox. We discuss about the bridged networking mode.

In this lecture, we discuss about the virtualbox Host only networking mode & understand its uses as per our needs.

In this lecture, we discuss about virtualbox Network Address translation (NAT) mode of networking.

In this lecture, we discuss about the Virtualbox NAT Network mode and its difference between the previously discussed plain NAT mode

In this lecture, we demonstrate how we can generate some logs for SSH server. Here, we have used PuTTY client for windows system for connecting to SSH service of our metasploitable VM.

In this demonstration, we generate some logs on our FTP server by connecting to it and performing some action. For this demonstration, we have used FileZilla FTP client for connecting through a windows machine. You can use any other preferred client to connect to our FTP server.

In this demonstration, we connect to the logs of the web server and generate some benign traffic by visiting random pages of the web application and view the generated logs in real time which can be used to further study the logs to understand the normal behavior of a visitor to the application.

In this video, we brief on our strategy of generating most common attack signature patterns which will be recorded on the logs of the target system. We will be using kali linux as our attacker machine and the target VM will be Metasploitable Linux VM

In this demonstration, we start to simulate malicious action by an attacker by generating scans using the tools nikto & nmap.

We use nikto which is a web server vulnerability scanner and nmap which is a network security scanner.

We fire multiple scans to generate traffic that mimics a malicious user trying to perform reconnaissance and vulnerability scans.

In this demonstration, we take 3 different tools to speed up our task for generating malicious traffic for a web application. We first use OWASP ZAP which is a web application testing suite and generate traffic using its scanning options. Next, we use the famous tool - sqlmap which is a SQL injection testing tool and we generate some test sql injection traffic signatures. Proceeding ahead, we use the command injection tool - commix to simulate command injection and simulate actions of a malicious attacker trying to test and break into the application.

In this video demonstration, we generate traffic for the vulnerabilities related to HTTP form bruteforce and Local File Inclusion (LFI) Vulnerabilities.

In this demonstration, we generate traffic for file upload vulnerabilities.

In this lecture, we generate traffic for SSH & FTP service and bruteforce the logins to generate traffic.

This lecture is the recap of the module 3 and we take a quick look on what we learnt so far in this module.

This lecture gives introduction about the module and its contents and knowledge accomplishments by the end of the module.

In this lecture, we discuss about the theoretical concepts of log analysis. We discuss about the concepts of Normalization and pattern recognition.

In this lecture, we discuss about the concepts of Tagging & event correlation.

In this lecture, we enumerate some of best in class tools and a general architecture of centralized log monitoring solution and how things work from a high level view.

In this lecture, we see it for ourselves that how logs look in raw text editor and how much difficult to manually sift through huge log data. This encourages us to use solutions that are specific to our analysis task. We can still use a regular text editor to analyse logs based on some of the search criteria we set. But for a larger file size, this gets somewhat difficult.

In this lecture, we demonstrate on installation of ubuntu server which will be used for analysis of logs that we have previously created in earlier module. We will be using those logs for analysis on other platforms.

In this demonstration, we download and install GoAccess tool for visual analysis of logs that we have generated in the earlier modules.

In this lecture, we discuss about the ingested logs in GoAccess tool. We look at the interface of the tool and understand the patterns and some typical telltale signs of attacks on the web server.

In this demonstration, we utilize the power of cloud-based log ingestion and analysis tool LogDNA and demonstrate how we can push our logs to this cloud platform via agent install on our ubuntu server. After the installation and configuration, we can see our server logs being pushed in real-time to the LogDNA cloud platform.

In this demonstration, we show how we can select a log entry and select the required log entry parameters and view and download the log data as required.

In this demonstration, we take a look and show how we can install and setup ManageEngine Event Log Analyzer tool for local ingestion of log data.

In this demonstration, we start by ingesting logs to the ManageEngine Platform and start to explore the platform and analyze the logs that we have generated in the earlier sections.

In this demonstration, we continue to analyze logs that we have imported into the tool ManageEngine Event Log Analyzer.

This is an assignment task for learners. Give a try and practice on the Logs provided, import the logs into the ManageEngine tool and try analyze the log entries.

In this video, we give the introduction about the module and its contents.

In this lecture, we discuss about threat agents to logging systems. These threat agents can arise due to poor application design and improper file system permissions. Threats such as Log forging & Log file poisoning are discussed in this section.

Here, we discuss about some of the attributes that relate to insufficient logging.

We discuss about the importance & benefits of security logging and explain how the security logging can help us achieve multiple benefits.

In this lecture, We introduce you about the best logging practices.

In this lecture, we talk about defining the purpose of logging the event data and taking into consideration, the event data sources from where we are going to source the log data.

In this lecture, we discuss about saving event log data and some considerations on where and how to store the event log data.

In this lecture, we discuss about the event attributes and the events that can be considered for logging.

In this lecture, we talk about what data can be excluded while setting up logging. The data that can be excluded is sensitive data as well as data as per the needs of the business.

This lecture discusses about the operational part of the logging operations and disposal of logs with general guidelines that can be followed in this step.

We discuss about more best practices of logging and we demonstrate on setting up the apache web servers logging verbosity by editing the configuration file.

Here, in this lecture, we demonstrate on setting up logrotate utility and explore the available options the logrotate utility offers us to generate and rotate logs at our desired time intervals. We discuss about setting up logrotate on hourly, daily, weekly or monthly intervals.

In this lecture, we demonstrate on setting up specific required fields for Microsoft IIS web server.

We discuss about setting up verbosity levels of SSH & FTP servers. As we set up more verbose levels of  logging for SSH & FTP servers. As we setup more verbose levels of logging, there is a chance that information that should not be logged in logs can also be logged which can cause a violation of privacy of users. However, it is recommended that the logging levels should be set purely on the basis of the requirements and users should refer the documentation of the SSH & FTP servers for more detailed information.

This lecture is the recap of the module and the end of the course.