Python for Automating Information Security

How do I look at the details of network traffic to visually understand it? The best tool for this is Wireshark.

   •  Open metasploitable_normal.pcap in Wireshark

   •  Explore the packet capture file and explain some basic features of Wireshark

When looking at traffic patterns, how do I understand what is considered normal and what is considered malicious? As with many other things in security, it simply takes time and experience, so the more we use Wireshark the better.

   •  Still with metasploitable_normal.pcap loaded in Wireshark, explain advanced features of Wireshark such as follow stream

How do I write a Python script to read from a packet capture file and look for specific malicious patterns? We will use the Scapy library for this.

   •  Introduce the Scapy package

   •  Walk through investigate_pcap_file.py

   •  Test investigate_pcap_file.py by running it against metasploitable_attacks.pcap and metasploitable_all.pcap

How do I write a Python script to read from a packet capture file and create a profile of the traffic? We will again use Scapy and some simple analysis techniques.

   •  Walk through the profile_network.py script

   •  Generate profiles for metasploitable_normal.pcap and metasploitable_attacks.pcap

How do I compare two network traffic profiles to look for malicious/anomalous activity? We will compare the two baselines we generated in the previous video, and flag based on a given threshold.

   •  Walk through the compare_network_profiles.py script

   •  Run the compare_network_profiles.py script against the profiles generated in the previous video

Get an introduction to open-source intelligence.

   •  Get introduced to OSINT concepts

How do I find out what hosts exist for my target on the Internet? The solution is DNS enumeration.

   •  Walk through dnsbuster.py and install required packages

   •  Choose domain to enumerate and walk through subdomains.txt supporting file

   •  Test dnsbuster.py against the chosen domain

Introduction to Google Dorking and prework for video 3.4.

   •  Introduce Google Dorking and give example(s)

   •  Introduce Google Hacking Database

   •  Set up Google API for steps in video 3.4

How do I quickly perform Google Dorking to gather intel? The solution is automation through Python.

   •  Walk through gdork.py

   •  Walk through dorks.json supporting file

   •  Test gdork.py against domain set up in video 3.3

How do I quickly gather subdirectories of a base URL? The solution is to automate web directory enumeration through Python.

   •  Walk through webdirbuster.py and choose URL to enumerate

   •  Walk through common_webdirs.txt supporting file

   •  Test webdirbuster.py against chosen URL

Walk through find_malicious_logs.py

   •  Test find_malicious_logs.py against output JSON from video 4.1

   •  Discuss results

   •  Test parse_log.py against access.log

How do I know what types of log messages are normal? The answer is with experience and understanding of the application that is creating the logs.

   •  Open output JSON from video 4.1

   •  Observe different types of log messages

   •  Categorize different types of messages

How do I automate searching for messages indicating malicious activity in a log file? The answer is to combine steps 4.1 and 4.2 and identify malicious messages from the parsed log file.

How do I profile log files to create a baseline and a test profile to compare against the baseline? First, we can parse the baseline and test log files, then create profiles off using Python.

   •  Walk through profile_logfile.py, and parse baseline_access.log supporting file with parse_log.py

   •  Use profile_logfile.py on parsed baseline_access.log JSON

   •  Use profile_logfile.py on parsed access.log JSON

How do I compare a test log profile against a baseline profile to find potentially malicious activity? We can write a Python script to compare log message rates to look for potential DoS activity.

   •  Walk through compare_log_profiles.py

   •  Run compare_log_profiles.py against profiles generated in 4.4

   •  Discuss results

Introduction to the nc tool.

   •  Demonstrate a simple client-server function in nc

   •  Demonstrate testing for an open port using nc

   •  Demonstrate sending a reverse shell using nc

How do we go about deciding what input arguments we need? We need to first decide exactly what we want our program to do.

   •  Investigate the command-line arguments for nc

   •  Decide what functionality we need for our program

   •  Write the corresponding input arguments

How do we go about writing the client component of the tool? We need to use the socket module in Python.

   •  Introduction to the socket module

   •  Write the client_send method

   •  Implement the client_send method in the main method

How do we go about writing the server component of the tool? We need to use the socket module again but add additional functionality beyond what is in the client component.

   •  Write the run_command and handle_client_connection methods

   •  Write the start_server method

   •  Implement the start_server method in the main method

How do we test the functionality? We simply need to test the connectivity between two systems and debug any issues found.

   •  Upload the program to another system

   •  Start the program in server mode on one system

   •  Connect to the other system using client mode

Introduction to how passive network capture works and why we may use it in penetration testing.

   •  Describe how passive network capture works

   •  Describe the purpose of passive network capture in a pentest

   •  Describe why we want to do this programmatically

How do I begin programmatically capturing and reading packets off a live network? We will use the Python package Scapy to do this.

   •  Walk through sniff.py and install the required packages

   •  Run sniff.py and observe output

How can I automatically gather credentials during a live packet capture? Again, we will use Scapy to capture and analyze the packets.

   •  Walk through sniffcreds.py

   •  Run sniffcreds.py and run snmpwalk against Metasploitable

   •  Run sniffcreds.py and run curl against Metasploitable with HTTP Basic Authorization

Introduction to ARP, ARP caching, and how to abuse it for penetration testing purposes.

   •  Describe how ARP works and its purpose in networks

   •  Describe how ARP cache poisoning works

How can I automate the process of ARP cache poisoning? We will again use Scapy, but this time in an active mode instead of passive packet capture.

   •  Walk through arp-poisoner.py

   •  Run arp-poisoner.py on Kali against Metasploitable IP

   •  Start tcpdump on Metasploitable and ping it from host machine

Introduction to MSF and its use in penetration testing. We will also talk about why we want to write something similar in Python.

   •  Give overview of MSF

   •  Demonstrate the interactive command line interface for MSF

   •  Talk about advantages to writing a similar program in Python

How do we begin implementing network service discovery functions in Python? We will interact with NMAP using the python-nmap package.

   •  Walk through discovery.py and install required packages

   •  Run discovery module as a stand-alone feature

We want to design a modular, extensible framework that allows us to easily add exploits over time. We will use dynamic class loading to do this.

   •  Walk through the abstract exploit class

   •  Walk through the Exploit Loader class

   •  Walk through the Revshell class

We want to design an interactive command shell to control our program.

   •  Walk through the two exploit classes

   •  Walk through the MSF Menu class

How do we test our program? Simply run our program against Metasploitable.

   •  Boot up Metasploitable

   •  Test discovery and all exploits