Python: Web Development and Penetration Testing

Code that is used in many views can be moved into middleware, allowing it to operate either before or after your view code is executed. In this lecture, you will learn the following topics:

• Process the Request and Process view hooks to run before your view
• Make Process Template Response and Process Response run after your view
• Get Process Exception running if there is an unhandled exception 

Middleware classes are not fancy; they simply have one or more methods that correspond to one of the five predefined hooks available. We will cover the following topics in this lecture:

• Decide which hooks your middleware should implement
• Create a standard- or new-style class
• Define a method corresponding to the hooks you want 

Showing users exceptions is unprofessional and may be a security problem. Let's examine an easier way to create middleware that catches and responds to unhandled exceptions. In this lecture, we will explore the following topics:

• Create a class and add it to middleware
• Define a process_exception method
• Capture and process exceptions before they get to the user 

In this lecture we will explore the possibility of creating custom filters to make frequently used behaviors easily reusable for everyone on our team:

• Create a submodule of template tags
• Register your filter with the register.filter decorator
• Include your custom filter on any template that uses it

If you need a functionality that is too sophisticated for a filter, it should be a template tag. These are of several kinds, such as simple_tag for basic functionality and a template inclusion tag to bring in entire templates.

• Create a submodule of template tags
• Register your template tag with the register.simple_tag decorator
• Include your custom tag on any template that uses it

With the growing popularity of Angular.js and similar tools, more applications need to provide a RESTful API. Tastypie is a Python module that does most of the work for us. Let's check it out in this lecture. You will learn the following topics:

• Install the Django-tastypie module
• Use the Resource or ModelResource modules to autogenerate an API from your data
• Activate your URLs to get a basic read-only API 

Sometimes you want to modify the data before it goes out to the API. Tastypie makes it easy to add or customize the data returned from the API.

• Take a look at how the tastypie dehydrates functionality and serializes our data
• Customize the dehydrate process to add custom fields
• Inspect the data in the API to check whether your changes were made 

Not all applications work with a read-only API. Sometimes, you want full CRUD support from your API. You need to add an authentication step before you make this functionality available. Fortunately, Tastypie does all of this for us. In this lecture, we will learn the following topics:

• Enable ApiKeyAuthentication and run a migration
• Create a key for a user and add an authorization header to your request
• Enjoy full support of get, post, patch, and delete 

Sometimes, you want to combine data from different models. It's not hard to do this with Python, but generic relationships allow Django to do this for you. In this lecture, you will learn the following topics:

• Include the ContentType and GenericForeignKey modules
• Define a model that has GenericForeignKey
• Ensure that the GenericForeignKey keys are all the same datatype 

Have you wished you could add or edit many records to the database at once? With only a few lines of code, you can add this capability to your application.

• Check out formsets that are based on the FormView class
• Use modelformset_factory as a form class
• Modify form as a list in our template 

Django's ORM works fine when your filters are combined using the AND operator. If you want to perform more complicated queries, you should use the Q function, which just gives you more flexibility with your logic.

• Understand the use of & or | to combine filter options with AND or OR
• Use parentheses to get the proper logical operations
• If you need more power, use the raw function and SQL 

Is your application running slow? You may be performing expensive operations more often than necessary. We can use low-level caching operations to have tight control over our application's performance. In this lecture, you will learn the following topics:

• Enable caching using memcached, redis, or your database
• Take a look at how cache.get and cache.set allow you to set and retrieve cached data
• Use database caching during development and another style in production 

Do you have data that is accessed on many views? We already know we can create middleware to reduce duplication, but why not add caching at to middleware to also speed up performance? In this lecture, you will learn the following topics:

• Create a middleware that responds to the process_request hook
• Use cache.get to check whether your data is cached
• Fetch your data and add it to the request 

If your cache needs are simpler, you can perform caching at the template level. This can provide a tremendous improvement in performance and is easy to use. In this lecture, you will learn the following topics: 

• Load the cache template tag in templates that need it
• Wrap the cache tags to specify the data to be cached and an expiration
• Consider a frontend cache, such as Varnish, to further speed up parts of your site 

Do you frequently perform the same query and filter operation in multiple views? Avoid duplication using model managers. In this lecture, you will learn the following topics:

• Models already have a manager called objects
• Override objects with your own manager to add custom functionality
• Add new methods for each of your frequently used filters 

You may have a similar logic in multiple areas of the site to deal with inserting data. You can create a manager function that accepts your data as a parameter and centralizes your insertion logic. In this lecture, you will learn the following concepts:

• Create a method on your manager that accepts your data
• Perform the validation according to your rules
• Model manager methods can access related fields 

Do you perform regular maintenance on your application? You can simplify periodic cron tasks by creating custom manage.py commands. In this lecture, you will learn the following topics:

• Create a module called management
• Create a submodule called commands, which is a manage.py command
• Improve the performance of your apps as well as reduce problems caused by code duplication 

In this lecture you will learn about the web app penetration-testing methodology, the toolset, and our lab environment. We will cover the following topics:

• Introduce web application penetration testing with the security of the application and methodology in performing such tests
• Learn about the tools most commonly used by professional pentesters
• Take a look at our lab environment 

You will learn about the traditional tools used by security professionals to perform penetration tests. This will provide a basic understanding of the most important type of tools used, and give us ideas on what we can build with Python. In this lecture you will cover the following concepts:

• Introduce HTTP proxies, the most important tool of the set
• Learn about Web Crawlers/ Spiders, Scanners, and brute forcers
• Explore tools specifics for certain tasks or scenarios 

We need to set up the testing environment and we would use VirtualBox, VM, text editor, and the vulnerable Web application we are going to use as target of our tests. In this lecture you will learn the following concepts:

• Install VirtualBox to run our Lab VM
• Access the testing application
• Finally we are going to see the text editor and an important warning 

In this lecture, you’ll learn about web application mapping, what it is, and how to do it:

• Revisit the penetration testing process
• Check out the different options of performing the mapping
• Check out a mapping example from Burp suite 

We want to create a web application crawler to help us map an application. In this lecture, you will learn the following concepts:

• Define the objective
• Create a Scrapy project
• Run the project and verify the results 

We created a basic crawler, but now, we want to make it recursive to cover all the web application content. In this lecture, you will learn the following concepts:

• Create the code to extract all the links
• Check whether the links are valid
• Run the crawler and check the results 

We need to extract information from the web application that will be useful for our Security testing.

• Learn how to extract e-mails
• Extract forms
• Extract comments 

Most applications have resources that are not linked and tools such as crawlers or proxies won’t find. So, we need to discover resources with other methods. In this lecture, we will learn the following concepts:

• Explain what resource discovery is and how it complements other tools
• Note why it is important to use resource discovery techniques
• Explore useful dictionaries that can be used to discover resources 

In order to find more resources that are not linked in a web application, we need to create a brute forcer in order to discover resources using dictionary files.

• Define the objective of the brute forcer
• Explain the code of the basic brute forcer
• Run the brute forcer and check the results

We need to improve the results of the brute forcer in order to facilitate the discovery of interesting resources. Let's do just this!

• Add colors by status code
• Add the number of characters, words, and lines of responses
• Test the script and see the results 

In this video, we will add the detection of redirections and generate more information about the responses, such as the time it takes the response and the MD5 hash of the content. In this lecture, we will cover the following concepts:

• Detect the redirections
• Add the response time and MD5 hash
• Test the script and note the improvements 

When conducting an analysis of big web applications, having a screenshot of the discovered resources could be very handy. We need to add this capability by taking a screenshot of all the resources that return a 200 status code. In this lecture, we will cover the following topics:

• Explain which libraries we will use
• Add the imports and the code to take the picture
• Test this new feature