Python by Example

Configuration files are used by both users and programmers. They are usually used for storing your application's settings or even your operating system's settings. Python's core library includes a module called configparser that you can use for creating and interacting with configuration files.

Python provides a very powerful logging library in its standard library. A lot of programmers use print
statements for debugging (myself included), but you can also use logging to do this. It's actually cleaner to
use logging as you won't have to go through all your code to remove the print statements.

In this lecture we'll cover the following topics:

• Creating a simple logger
• How to log from multiple modules
• Log formatting
• Log configuration

The os module has many uses. We won't be covering everything that it can do. Instead, we will get an
overview of its uses and we'll also take a look at one of its sub-modules, known as os.path. Specifically, we
will be covering the following:

• os.name
• os.environ
• os.chdir()
• os.getcwd()
• os.getenv()
• os.putenv()
• os.mkdir()
• os.makedirs()
• os.remove()
• os.rename()
• os.rmdir()
• os.startfile()
• os.walk()
• os.path

That looks like a lot to cover, but there is at least ten times as many other actions that the os module can do.

Python provides a couple of really nice modules that we can use to craft emails with. They are the email and smtplib modules. Instead of going over various methods in these two modules, we'll spend some time learning how to actually use these modules. Specifically, we'll be covering the following:

• The basics of emailing
• How to send to multiple addresses at once
• How to send email using the TO, CC and BCC lines
• How to add an attachment and a body using the email module

SQLite is a self-contained, server-less, config-free transactional SQL database engine. Python gained the sqlite3 module all the way back in version 2.5 which means that you can create SQLite database with anycurrent Python without downloading any additional dependencies. Mozilla uses SQLite databases for its popular Firefox browser to store bookmarks and other various pieces of information. In this lecture you will learn the following:

• How to create a SQLite database
• How to insert data into a table
• How to edit the data
• How to delete the data
• Basic SQL queries

In other words, rather than covering bits and pieces of the sqlite3 module, we'll go through how to actually use it.

The subprocess module gives the developer the ability to start processes or programs from Python. In other words, you can start applications and pass arguments to them using the subprocess module. The subprocess module was added way back in Python 2.4 to replace the os modules set of os.popen, os.spawn and os.system calls as well as replace popen2 and the old commands module. We will be looking at the following aspects of the subprocess module:

• the call function
• the Popen class
• how to communicate with a spawned process

The sys module provides system specific parameters and functions. We will be narrowing our study down to the following:

• sys.argv
• sys.executable
• sys.exit
• sys.modules
• sys.path
• sys.platform
• sys.stdin/stdout/stderr

Python has a number of different concurrency constructs such as threading, queues and multiprocessing. The threading module used to be the primary way of accomplishing concurrency. A few years ago, the multiprocessing module was added to the Python suite of standard libraries. This lecture will be focused on how to use threads and queues.

Python gives the developer several tools for working with dates and time. In this lecture, we will be looking at the datetime and time modules. We will study how they work and some common uses for them.

Python has built-in XML parsing capabilities that you can access via its xml module. In this article, we will be focusing on two of the xml module's sub-modules:

• minidom
• ElementTree

We'll start with minidom simply because this used to be the de-facto method of XML parsing. Then we will
look at how to use ElementTree instead.

Python comes with its own debugger module that is named pdb. This module provides an interactive source code debugger for your Python programs. You can set breakpoints, step through your code, inspect stack frames and more. We will look at the following aspects of the module:

• How to start the debugger
• Stepping through your code
• Setting breakpoints

Python decorators are really cool, but they can be a little hard to understand at first. A decorator in Python is a function that accepts another function as an argument. The decorator will usually modify or enhance the function it accepted and return the modified function. This means that when you call a decorated function, you will get a function that may be a little different but that may have additional features compared with the base definition. But let's back up a bit. We should probably review the basic building block of a decorator, namely, the function.

The Python lambda statement is an anonymous or unbound function and a pretty limited function at that. Let's take a look at a few typical examples and see if we can find a use case for it. The typical examples that one normally sees for teaching the lambda are some sort of boring doubling function. Just to be contrary, our simple example will show how to find the square root. First we'll show a normal function and then the lambda equivalent.

Code profiling is an attempt to find bottlenecks in your code. Profiling is supposed to find what parts of your code take the longest. Once you know that, then you can look at those pieces of your code and try to find ways to optimize it. Python comes with three profilers built in: cProfile, profile and hotshot.

According to the Python documentation, hotshot "no longer maintained and may be dropped in a future version of Python". The profile module is a pure Python module, but adds a lot of overhead to profiled programs. Thus we will be focusing on cProfile, which has an interface that mimics the profile module.

Python includes a couple of built-in modules for testing your code. They two methods are called doctest and unittest. We will start with doctest and then use some Test Driven Development techniques to learn how to use unittest.

When you're first starting out as a Python programmer, you don't think about how you might need to install an external package or module. But when that need appears, you'll want to know how to in a hurry! Python packages can be found all over the internet. Most of the popular ones can be found on the Python Package Index (PyPI). You will also find a lot of Python packages on github, bitbucket, and Google code. In this lecture, we will be covering the following methods of installing Python packages:

• Install from source
• easy_install
• pip
• Other ways to install packages

Python comes with a handy module called ConfigParser. It's good for creating and reading configuration files (aka INI files). However, Michael Foord (author of IronPython in Action) and Nicola Larosa decided to write their own configuration module called ConfigObj. In many ways, it is an improvement over the standard library's module. For example, it will return a dictionary-like object when it reads a config file. ConfigObj can also understand some Python types. Another neat feature is that you can create a configuration spec that ConfigObj will use to validate the config file.

In Part I, we looked at some of Python's built-in XML parsers. In this lecture, we will look at the fun third-party package, lxml from codespeak. It uses the ElementTree API, among other things. The lxml package has XPath and XSLT support, includes an API for SAX and a C-level API for compatibility with C/Pyrex modules.

Here is what we will cover:

• How to Parse XML with lxml
• A Refactoring example
• How to Parse XML with lxml.objectify
• How to Create XML with lxml.objectify

Python code analysis can be a heavy subject, but it can be very helpful in making your programs better. There are several Python code analyzers that you can use to check your code and see if they conform to standards. pylint is probably the most popular. It's very configurable, customizable and pluggable too. It also checks your code to see if it conforms to PEP8, the official style guide of Python Core and it looks for programming errors too.

Note that pylint checks your code against most, but not all of PEP8's standards. We will spend a little time learning about another code analysis package that is called pyflakes.

The requests package is a more Pythonic replacement for Python's own urllib. You will find that requests package's API is quite a bit simpler to work with. You can install the requests library by using pip or easy_install or from source.

SQLAlchemy is usually referred to as an Object Relational Mapper (ORM), although it is much more full featured than any of the other Python ORMs that I've used, such as SqlObject or the one that's built into Django. SQLAlchemy was created by a fellow named Michael Bayer. Since I'm a music nut, we'll be creating a simple database to store album information. A database isn't a database without some relationships, so we'll create two tables and connect them. Here are a few other things we'll be learning:

• Adding data to each table
• Modifying data
• Deleting data
• Basic queries

Virtual environments can be really handy for testing software. That's true in programming circles too. Ian Bicking created the virtualenv project, which is a tool for creating isolated Python environments. You can use these environments to test out new versions of your software, new versions of packages you depend on or just as a sandbox for trying out some new package in general. You can also use virtualenv as a workspace when you can't copy files into site-packages because it's on a shared host. When you create a virtual environment with virtualenv, it creates a folder and copies Python into it along with a site-packages folder and a couple others. It also installs pip. Once your virtual environment is active, it's just like using your normal Python. And when you're done, you can just delete the folder to cleanup. No muss, no fuss. Alternatively, you can keep on using it for development.