IT Infrastructure Automation with Puppet - A Complete Guide!
- 12.5 hours on-demand video
- 1 downloadable resource
- Full lifetime access
- Access on mobile and TV
- Certificate of Completion
Get your team access to 4,000+ top Udemy courses anytime, anywhere.Try Udemy for Business
- Build complex configuration files with dynamic data using Puppet's EPP template mechanism.
- Manage files, directories, links, users, and packages with Puppet.
- Learn to manage dependencies and execute commands on remote servers.
- Secure Puppet deployments and master methods to harden infrastructure using community modules.
- Create a scalable and performance-efficient Puppet infrastructure.
- Configure a highly available Puppet infrastructure along with storeconfigs and external node classifiers.
- Understand configuration using Docker and Puppet along with orchestration using Mcollective.
- Basic understanding of Puppet Configuration Management will be useful (Not mandatory).
With the ever-increasing challenges of IT infrastructures, it's no surprise that IT professionals are looking for ways to automate as much as possible! The need to automate common activities has become the major requirements in many large organizations. Puppet is one such configuration management tool that allows to efficiently build, manage and automate all your IT infrastructure efficiently! If you’re a developer, a system administrator, or you're simply curious about learning Puppet, then go for this Learning Path!
This comprehensive 4-in-1 course is a comprehensive guide, filled with quick step-by-step instructions that will help you automate your IT infrastructure with the latest techniques in Puppet! Initially, you’ll get a run-through of the strategies and patterns to introduce Puppet automation. Write custom plugins for Puppet such as facts, resource types, providers, and functions. Also, Deploy configuration files and templates at superfast speeds and manage user accounts and access control appear like a cakewalk. Finally, automate your IT infrastructure with Puppet.
By the end of the course, you’ll become a pro with Puppet 5 by automating your IT infrastructure with the latest techniques!
This Learning Path takes a practical approach with step-by-step recipes to conduct effective penetration testing using the powerful Kali Linux! At the very beginning, you’ll develop your skills using attacks such as wireless cracking, Man-in-the-Middle, and Denial of Service (DOS). Also, explore tools such as Metasploit, Wireshark, Nmap, and much more to detect vulnerabilities with ease! Finally, you’ll master all phases in a typical penetration testing project focusing on relevant Windows tools and techniques.
By the end of the course, you’ll be one step ahead of hackers by discovering and patching your network vulnerabilities as well as performing professional-level web penetration testing!
Contents and Overview
This training program includes 4 complete courses, carefully chosen to give you the most comprehensive training possible.
The first course, Puppet 5 – From Beginner to Pro, is a practical guide with the latest features of Puppet 5. This course teaches you how to install packages and config files, create users, set up scheduled jobs, provision cloud instances, build containers, and so much more. Every example in this course deals with something practical that you're likely to need in your work, and you'll see the complete Puppet code that makes it happen, along with step-by-step instructions on what to type and what output you'll see.
The second course, Puppet Configuration Management, covers Puppet to automate all your work! In this course, you will be introduced to Puppet development and learn how to set up your own private development environment, so that you can develop your code without fear of hurting live environments. From there on out we get familiar with the Puppet DSL and then deploy our code to a puppet server and puppet node like you would in a production environment. After completing this class you should have all the handles you need to start utilizing puppet and know how to further progress and become a true Puppeteer!
The third course, Puppet Techniques for Enterprise Deployments, covers Automating your IT infrastructure with the latest techniques in Puppet! This video course will begin by covering methods of reporting with Puppet and configuring various types of open source dashboards along with procedures to use report processors and create custom ones. You will then exploit PuppetDB to configure and create efficient Puppet reports and also to monitor Puppet changes and alert based on the same. Finally, you will learn about how to implement Nagios based monitoring using Puppet. Towards the end of the video course, you will concentrate on Puppet security. You will also learn about procedures to secure the Puppet deployments and methods to harden infrastructure using community modules. Finally, we’ll learn how to debug errors in the Puppet environment.
The fourth course, Advanced Puppet Techniques, covers writing custom plugins for Puppet such as facts, resource types, providers, and functions. This video course starts with taking your Puppet knowledge to the next level by delving deep into concepts such as scaling the Puppet infrastructure, utilizing storeconfigs with PuppetDB, and using different external node classifiers. You’ll then get your hands dirty by running Puppet components in Docker containers, using Puppet to create Docker images and running applications on Docker along with orchestration using Mcollective. At the end of the course, you will learn how to write custom plugins for Puppet such as facts, resource types, providers, and functions.
By the end of the course, you’ll be able to automate your IT infrastructure with ease with the latest techniques in Puppet!
About the Authors
John Arundel is a DevOps consultant, which means he helps people build world-class web operations teams and infrastructures and has fun doing it. He was formerly a senior operations engineer at global telco Verizon, designing resilient, and high-performance infrastructures for major corporations such as Ford, McDonald's, and Bank of America. He is now an independent consultant, working closely with selected clients to deliver web-scale performance and enterprise-grade resilience on a startup budget. He likes writing books, especially about Puppet (Puppet 2.7 Cookbook and Puppet 3 Cookbook are available from Packt). He also provides training and coaching on Puppet and DevOps, which, it turns out, is far harder than simply doing the work himself. Off the clock, he is a medal-winning, competitive rifle and pistol shooter and a decidedly uncompetitive piano player. He lives in a small cottage in Cornwall, England and believes, like Cicero, that, if you have a garden and a library, then you have everything you need. You may like to follow him on Twitter at @bitfield.
Toni Schmidbauer has been working with various Unix/Linux and BSD systems for the last 17 years. He started working with configuration management in 2005 with CFEngine 2 and switched to Puppet in 2010. Currently, he is responsible for a Puppet infrastructure containing around 2000 nodes running Solaris, AIX, and Linux. When he's not writing Puppet code he likes to tinker with different programming languages and computers in general.
Anirban Saha is an infrastructure professional with more than seven and half years' experience in infrastructure management at various industries and organizations ranging from early startups to corporate environments. He has worked extensively with configuration management and automation tools including Puppet, Chef, Ansible, Saltstack, and Terraform, to name a few. He has extensive experience in architecting, deploying, and managing large infrastructures and speaks at various conferences on the latest technologies.
- System administrators, DevOps professionals, and Developers who want to master advanced topics in Puppet as well as implement added functionalities that Puppet provides for enterprise-level deployments.
What do we mean by configuration data? There will be lots of pieces of information in your manifests which we can regard as configuration data. Let’ see them in detail.
Take a look at why data needs to be maintained
See the settings that depend on nodes and operating systems that differ
Set up Hiera
So far, we've only used a single Hiera data source. Actually, you can have as many data sources as you like. Each usually corresponds to a YAML file, and they are listed in the hierarchy section of the hiera.yaml file.
Deal with multiple values
See merge behaviors
Take a look at Data sources based on facts
Puppet often needs to know your secrets; for example, passwords, private keys, and other credentials need to be configured on the node, and Puppet must have access to this information. Let’s see how to do that.
Set up GnuPG
Add an encrypted Hiera source
Create and edit an encrypted secret
One of the oldest-established Puppet Forge modules is puppetlabs/stdlib, the official Puppet standard library. Let's look more closely now and see what the standard library provides and where you might use it.
Safely installing packages with ensure_packages
Modify files in place with file_line
See some other useful functions
As we've seen, a Puppet module is a way of grouping together a set of related code and resources that performs some particular task, like managing the Apache web server or dealing with archive files. But how do you actually create a module?
Create a repo and write module code
Create and validate the module metadata
Install and apply your module
Now that we know how to include different sets of classes on a given node, depending on the job the node is supposed to do, let's think more about how to name those classes in the most helpful way.
Create a class called role::app_server
Rewrite the app_server role to include profiles
The methods we’ve learnt so far are very useful, but they are limited in one respect: they can only use static text, rather than building the contents of the file dynamically. Let’s see how we can do that with templates.
See the dynamic data problem
Take a look at Puppet template syntax
Since the end result of a template is a file, you won't be surprised that we use Puppet's file resource to work with templates. In fact, we use an attribute of the file resource that you've seen before: the content attribute.
Reference template files
See computations in templates
See conditional statements in templates
If we can generate parts of a file from Puppet expressions, and also include or exclude parts of the file depending on conditions, could we generate parts of the file with a Puppet loop?
That is to say, could we iterate over an array or hash, generating template content for each element?
Iterate over Facter data and structured facts
Iterate over Hiera data and work with templates
Pass parameters to templates and render templates on the command line
Although the technology behind containers is at least thirty years old, it's only in the last few years that containers have really taken off. This is largely thanks to the rise of Docker, a software platform which makes it easier to create and manage containers.
Take a look at the deployment problem
Build Docker containers
Although Dockerfiles are a fairly powerful and flexible way of building containers, they are only static text files, and very often you will need to pass information into the container to tell it what to do. We might call such containers—whose configuration is flexible and based on data available at build time—dynamic containers.
There are two ways of persisting data in a container: the first is to mount a directory from the host machine inside the container, known as a host-mounted volume, and the second is to use what's called a Docker volume. We'll look at both of these in this video.
See Host-mounted volumes
Take a look at Docker volumes
We started off the section by saying that containers are completely self-contained, and have no access to each other, even if they're running on the same host. But to run real applications, we need containers to communicate. Fortunately, there is a way to do this: the Docker network.
Learn container orchestration
Before exploring the advantages of cloud computing, perhaps we should define what it is. In the pre-cloud days, if you needed computing power, you bought an actual, physical computer.
But from the customer's point of view, we don't necessarily want a computer: we just want to compute.
Automate cloud provisioning
Use CloudFormation and Terraform
In the previous video, we used the pre-existing default VPC and subnet to create our instance. In production you'll want to use a dedicated VPC for your Puppet-managed resources, to keep it separate from any other resources. You could, for example, have a staging VPC and a production VPC.
Create an instance in a custom VPC
See other AWS resource types
In this video, we will learn how to install a puppet server on a CentOS 7 machine. We will also install the puppet agent on a second virtual machine. We will do our first puppet run.
Bootstrap the course environment
Install the puppet server in VM1
In this video, we will learn how to install a puppet server on a CentOS 7 machine. We will also install the puppet agent on a second virtual machine. We will do our first puppet run
In this video, we will learn how certificates work in a puppet client/server architecture. We will also see how the puppet server controls which client is able to access the server using certificates.
Explain why and how puppet uses certificates
Explain how a puppet agent requests a certificate
Gain insights into import configuration settings
In this video, we are going to explain how certificate auto signing works. We will also learn about different ways to control auto signing of agent certificates.
Explain what auto signing is used for
Identify the different types of auto signing
Identify important configuration settings
We will learn how puppet uses the main manifest to compile a catalog for a puppet node. We will also see how puppet finds additional puppet modules and get a short introduction to puppet environments.
Explain how puppet finds the main manifest
Explain how puppet finds additional modules
A short introduction to puppet environments
We will explain what puppet environments are and how we can assign different environments to nodes. We will also learn about environment specific settings.
Explain puppet environments
Explain how environments are assigned to nodes.
Understand how to set environment-specific settings.
We will learn about version control systems and their importance in managing puppet code on production servers. We are going introduce GIT, a distributed version control system.
Explain version control.
Introduce GIT, a distributed version control system.
Give examples of repository hosting.
We will learn how to integrate GIT with r10k, a tool for deploying puppet code. This covers using GIT branches for creating puppet environments. We will also deploy a custom module with the help of r10k.
Introduce r10k, a tool for puppet code deployments.
Understand the control repository.
Deploy custom modules with r10k.
We will learn how separating data and code allows more flexibility when writing puppet code. Hiera, a hierarchical database allows us to separate data and code in Puppet.
Explain the purpose of hiera, a hierarchical database.
Explain hiera data lookup.
Use hiera for assigning classes to nodes.
We will learn where to find import log files and how to run the puppet server in the foreground and debug mode. We will also demonstrate some tools for debugging connection problems.
Identify important log files.
Run the puppet master in the foreground and debug mode.
Demonstrate the tools for debugging connection problems.
Configuration and management of infrastructure is important. Equally important is the process of reporting, without which there would be no way to find out if the infrastructure is behaving the way we want it to.
Understand Puppet reports and configuration parameters for reporting
Explore built-in report processors
Configure and test reporting in Puppet server and agents
PuppetDB already provides us with enormous possibilities to find out extremely granular details of our infrastructure. Puppetboard, a community project, leverages the features of PuppetDB to provide a very decent dashboard for reporting.
Understand and install Puppetboard as Puppet dashboard
Create the Python config files and configure Apache web server
Run and explore the Puppetboard dashboard
Puppetexplorer is another community project to assist users in reporting and displaying an attractive dashboard to use PuppetDB data.
Understand and explore Puppetexplorer as Puppet dashboard
Install and configure Puppetexplorer
Explore Puppetexplorer dashboard to view Puppet report data
Dashboards are definitely useful for viewing reports. However, their usefulness increases many folds as soon as we are able to manipulate data on the dashboard and integrate it with tools such as Puppet to manage the infrastructure. Foreman is one such tool.
Understand, install and configure Foreman
Configure report processor for Foreman
Explore Foreman dashboard and view reports
Apart from acting as a reporting dashboard for Puppet, Foreman has additional features such as acting as an External Node Classifier for Puppet. With minimum changes to the configuration, an efficient ENC can be configured on the Foreman.
Understand and configure foreman-proxy on the Puppet Server
Configure Puppet proxy and authentication rules for Foreman proxy
Configure ENC script to provide catalog and push facts
The community has huge support of all kinds and the number of resources that can be found to be used readily are overwhelming. However, at times, the need to build custom solutions is unavoidable. Puppet has support for custom reporting which can be used to create reporting mechanisms of various types.
Understand report processor and directory structure
Create report processor and add code to send email reports
Demonstrate email report generation via Puppet run
Puppetboard and Puppetexplorer are definitely great resources for readily implementing Puppet reporting. However, Puppet is flexible enough to allow us to create our own reports from PuppetDB by querying the data.
Understand query mechanism of PuppetDB
Create Ruby script to query PuppetDB and fetch data
Use PuppetDB data to create formatted file and HTML file
One of the excellent dashboard platform out in the infrastructure space is the Elastic stack created with Elasticsearch, Logstash, and Kibana. It has taken data collection and dashboards to a whole new level and Puppet can also take advantage of it by sending reports to Logstash and viewing data in Kibana.
Understand the Elastic stack applications and tools
Configure the Elastic stack using Puppet modules
Use Puppet module to configure custom reporter for Logstash
One of the oldest monitoring tool in the infrastructure space, Nagios has its exclusive Puppet types available. When used with PuppetDB, it can create a dynamic and efficient monitoring workflow.
Explore available Nagios Puppet types
Create Nagios module with server and client manifests
Use PuppetDB to export and collect virtual resources for dynamic configurations
Using Puppet to deploy changes is great but making sure that Puppet is running when it is supposed to run is one of the critical tasks to keep in mind. Using a custom script and the power of Nagios, this task can be make very simple and elegant.
Understand the concept of Puppet run time monitoring
Create Nagios monitoring script to monitor Puppet run time
Integrate script with Nagios infrastructure and run checks successfully
In a fairly large infrastructure, making sure that only valid hosts are able to get catalogs from the Puppet server is of utmost importance. With policy based certificate signing feature of Puppet, this additional security step is possible.
Understand policy based certificate signing
Create a script to check certificate signing request on Puppet server
Add certificate signing request attributes to agent nodes
Complex workflows often require that some configurations or data are kept restricted for special nodes or resources. Restricted mount points can be configured in Puppet to exactly implement this requirement by modification of few configuration files.
Understand and create new Puppet mount point
Add restriction to mount point by editing auth.conf file
Test restricted mount point on agent nodes
Tracking active changes or finding what will change without actually changing them is often a requirement in highly critical infrastructures and can also be used for audit purposes. Puppet metaparameters are extremely helpful in implementing such workflows.
Understand audit and noop metaparameters
Test the metaparameters in a module
Track changes using noop metaparameter and resources type
Identifying vulnerable system components and then keeping them secure is one of the base requirements of any infrastructure. The enormous database of community modules provide us with a lot of modules which are specifically created to help with this process.
Identify system components where security is applicable
Explore community modules for security and hardening
Test security modules on agents
No tool or platform can be deployed successfully without running into errors and problems. It can be tedious to find out underlying problems from error messages. However, some of the most common problems and errors can be readily identified with some help.
Follow best practices to avoid errors and problems
Identify problems from the errors
Apply solution for the identified problem
As infrastructure grows to hundreds and thousands of nodes, the capacity of the Puppet server reaches a limit. Load balancing in multiple Puppet servers is an optimal way to handle load and provide redundancy.
Understand a highly available and scalable Puppet architecture
Configure Puppet servers to work with load balancers
Configure Apache HTTP server as a load balancer for Puppet service
As infrastructure grows, isolating the Puppet server components is a better way to manage them, rather than having a monolithic Puppet server node. Let’s take a look at isolating and centralizing the Puppet certificate authority service, which is an important step in this process.
Understand the split CA architecture
Create and configure the Puppet CA node
Integrate the Puppet CA node with the load balanced infrastructure
With the growth in the number of agent nodes, maintaining a database and the history of node configuration and metadata is crucial for any infrastructure. PuppetDB helps to maintain this database and also aids in configuration via exported resources.
Install and configure the PostgreSQL database
Install and configure the PuppetDB, and connect it to the PostgreSQL database
Integrate PuppetDB with Puppet server
Infrastructure management involves services with server and client based models. The server has to push or pull the data from various nodes in the infrastructure for which it needs the node specific information. To automate this process, the methods of exporting and collecting in Puppet modules are extremely useful.
Define virtual or exported resources in Puppet client modules
Send exported resources information to PuppetDB
Get exported resources data from PuppetDB and generate configuration using server modules
After the data from thousands of nodes are stored in a database, they are not very useful, unless they can be queried and visualized using structured querying mechanisms. Different Puppet query mechanisms and dashboards helps in solving these problems.
Configure workstation, and understand methods to query PuppetDB
Understand PuppetDB API endpoints and perform queries.
Use default PuppetDB dashboard for visualization
As the number of nodes and service roles grow in an infrastructure, it is important to have proper node classification mechanisms so that nodes are able to receive catalogs correctly and without latency. Proper manageable node classification methods are also desirable.
Understand external node classification concept
Create scripts to provide node definition and classification data
Configure Puppet server to use scripts as source of node classification data
Hostname based node classification often gets very complex and it is easier to switch to other node classification methods. Apart from providing data isolation from code, Hiera can also be an effective framework to provide node classification data based on metadata such as facts.
Understand Hiera file structure for node classification
Configure Hiera for node classification
Populate Hiera data file with fact specific data and module information
Running Puppet components in Docker containers is a bit different than doing it on a physical or virtual machine. It is important to understand how the service components work in an isolated manner and also in the form of an application stack.
Understand containerized service deployment method and configure Docker on workstation
Configure and run Puppet server in Docker container.
Configure and run Puppet server side components using Docker Compose
Running Puppet agent in Docker Containers to automate them is different from the general process; given the fact that container based dynamic environments don’t have hostnames for the containers and system based process control systems are unavailable in containers.
Create a Docker file for Puppet agent
Run Docker containers with Puppet agent
Use Hiera to fetch service specific catalog data
The fact that containers are able to be spawned within few seconds is true. This is due to the fact that Docker images already have all the configurations baked in them and provisioning during the container launch is not required, thus bringing down the deployment time. Creating images is a crucial step in container based environments.
Build and push image manually
Build images with dependencies using image_build Puppet module
Use Packer to build, tag and push Docker images
Once Puppet server side components and containers are ready to be spawned using Docker images, it is important that we are able to launch application stacks involving multiple components such as application and database. Docker Compose can be managed efficiently by use of respective Puppet modules and help us in managing container based application stacks.
Understand community modules to manage Docker
Create Puppet module to configure Docker environment
Create Puppet module to deploy application stacks using Docker Compose
Once infrastructure reaches a considerable size, a requirement is to be able to perform specific tasks on specific nodes. Managing and maintaining an inventory of role based nodes is a tedious process. MCollective can be a very efficient tool in implementing orchestration which helps us to target nodes based on metadata and run specific tasks
Understand Orchestration and various components of the framework
Configure ActiveMQ message broker as the middleware
Configure MCollective servers and clients using Puppet modules
Although facter provides a lot of system facts, often in complex environments, they are not enough. By writing custom facts based on our infrastructure, we can add meaningful custom data to our infrastructure nodes based on which complex tasks can be performed.
Understand custom fact concept and directory structure
Write custom fact
Make facts dependent on system or other custom facts
Puppet built-in types are very helpful in writing configurations. However, for simplified use of Puppet modules, often writing custom types are very important. For users who don’t wish to get into the complex methods of writing Ruby based Puppet types, resource types are a simpler alternative where custom types can be written using the Puppet language.
Learn concept of resource type
Write custom resource type using ‘define’ method
Apply resource type to agent nodes
With the numerous tools and services available in the infrastructure ecosystem, it is quite exhausting and time consuming to write Puppet modules for them from scratch using built-in Puppet types. For such scenarios, advanced Puppet types can be written using Ruby which can simplify custom tasks.
Understand custom Puppet type concept and directory structure
Create custom type and understand options
Apply custom Puppet type to agent nodes
Puppet types are useful in defining custom tasks and processes, however without Providers, Puppet types are unable to do much. If Puppet types are the interface to create custom configurations, providers are the backend logic which actually performs the tasks on the systems. Multiple providers can be written for systems or services with different behaviors.
Understand custom provider concept and directory structure
Create custom provider and add functionality based on Puppet type
Apply custom provider to agent nodes
Although Puppet types and Providers help us in creating advanced tasks and interfaces, for some smaller tasks and scenarios, it is enough to create functions. Custom functions can be created in the Puppet language and Ruby to handle such scenarios.
Understand custom function concept and directory structure
Create custom function and use it in Puppet manifest
Apply manifest to agent nodes