Mastering Shell Scripting: From Basics to Automation

In this session, we introduce the foundational concepts of shell scripting and establish its critical role in modern IT and DevOps environments. The core purpose of scripting is presented as the primary method for automating repetitive tasks, thereby increasing efficiency, reducing errors, and saving time. A clear distinction is drawn between programming languages, which are used to build tools and commands, and scripting languages, which are used to orchestrate those existing commands into a powerful, automated sequence. The session defines the "shell" as the command-line interpreter that executes these commands and scripts.

Key topics covered include:

• The primary goal of scripting: to automate day-to-day operational tasks.

• The difference between a programming language and a scripting language.

• The role of the shell (e.g., Bash) as the environment where scripts are executed.

• Practical industry use cases, including automation in system administration, networking, cloud infrastructure, and CI/CD pipelines.

By the end of this session, you will understand what shell scripting is, why it is an essential skill for technology professionals, and how it is used to solve real-world automation challenges.

This session focuses on the Bash (Bourne-Again SHell) shell, the most common shell used in Linux and other Unix-like systems, and explains how to set it up on Windows using Git Bash. It clarifies the important distinction between commands that are built directly into the shell (internal commands like echo or keywords like if) and commands that are separate programs provided by the operating system (external commands like cal or date). While shell syntax is portable, the availability and behavior of external commands depend on the underlying OS.

Key topics covered include:

• An introduction to the Bash shell as the target environment for this training.

• The difference between internal shell commands and external OS commands.

• Real-world scripting use cases, such as creating automated backups, monitoring system performance, and managing log files.

• The relationship between a script and the sequence of commands a user would otherwise type manually.

By the end of this session, you will understand the role of the Bash shell, how to access it, and be able to identify practical scenarios where a shell script can provide significant value.

In this session, we cover the essential building blocks for writing a shell script. We explore how to store and retrieve data using variables and how to print output to the screen using the echo command. A critical concept covered is the difference between single and double quotes, which determines whether variables inside a string are expanded or treated as literal text. The session also introduces command substitution, a powerful technique for capturing the output of a command and storing it in a variable for later use.

Key topics covered include:

• Declaring and referencing variables (e.g., x=5, echo $x).

• The functional difference between double quotes (") and single quotes (').

• Capturing command output into a variable using command substitution (backticks `command` and the $(command) syntax).

• Reading user input from the keyboard using the read command.

By the end of this session, you will be able to write a basic shell script that can store data in variables, display formatted output, capture the results of other commands, and interact with the user.

This session introduces methods for making scripts more dynamic and reusable by interacting with the command-line environment. We cover positional parameters, which are special variables that allow a script to access arguments passed to it when it is executed (e.g., $1 for the first argument). This enables the creation of flexible scripts that can operate on different inputs without being modified. We also explore output redirection, which provides control over where a command's output is sent, such as saving it to a file instead of displaying it on the screen.

Key topics covered include:

• Using positional parameters ($0, $1, $2, etc.) to read command-line arguments.

• Handling arguments that contain spaces by enclosing them in quotes.

• Redirecting standard output to a file, overwriting its contents with >.

• Appending standard output to the end of a file without overwriting with >>.

By the end of this session, you will understand how to design scripts that accept inputs from the command line and how to manage the script's output by saving it to files.

In this session, we begin applying our knowledge to a practical, real-world project: a script to monitor system memory usage. We introduce the free command to retrieve memory statistics and the expr command to perform arithmetic calculations. A key challenge in shell scripting is addressed: handling special characters that have a different meaning to the shell. Specifically, we demonstrate how to "escape" the asterisk character (\*) so it can be used for multiplication instead of as a filename wildcard.

Key topics covered include:

• Using the free command to get system memory information.

• Performing integer arithmetic with the expr command.

• Escaping special shell characters (metacharacters) to use their literal meaning.

• Identifying the need to parse and extract specific data from a command's text output.

By the end of this session, you will have started building a functional monitoring script and will understand how to perform calculations in the shell and manage special characters.

This session provides the solution to the data extraction problem introduced previously. We introduce the pipeline (|), one of the most powerful concepts in the shell, which allows you to chain commands together by sending the output of one command as the input to the next. Using this technique, we build a multi-stage command to precisely extract the memory usage figures from the free command's output. This involves using the head and tail commands to isolate the correct line and the versatile awk utility to select specific columns of data from that line.

Key topics covered include:

• The concept of a pipeline (|) for chaining commands.

• Using head and tail to filter for specific lines of text.

• An introduction to the awk command for field-based text processing.

• Combining multiple commands in a pipeline to parse complex output.

By the end of this session, you will be able to construct command pipelines to effectively parse, filter, and extract any piece of information from text-based command outputs.

In this final session, we achieve true automation by scheduling our memory monitoring script to run automatically at regular intervals. We introduce cron, the standard job scheduler in Linux and Unix-like systems. The session explains how cron works and how to edit a user's crontab file to define a schedule for a command or script. We configure a cron job to execute our script every minute, transforming it from a manually-run tool into a fully automated, hands-free monitoring solution that generates a continuous report over time.

Key topics covered include:

• The purpose of a job scheduler for automation.

• An introduction to the cron service.

• The structure and syntax of a crontab file (minute, hour, day, month, week).

• Configuring a cron job to run a shell script on a recurring schedule.

By the end of this session, you will understand how to use cron to schedule any script to run automatically, completing the final step in the automation workflow.

In this session, we explore the fundamental mechanism for determining whether a command has executed successfully or failed. It introduces the concept of the exit code, a numerical value that every command returns upon completion. By convention, an exit code of 0 signifies success (or "true"), while any non-zero value indicates a failure (or "false"). This concept is presented as the foundation for all decision-making in shell scripting.

Key topics covered include:

• The problem with relying on a command's visible output to determine its success.

• The concept of an exit code as the definitive status of a command's execution.

• The special shell variable $?, which always holds the exit code of the most recently executed command.

• The convention that an exit code of 0 means success/true, and any other number means failure/false.

• A practical example using the ping command to check internet connectivity, where a successful ping returns 0 and a failed ping returns a non-zero exit code.

By the end of this session, you will understand how to programmatically verify the success or failure of any command by checking its exit code using the $? variable.

This session builds directly on the concept of exit codes to introduce conditional logic using the if-else construct. It explains that the if statement in shell scripting does not evaluate boolean expressions directly but instead checks the exit code of a command. If the command's exit code is 0 (success), the then block is executed; otherwise, the else block is executed. This allows scripts to make decisions and perform different actions based on the outcome of a command.

Key topics covered include:

• The basic syntax of the if-else-fi block in shell scripting.

• How the if statement uses the exit code of a command to make decisions.

• Using comparison keywords for numbers (-eq, -gt, -lt) with the test command.

• Building a practical script to check for internet connectivity and print a user-friendly message based on the success or failure of the ping command.

• How to suppress unwanted command output by redirecting it to /dev/null.

By the end of this session, you will be able to write scripts that can make intelligent decisions by using the if-else construct to react to the success or failure of commands.

In this session, we reinforce the use of the if-else construct by focusing on scenarios where a decision is based directly on the success or failure of a command's execution. This is a common pattern in scripting, used for tasks like ensuring a resource exists before using it. The session demonstrates a practical example of creating a directory, where the script first checks if the directory can be created successfully.

Key topics covered include:

• The two primary syntaxes for if statements: one for direct command execution and another for data comparison using test.

• A use case for creating a directory (mkdir), where the if statement directly runs the mkdir command.

• If mkdir succeeds (exit code 0), a success message is printed.

• If mkdir fails (e.g., the directory already exists), the else block is triggered, allowing for alternative actions, such as creating a directory with a different name.

• The importance of checking command status in critical operations like backups or security implementations to ensure they have not failed silently.

By the end of this session, you will understand how to write robust scripts that don't just run commands blindly but can confirm their success and handle failures gracefully.

This session expands on conditional logic by introducing more complex comparisons and the concept of nesting if statements. It clarifies the different operators used for comparing numbers (-eq, -gt) versus strings (==, !=). The modern [[ ... ]] syntax is introduced as a more flexible and preferred alternative to the older test command. These concepts are then combined to build an interactive script that asks the user for confirmation before performing an action.

Key topics covered include:

• Comparison operators for numbers (-eq) vs. strings (==).

• Using the double square bracket [[ ... ]] syntax for conditional tests.

• Creating a nested if statement, where an outer if block contains another complete if-else structure.

• Adding conditional logic to the memory monitoring script, enabling it to trigger an alarm (using the espeak command) only when memory usage exceeds a specific threshold (e.g., 80%).

• Using curly braces ({..}) for generating number sequences to simulate memory load for testing purposes.

By the end of this session, you will be able to build more sophisticated scripts with complex decision-making logic, including handling different data types and nesting conditions.

In this session, we introduce loops as a way to execute a block of code repeatedly. The focus is on the while loop, which continues to execute as long as a specified condition is true. The session demonstrates how to create both finite loops (that run a specific number of times) and infinite loops (that run forever). This concept is then applied to the memory monitoring script, transforming it from a single-run check into a continuous, 24/7 monitoring service.

Key topics covered include:

• The fundamental concept of a loop for repeating tasks.

• The syntax of the while loop (while [ condition ]; do ... done).

• Creating an infinite loop using while : or while true.

• Using the sleep command to introduce a delay within a loop, preventing it from consuming excessive CPU resources.

• Integrating an infinite while loop into the memory monitoring script to make it run continuously, checking the system status every few seconds.

• Using the exit command to terminate a script programmatically from within a conditional block.

By the end of this session, you will be able to use while loops to create scripts that can perform tasks continuously, a core requirement for monitoring and daemon-like services.

In this session, we delve into some of the finer points of shell syntax and the importance of properly handling variable data, especially when it might be empty. It introduces the use of curly braces for safe variable expansion, which helps the shell distinguish between a variable's name and adjacent text (e.g., ${X}th). The session also covers how to test if a variable is empty, a crucial check before using it in a command that might fail with no input.

Key topics covered include:

• Using cheat sheets as a reference for the vast number of shell keywords and syntaxes.

• Using curly braces (${variable}) to clearly define variable boundaries, preventing ambiguity (e.g., ${X}th vs $Xth).

• The importance of validating data, especially input from users or other programs.

• Using the test -z operator to check if a variable string is empty (null).

• Building a more robust script that checks for empty input and prompts the user to re-enter data if necessary, preventing the script from failing unexpectedly.

By the end of this session, you will be more familiar with nuanced shell syntax and understand how to write more defensive scripts that can gracefully handle missing or empty data.

This session introduces Regular Expressions (Regex) as the definitive tool for powerful and precise pattern matching. It explains that simple string searches are often insufficient and can lead to bugs, as seen in the previous session. Using the online tool regex101.com for demonstration, the session covers the foundational building blocks of regex, allowing you to define complex patterns that go far beyond simple keyword searching.

Key topics covered include:

• The definition of regex as a language for describing text patterns.

• The . (dot) metacharacter, which matches any single character.

• Character sets [...], which match any single character from a specified list or range (e.g., [bde], [a-z]).

• Negating a character set with [^...] to match any character not in the set.

• Quantifiers * (zero or more occurrences) and + (one or more occurrences) for matching repeating characters.

• Using ^ for the start of a line and $ for the end of a line to anchor a pattern's position.

By the end of this session, you will understand the core concepts of regular expressions and be able to write basic patterns to match specific sequences of characters.

This session demonstrates how to apply the learned regex concepts to solve practical data extraction problems. It solves the previous bug with grep by using regex anchors (^ and $) to ensure that the script matches an exact username at the beginning of a line or a specific shell at the end of a line. The session provides two hands-on tasks: writing a regex to extract all valid email addresses from a block of text and writing another to extract all valid IP addresses.

Key topics covered include:

• Applying regex patterns with shell commands like grep and awk.

• Using anchors (^, $) to solve the pattern-matching bug and ensure exact matches.

• The concept of character classes like \w (any word character) and \d (any digit) as shortcuts.

• A step-by-step approach to building a complex regex for matching email addresses.

• The challenge of pattern matching vs. validation, highlighting that regex is excellent for finding patterns (like IP addresses) but may require further scripting for strict validation (e.g., checking if an octet is > 255).

By the end of this session, you will be able to apply regular expressions in your shell scripts to perform sophisticated and accurate data extraction and filtering from an

This session introduces basic networking concepts in shell scripting using the netcat (nc) utility. It demonstrates how to set up a simple client-server connection where one machine listens on a port and another connects to it to send data. This is used to build a practical, multi-system ETL (Extract, Transform, Load) pipeline. In the demo, Computer A sends raw data, Computer B (the central server) receives and filters it (the "transform" step), and Computer C receives the final, clean data for storage.

Key topics covered include:

• An introduction to the netcat (nc) command for creating TCP/UDP connections.

• Using nc -l [port] to set up a listening server.

• Using nc [host] [port] to connect as a client and send data.

• Combining nc with pipelines to create a simple ETL process across multiple machines.

• A demonstration of a three-computer setup where the middle machine acts as a filter, removing lines containing the word "virus" before forwarding the data.

By the end of this session, you will understand how to use netcat to send and receive data between different machines and how to construct a basic distributed data processing pipeline.

This session extends the networking ETL pipeline by introducing a more advanced transformation technique: data masking. Instead of completely removing sensitive lines, this approach replaces sensitive information with placeholder characters (like ***). The session introduces the sed (Stream Editor) command, a powerful tool for performing text substitutions. The demonstration shows how to replace the word "virus" with asterisks before forwarding the data, effectively masking it.

Key topics covered include:

• The concept of data masking for protecting sensitive information while preserving data structure.

• An introduction to the sed command for stream editing.

• Using sed's substitute command (s/pattern/replacement/g) to find and replace text.

• Integrating sed into the netcat-based ETL pipeline to perform real-time data masking.

• The real-world relevance of this technique, comparing it to expensive commercial data masking tools and highlighting the power of shell scripting to build custom solutions.

By the end of this session, you will be able to use the sed command to perform find-and-replace operations and build a script that can mask sensitive data as it flows through a pipeline.

In this session, we dive into the concepts of process management and signals. It explains that when a program runs, it becomes a process with a unique Process ID (PID). The session introduces signals, which are notifications sent to a process to trigger specific actions, such as termination. It demonstrates how to use the kill command to send different signals to a process, such as SIGSTOP (signal 19) to pause a process and SIGCONT (signal 18) to resume it.

Key topics covered include:

• The relationship between a program, a process, and a PID.

• The concept of signals as a form of inter-process communication.

• A list of common signals and their purposes (kill -l).

• Using the kill command to send signals to a process by its PID.

• A practical demonstration of pausing and resuming a graphical application (Firefox) from the command line, illustrating the power of signals to control running processes.

By the end of this session, you will understand what processes and signals are and how to use the kill command to control the state of running applications.

This session introduces the trap command, a powerful shell built-in that allows a script to "catch" or intercept signals and execute a custom piece of code before the signal's default action occurs. This enables scripts to handle interruptions gracefully instead of just abruptly terminating. The session demonstrates how trapping a signal like SIGINT (sent by Ctrl+C) can prevent a script from stopping, and explains that some signals, like SIGKILL (signal 9), cannot be trapped.

Key topics covered include:

• An introduction to the trap command for handling signals within a script.

• The syntax for trap: trap 'command_to_run' SIGNAL_NAME.

• A demonstration of trapping Ctrl+C to run a custom command instead of terminating the script.

• Understanding that SIGKILL (signal 9) is a special signal that bypasses trap and will always terminate a process forcefully.

• A "hacker tool" example showing how trap could be used maliciously to relaunch a script when someone tries to kill it.

By the end of this session, you will be able to use the trap command to make your scripts aware of external signals and execute custom code in response to them.

This session presents the most important and practical use case for the trap command: ensuring a graceful shutdown. It demonstrates a common scenario where a script creates a temporary file containing sensitive data (like a password). If the script is interrupted with Ctrl+C before it can clean up, the sensitive file is left on the disk, creating a security risk. The session introduces functions as a way to organize reusable blocks of code and then uses trap to call a cleanup function upon interruption.

Key topics covered include:

• The concept of a graceful shutdown and the importance of cleanup operations.

• The security risk of leaving temporary files behind after an unexpected script termination.

• How to define and call a function in shell scripting.

• Combining trap with a function to create a robust cleanup handler that removes temporary files and exits cleanly, even when the script is interrupted.

• The exit command, which is used within the function to ensure the script terminates properly after the cleanup is complete.

By the end of this session, you will be able to write professional, resilient scripts that use functions and trap to handle interruptions safely and leave the system in a clean state.