Advanced Bash Shell Usage

David Cohen
A free video tutorial from David Cohen
10+ year Cloud, Linux, DevOps, and Software Professional
4.6 instructor rating • 1 course • 3,573 students

Lecture description

You'll need more than just the absolute basics of command-line/shell usage with Bash to be a competent system administrator. Here's the rest of the bash basics you need to have under your belt. In this video, I'll cover:

  1. Hidden files and directories.
  2. Input and output redirection (appending to files, overwriting files, taking input from a file).
  3. Logical AND; Logical OR
  4. Command substitution (how to insert the output of one command into a string, or another command).
  5. How to use the 'grep' command to search for matching lines in files.
  6. How to use Pipes in Linux, to connect the output of one command to the input of another command.

Learn more from the full course

Hands-on Linux: Self-Hosted WordPress for Linux Beginners

A practical, project-based crash course that prepares you for real-life Linux and Cloud work

08:54:52 of on-demand video • Updated January 2021

  • Learn basic Linux system administration by setting up a WordPress hosting platform
  • Configure a production-grade WordPress install on Linux
  • Configure your hosting platform on Amazon Web Services (AWS)
  • Set up TLS (for free!) with letsencrypt so your users can enjoy the security of HTTPS
  • Tune webserver performance and set up caching for lightning-fast page loads
  • Set up and configure the popular MySQL database
  • Be comfortable working with an nginx web server
  • Configure monitoring for your web hosting server
  • Create and manage Linux system users
  • Manage Linux file permissions
  • Understand the basics of how HTTP, the Web protocol, works
  • Understand basic and more advanced Bash shell concepts and skills
  • Schedule commands to run periodically on Linux with Cron
  • Manage remote servers using SSH
  • Automate repetitive tasks with Ansible, a powerful automation and configuration management tool
  • Create and Restore website backups, both on the filesystem and in the MySQL database
  • Effectively perform security hardening on Linux servers and services
English [Auto] And this video we're going to explore a little bit further on the shell. You've got the absolute basics down but you need a little bit more to get through the rest of this course. These are things you'll use every day. We're going to cover hidden files and directories how to see them create them how to manage them. Input and output redirection taking output that would normally be printed to your shell and redirecting it to let's say a file will need some dynamic stuff in our commands. So I'm going to show you the basics of command substitution how to embed the output of some command inside of something else in bash We may recover logical and logical or basic grepping and pipes. So this is definitely a crash course. I'll give you some resources at the end. For more study Bash is just one of those things that you need to be competent at as a science admin but this should be enough for the rest of the course. So let's get started with hidden files and directories. This is the simplest thing we're going to cover. If I list this directory it looks nice and empty. If I list all I'm going to see hidden files and directories there's three items here. So let's just go through them. You've already covered in the first specialty of. This is shorthand syntax for this current directory. This is the directory above where we current currently are. So if I say a city dot dot that means go up one directory to my home directory CD to their current directory CD datt just means go to the directory that I am currently in. Which means as much as nothing. You stay here. But there's this other file that we can see now which is a generally it's probably fine it's some file. The only thing that's special about it is that the name begins with a period. It's just a regular file but it's been hidden because our shell displays files and directories that begin with a period as well. It doesn't display them unless you use the all flag. So list a will show hidden files and directories there's no special trick to creating these. I'd created this file with them echo and some output redirection. You could just use touch to create an empty file like or make dir. And you can see list all you can see them and a regular list. You can't let's move on to input and output redirection. For those of you not familiar with the echo command it just echoes back whenever we give it one nice thing you can do with it is redirect its output. So if we let's say have a log file called log file log doesn't exist yet. But this greater than symbol will we could think of it as an arrow. It takes whatever output would be printed to standard out which is sort of the output that we get on the shell here. It just redirects it over it sort of put follows this arrow. You can think of it as. So we're going to pour this into file out log. When all is said and done log file that contains that line. Now you're thinking great I can use this for logging. Right. This is how I'm going to do my logs. There's actually a problem. If we run this again we still have the same one line we run it again we're expecting three lines here. But there's still only one line because a single greater than sign overwrites. So this is output redirection. But what we actually want is not to overwrite this file but to append to it. Appending is done with two greater than symbols. So echo Hi there into log file log. This will depend and if we do this two or three more times you'll see those lines actually add up instead of overwriting the files content each time. This is a classic fun beginner mistake which you will probably make at some point and you'll think that what I said was just one line in the logfile I just can't figure it out. Well this is probably what you're doing. Input redirection is also a topic projet if you'd like to mail something we could say mail subject important greetings to Dave tutorial and dot com. And then instead of specifying content on the command line if this runs several ones we use the smaller than sign to take input from our logfile that makes sense. So in this case you can still think of it as an arrow that's taking whatever the output or content of this file or command is and pouring it into this command over here which will mail with the subject important. Greetings to David to the next dot com. So let's input and output redirection. Something you'll be just using all the time. So make sure this is second nature. Not very complicated. So we've got our little logfile here but it doesn't really do anyone any good to have a bunch of wind output without a time stamp. So we need a time stamp in Linux and Unix the date command can be run to get a current time stamp. You can pass options to the date command. You can check that out with a man date but we're just going to use it the way it is here. More about man pages in just a moment. So let's say we want to echo something that's preceded by the state time stamp into our command the way we would do it so by saying Been date. So we want to separate it with a hyphen Hi there into live filed outloud and we're appending to that file. The syntax I'm using here is dollar sign open parentheses the full absolute path to that command it's been Date close paren and then whatever text someone afterwards. So let's do that a couple of times and check out our logfile again. Not too bad. This is what we're looking for. Just a couple of notes on how I sort of went about that if you know the date command works and you'd like to get the absolute path to that commands. Just more secure to store it like that in scripts. You would say which date and that shows you exactly where the date program that's then run is actually stored. So it shows you where it finds the first date program that it sees. So again command substitution. You simply use a dollar sign open paren the command you want. You could if you repassing options you would just pass them in here just as if it were a brand new shell and then close friends and continue with whatever string you're writing or whatever you're doing just because I mentioned man pages there will be a separate video on this later on. But just so you know if you ever have questions about a certain commands that we're running try using man for manual and then your command. For example if you're interested in options you can use with Date. Just type in Mandate sense a little bit strange admittedly but this gives you the manual page for the date command gives you a synopsis a description of all the options you can pass to it. And this is a fairly large man page so don't be intimidated. Most commands are a little bit smaller man page than this. A good man page will also give you examples if you want to see amazing examples of manual pages. Check any of the BSD is open BSD Free BSD not BSD. Dragon-Fly BSD etc.. So that's how you sort of quickly get help on a specific command. OK so we've got dynamic stuff. Time to look at logical and logical or go use these a little bit less but it's good to understand them and know that they're there. I'm going to echo Hi again then I'm using this ampersand ampersands syntax to say this is a logical and will say Echo. Goodbye. What this is basically saying is I would like both of these to be true. This whole expression will return true if both of the things on each. So the thing on the left and the thing on the right of the end is true and is true if this and that are true. So this is an echo statement which will exit with no problem. And the other thing will also be an echo statement which exits with no problem. So they're both true. If I had an error on the first command. By calling a non-existent commands then the goodbye would not run because the and has the short circuit behavior where if the first thing isn't true. Well then it knows that both can't be true so it returns false. Right A and B can't both be true which is and is asking if the first thing is false so it won't ever even evaluate the second part of this expression. This is a little bit abstract but I'm going to show you just a silly practical use for this in just a moment. So why don't we add something to our logfile that echoes done when it worked. OK so we just wrote something to our logfile. You can pretend that this is something that takes you know seven hours to complete. And then if it completes successfully we echo done. Well there it is standard out show's done. This is something that breaks like something that only returns output and standard error that is there's an error while then this will not run. So there's never a done for us to see. Let's try this one on. I'm going to call a non-existent command here and use the OR operator. Here is a logical or so we're saying is this or can return true this whole expression can return true if one of the parts of this expression evaluates to true. So this is the non-existing command. But this can still return true so it will still be tried. This is a nice way of guarding against some kind of failure of doing something in the case of failure when we execute this we get the command. I found that we expect. And now this will have created a failure like with this content inside the inverse of this is lists let's say we're just calling latests or that is if the listing for some reason fails and we're going to redirect that appendant to the failure log. Well the list is going to run so the OR already knows you have one of these is true and that's all I care about so I'm going to return True which means the second part of the expression isn't run. So if we checked out failure like there's only that first line nothing's been added. The list fails will only get added this command didn't work. So evaluate it to False in the context of this logical or I know this is a little bit you know above our pay grade as beginners but know that this is there and this will give you something to refer back to when you're curious about using that behavior. I don't recommend having commands that you know use a ton of these sorts of logical operators because it can get complicated just evaluating everything in your head to see what's actually going on. But they are certainly useful and there's a time and place to use them. Now we're ready for something really fun and that is great. Open the main page for this because this is one of the most venerable commands in the Linux and Unix toolbox. It's huge. It's flexible it's powerful. It's amazing. Learning graph and learning different flags that it has and making it do new things and expanding your grep abilities specifically with regular expressions which we're not covering in this course will help you find things like a freakin Wizard your windows friends will envy you. So you file UTC. Network interfaces. This is just our interface file. But in fact I'm actually going to log into our server to show you there because the interface files a little bit more interesting there. OK so we've actually got a static IP before address the network interfaces file you don't need to memorize this right now. It's just a file where you can see the interfaces that your system is going to try to bring up after booting. Here it's saying automatically bring up ethernet interface zero. We're going to say interface Internet 0 at static it's getting a static Internet address the address will be this address. Here are net mask. Don't worry about it. As for when you start doing networking This will make a lot more sense. Or gateway address the router we're talking to which is going to connect us out to the rest of the network with the rest of the Internet is here. So this is how we get out. And couple other settings. Let's say we want to get all the interfaces that are brought up automatically so that would be loopback and the threat. Loopback is just a local system only interface. So what we can do is say crep auto PGC network interfaces and this will find and highlight all the lines containing auto in that file. Doesn't give any context. But that's basically what it does. So you can search through files find matching lines with grep. It's an incredibly enormously powerful tool and you'll be using it all the time. I'm only showing you the very very very very basics here. But we're going to use it in just a moments when we cover pipes water pipes pipes are this character. You might remember it from the logical or but just a single pipe character here this character is just to the left of your Enter key on an American U.S. English keyboard layout. It's in all kinds of weird places on different keyboard layouts. I'm sorry but you can tie together multiple commands and what I mean by that is for example let's use this command again. We're grepping for auto in that interfaces file. That's going to return these two lines. Let's say we want to filter these down some more. Let's say we're only interested in the if zero line so we can just say grep easier and we'd get just that line. I know this doesn't make a lot of sense in this specific context but I'm just showing you what the tool does. He wanted to use something like cut with a delimiter of space. Meaning this cuts the matching lines apart at the space character and we'll take a field too. That would give us just the second field. You don't need to worry about the cut command I'm just showing you this takes the output of one commands and feeds it into the input of another command. It's called a pipe because it connects together to separate commands much like a pipe connects different parts of plumbing. This is one of the real magical parts of Linux and Unix shell programming because pipes are really what make UNIX Linux so great the environment is such that you have many small sharp tools that is very specific commands that do one thing very very well and it's up to you to know those tools and tie them together for whatever you need. For example a command like this which will use to create database passwords or any secure password that we need uses multiple pipes can see we're dumping out whatever data we've got and dev random using a pipe to take that data then and feed it into this commands. Now really this command and then we're taking whatever comes out of that we're basically just looking for alphanumeric characters here and then we're feeding those into the head command and grabbing just 14 characters. So make sense. This is really stringing together a bunch of different things. We're reading a file We're then running a command on the output of that file and trimming it down with another command to just the stuff we want. This is an incredibly powerful paradigm for working on an operating system because it saves you from having to make the the 14 character password application or program you can simply string together stuff that you've already got in exactly the way you need right now. So this simply gives us a fortune character password. You could say I just added one more thing to the end and that is an empty statement. So we get this on a separate line. So that's pretty much it. It's a whirlwind tour. I know this is a lot to take in but it's all important and understanding these things specifically input and output redirection pipes and command substitution. They are things that will make you instantly more powerful on the command line and they make it possible to read you know cryptic and terrifying looking commands like this and you realize you can kind of decompose it. There's no magic here. First we're reading out a file then we're you know passing an environment variable where running a command with running another command you can start decomposing and understanding what's actually going on. You can dig much much much much deeper and I'll have some resources for you for further bash study but that should get you set for now. So congratulations you've got the basics and you can now continue on on your journey a little bit more competent than you were before. See you in the next lesson.