What is Deep Learning?

Woman: What Alison should know-- Man: What is the Internet, anyway? Alison: Internet is that massive computer network, the one that's becoming really big now. Man: What do you mean that's big? How does (mumbles), you write to it, like mail? -: No, a lot of people use it to communicate with, I guess they can communicate with NBC, writers and producers. Alison, can you explain what Internet is? (dramatic music) Kirill: How amazing is that? Just over 20 years ago, people didn't even know what the Internet was and today we can't even imagine our lives without it. Welcome to the Deep Learning A-Z course. My name is Kirill Eremenko, and along with the co-instructor Hadelin de Ponteves, we're super excited to have you on board. And today we're going to give you a quick overview of what deep learning is and why it's picking up right now. So, let's get started. Why did we have a look at that clip? And what is this photo over here? Well, that clip was from 1994, this is a photo of a computer from 1980. And the reason why we're kinda delving into history a little bit is because neural networks, along with deep learning, have been around for quite some time, and they've only started picking up now and impacting the world right now. But if you look back at the '80s you'll see that, even though they were invented in the '60s and '70s, they really caught onto a trend or caught wind in the '80s. People started talking about them a lot, there was a lot of research in that area, and everybody thought that deep learning or neural networks were this new thing that is going to impact the world, is going to change everything, is gonna solve all the world problems, and then it kind of slowly died off over the next decade. So what happened? Why did the neural networks not survive and not change the world? The reason for that, that they were just not good enough, that they're not that good at predicting things and not that good at modeling and, basically, just not a good invention, or is there another reason? Well, actually, there is another reason. And the reason is in front of us, it's the fact that technology back then was not up to the right standard to facilitate neural networks. In order for neural networks and deep learning to work properly, you need two things. You need data, and you need a lot of data, and you need processing power, you need strong computers to process that data and facilitate the neural networks. So let's have a look at how data, or storage of data, has evolved over the years, and then we'll look at how technology has evolved. So, here we've got three years. 1956, 1980, 2017. How did storage look back in 1956? Well, there is a hard drive, and that hard drive is only a 5, wait for it, megabyte hard drive. That's 5 megabytes right there on the forklift, the size of a small room, that's a hard drive being transported to another location on a plane. That is what storage looked like in 1956. You had to pay, a company had to pay $2,500, of those days dollars, to rent that hard drive. To rent it, not buy it, to rent it for one month. In 1980, the situation improved a little bit. So here we got a 10 megabytes hard drive for $3,500. It's still very expensive and only 10 megabytes, so that's like one photo these days. And today, in 2017, we've got a 256 gigabyte SSD card for $150 which can fit on your finger. And if you're watching this video, a year later or, like, in 2019 or 2025, you're probably laughing to yourself because by then you have even stronger storage capacity. But, nevertheless, the point stands. If we compare these across the board and, without even taking price and size into consideration, just the capacity of whatever was trending at the time, so from 1956 to 1980 capacity increased about double and then it increased about 25,600 times. And the length of the period is not that different, from 1956 to 1980 is 24 years, from 1980 to 2017, 37 years, so not that much of an increase in time but a huge jump in technological progress, and that stands to show that this is not a linear trend, this is an exponential growth in technology. And if we take into account price and size, it'll be in the millions of increase. And here we actually have a chart on a logarithmic scale, so if we plot the hard drive cost per gigabyte you'll see that it looks something like this. We're very quickly approaching zero. Right now you can get storage on DropBox and Google Drive, which doesn't cost you anything, Cloud storage, and that's going to continue. And, in fact, over the years, this is going to go even further. Right now, scientists are even looking into using DNA for storage. Right now it's quite expensive, it costs $7,000 to synthesize 2 megabytes of data and then another $2,000 to read it, but that kinda reminds you of this whole situation of the hard drive and the plane, that this is gonna be mitigated very, very quickly with this exponential growth. 10 years from now, 20 years from now, everybody's gonna be using DNA storage if we go down this direction. Here are some stats on all that. You can explore this further, maybe pause the video if you want to read a bit more about this. This is from nature.com. And, basically, you can store all of the world's data in just 1 kilogram of DNA storage. Or you can store about one billion terabytes of data in one gram of DNA storage. So, that's just something to show how quickly we're progressing and that this is why deep learning is picking up now that we are finally at the stage where we have enough data to train super cool, super sophisticated models. Back then, in the '80s when it was first initiated, it just wasn't the case. And the second thing we talked about is processing capacity. So here we've got an exponential curve, again, on a log scale. It's not ideally portrayed here but on the right you can see it's a log scale. And this is how computers have been evolving. So, again, feel free to pause this slide. This is called Moore's law, you've probably heard of it, how quickly the processing capacity of computers has been evolving. Right now we're somewhere over here, where an average computer you can buy for 1,000 bucks thinks at the speed of the brain of a rat. And by 2025 it'll be the speed of a human, or 2023. And then by 2050 or 2045, we'll surpass all of the humans combined. So, basically, we are entering the era of computers that are extremely powerful, that can process things way faster than we can imagine, and that is what is facilitating deep learning. So, all this brings us to the question, what is deep learning? What is this whole neural network situation? What is going on, what are we even talking about here? And you've probably seen a picture of something like this, so let's dive into it. What is deep learning? This gentleman over here, Geoffrey Hinton, is known as the godfather of deep learning. He did research on deep learning in the '80s and he's done lots and lots of work, lots of research papers he's published in deep learning. Right now he works at Google. So, a lot of the things that we're gonna be talking about actually come from Geoffrey Hinton. You can see he's got a quite a few Youtube videos, he explains things really well, so, highly recommend checking them out. So, the idea behind deep learning is to look at the human brain, and there's gonna be quite a bit of neuroscience coming up in these tutorials, and what we're trying to do here is to mimic how the human brain operates. We don't know that much, we don't know everything about the human brain, but that little amount that we know, we want to mimic it and recreate it. And why is that? Well, because the human brain seems to be one of the most powerful tools on this planet for learning, for learning adapting skills and then applying them. If computers could copy that, then we could just leverage what natural selection has already decided for us, all of those algorithms that it has decided are the best, we're just gonna leverage that, (mumbles). So, let's see how this works. Here we've got some neurons. These are neurons which have been smeared onto glass and then have been looked at under a microscope with some coloring, and you can see what they look like. They have like a body, they have these branches, and they have like tails, and so on, you can see then they have a nucleus inside in the middle. That's basically what a neuron looks like. In the human brain there is approximately a 100 billion neurons altogether. These are individual neurons, these are actually motor neurons because they're bigger, they're easier to see. But nevertheless there's a 100 billion neurons in the human brain, and each neuron is connected to as many as about a 1,000 of its neighbors. So, to give you a picture, this is what it looks like. This is an actual section of the human brain, this is the cerebellum, which is this part of your brain. At the back. It's responsible for motorics and for keeping your balance and some language capabilities and stuff like that. This is just to show how vast, how many neurons there are. There are billions and billions and billions of neurons all connecting your brain. It's not like we're talking about five or 500 or a 1,000 or a million, there's billions of neurons in there. Yeah, so, that's what we're gonna be trying to recreate. So, how do we recreate this in a computer? Well, we create an artificial structure, called an artificial neural net, where we have nodes, or neurons. And we're gonna have some neurons for input value, so these are values that you know about a certain situation. For instance, if you're modeling something, you want to predict something, you're always gonna have some input, something to start your predictions off. That's called the input layer. Then you have the output, so that's a value that you want to predict, whether it's a price, whether it's is somebody going to leave the bank or stay in the bank, is this a fraudulent transaction, is this a real transaction, and so on. So that's gonna be output layer. And in between, we're going to have a hidden layer. As you can see, in your brain you have so many neurons, so some information is coming in through your eyes, ears, nose, basically your senses, and then it's not just going right away to the output where you have the result, it's going through all of these billions and billions and billions of neurons before it gets to the output. And this is the whole concept behind it that we're going to model the brain, so we need these hidden layers that are there before the output. So, the input layers neurons are connected to the hidden layer neurons, the hidden layer neurons are connected to output value. So this is pretty cool but what is this all about? Where is the deep learning here? Why is it called deep learning? There's nothing deep in here. Well, this is kind of like an option which one might call shallow learning, where there isn't much indeed going on. But, why is it called deep learning? Well, because then we take this to the next level. We separate it even further and we have not just one hidden layer, we have lots and lots and lots of hidden layers, and then we connect everything, just like in the human brain. Connect everything, interconnect everything, and that's how the input values are processed through all these hidden layers, just like in the human brain, then we have an output value. And now we're talking deep learning. So that's what deep learning is all about on a very abstract level. In the further tutorials we're going to dissect and dive deep into deep learning, and by the end of it you will know what deep learning is all about and you'll know how to apply it in your projects. Super excited about this, can't wait to get started, and I look forward to seeing you on the next tutorial. Until then, enjoy deep learning.