Image to text

Hi and welcome back to this NLP course. So now that we have seen how CNNs are applied to images for computer vision, let's see how we can apply them to text for NLP. And so the first part will be to see how we can transform a text so that it's a valid input for a CNN. So the idea was to say, okay, so CNN for images, what does it do? It searches for local features in an image. So why don't we want to process text or sentences the same way by looking for local features throughout the whole sentence? But in order to do that, we have to have the same kind of inputs as images which were mattresses. So we want to get a matrix out of a sentence. We want to represent a sentence as a matrix, because for now it's just a list of characters or a list of words, but it doesn't really look like a matrix so far. So what we would like to have is something like this. And the most intuitive way is to say that if we have a matrix, each row will correspond to a word and each column will correspond to whatever we will find out later. But this representation means that each word will be a vector. So let's work around that. The easier and maybe more natural way to represent a word with a vector is also the less effective one. It is what we call the one hot encoding. So basically, if we say that our vocabulary has 100,000 words, each word will be a vector of that size and it will be mostly zeros. It will only have one one. And for example, for dog, if we say that dog is word number 238, well at the index to 38 of our vector, we will see a one and all the other numbers will be zero. So that way we have a unique representation for each of our 100,000 words, but there is absolutely no relation between each of them. There is no mathematical relation that appears. There is no meaning that is conveyed through it. It shows how little meaning and information it conveys. Well, it conveys information about which word we are dealing with, but nothing about the relations between those words. So what we would like to have is a smaller representation of each words. So instead of having a vector of size 100,000, we would have a vector of smaller size, let's say for instance, 64. So that's a common thing in science. We want to make the vector smaller. So that means that we add more constraints to our information. In a certain way, it has less liberty and that forces our system to create relations links or even meaning in the process if we do it the right way, of course. So this time dog, for instance, instead of being a vector of size, 100,000 with all zeros but one, it will be a vector of size 64 and it won't be binary anymore. It will be numbers between 0 and 1. So before going into mathematical details or to see how we do that, most of the time we will just have a look at some cool visual representations to see what it does and what it performs. So the first one is this quite well known effect of the word embedding. Now that they are embedded in a smaller vector space, we have some mathematical relations between the words as vectors and we can actually sum the vectors. And summing the vectors means summing meaning in our embedded space so we can actually get some very cool stuff. Like if we take the word king, for instance, and that we say king minus man plus woman, we actually get the word queen. So if the embedding has been done properly, we can actually subtract a part of the meaning of a word and add another one. Like for King, we can actually get rid of the male parts of King and add a female part to this result in order to get queen. And another example very simple is Paris minus France, plus Italia. We get Rome. So with Paris minus France, we get the capital ID and if we add Italia, then we get the capital of Italy, which is Rome. So that means that we are now able to apply mathematics to our words, to our vectors, and that is very, very useful in computer science, of course. So very good points. Second, visual ID, you may have to get a little bit closer to your screen, but this is a two dimensional representation of some words that have been embedded. And the important thing here is to realize that words that have similar meanings will be very close in the embedded space. So, for instance, if we have a look at some words here, we can see that kingdom will be close to states or that data will be close to information. Also, there is relations between the tenses of the verbs. Here we can see that there is become that is close to became close to being also close to was. Finally we can see that he is very close to she which is close to it also. So by embedding vectors into a smaller dimensional vector space, we added constraints as I said before, and it forced, of course, if we do the embedding properly, it forces the positions of vectors or the values of our vectors to convey meaning. And so to make words of similar meaning close in our embedding space. So that's cool. We get a vector representation of words. So that will allow us to have mattresses out of sentences. And it seems to be a very powerful tool because it conveys meaning. We can do mathematical stuff with it. So awesome. But how does it work exactly like the process, the embedding process, mathematically, What do we do exactly? The idea is that with an input vector as a one hot encoded vector as we saw before. So all zeros, but only one one for each word. We want to multiply it by a matrix, an embedding matrix and get our embedding vector right here. Actually, the fact that we multiply a one hot encoded vector by a matrix, it just means that if we have, for instance, the words I. So there is only x I which is activated. That means that we just take the row I in our matrix. So this matrix is just a list of all the embedded vectors, all the embedded words, but that's the details. So we want to multiply any one hot encoded vector by this matrix and have an embedded vector. But the question is how do we train this matrix? How do we train the weights? How do we learn the embedded vectors? We can't, of course, say that we want from this to get back to the original one because we will just have to apply the opposite operation just to see which row it corresponds to. And so get the in this right here. So the idea is that we want to multiply by another matrix, which we'll call the context matrix. You'll see why and gets again a vector of size vocab size. But we want to get something else that the original input vector, but we still want it to have a meaning to have a correlation with the original words. We want this output, this vector to have a semantic correlation with this one. So one idea to do that is to use what we call contexts. So this is the skip gram model where basically for an input words, we will pick several other words which we will call contexts and we want them to appear in this output vector. So let's say that word number one corresponds to context words number ten, 20 and 30, for instance, at the end of this operation. So after embedding the word number one and after retrieving information with the context matrix, we want he to have many zeros, but we want to have high number in position. Ten, 20 and 30, which are the indices that correspond to the context words that we defined before for the words number one. So in order to get those context words, we actually take a huge corpus of texts, and for each words we will take, for instance, the two previous words and the two next words as context. So in this sentence, in spite of everything, I still believe people are really good at hearts, which comes from The Diary of Anne Frank. If we take the word good, it's surrounded by the words are really at and heart. So this will produce four pairs of input and contexts which will be good are good, really good hearts and good hearts. So when we put the words good as inputs, so there will be a one only in the position that corresponds to this word goods. And after embedding it, after multiplying it by the context matrix, we want to have the indices corresponding to are really hot and hot activated. Of course, as good will appear many times in our corpus, we'll have a lot more context vectors as those ones. But the idea is that it's time we use a pair of input context words. We will change those matrices, the W and W prime. We will tune the coefficients in order to activate the context words out of the input words. So the global idea about that was from a single word in a one hot encoded version. Get a smaller vector and from this smaller vector get several words or activate several words that are often close to the initial words in our corpus. So that's pretty clever because we have our embedding phase right here, but it must convey meaning. It must be semantically efficient in order to retrieve a context information after this operation. And of course, this part was only used for the training in order to have something that has meaning. But when we embed a vector, we only use this part. This is the one that we are interested in in the end. Important and interesting thing to notice is that if two words have a close meaning. This means that they probably in all corpus will have similar contexts, and that also means that they will have a close embedding. So if we take this sentence again, we could argue that the word people could be replaced with humans, for instance. So I still believe humans are really good at heart. And that makes sense because humans and people, they convey a similar meaning. So in our corpus, we could expect humans and people to usually be surrounded by the same words. So that means that after we embed the words people or the words humans, we get two different embedded vectors, embedded words. But when we apply the context matrix, we want to get very close results. Actually pretty much the same results. So that means that the embedded versions of those two words have to be close, and that's where it comes from. What we saw earlier that two words of similar meaning will be close in our embedding space. So that was it for the global idea of word embedding, which will allow us to get efficient vectors from words. And so to get a matrix out of a sentence, if we just want to summarize this word embedding phase, using this Skipgram model is that it finds a dimension reduction in vector representation of words while adding a semantic relation between them. And I would add that this relation is mathematical so that we can use it with standard operations in artificial intelligence. So that was it for the very first step of CNN applied to NLP, which is how to get valid input for our CNN, of course. And now we are ready to feed them to CNN's and to see what it happens in our model and what we will have to change in order to have properly working CNN's for texts. So that's the topic of the next part and see you soon.