Optical Flow Coding with OpenCV - Part One

Jose Portilla
A free video tutorial from Jose Portilla
Head of Data Science, Pierian Data Inc.
4.6 instructor rating • 32 courses • 2,266,427 students

Learn more from the full course

Python for Computer Vision with OpenCV and Deep Learning

Learn the latest techniques in computer vision with Python , OpenCV , and Deep Learning!

14:03:33 of on-demand video • Updated March 2021

  • Understand basics of NumPy
  • Manipulate and open Images with NumPy
  • Use OpenCV to work with image files
  • Use Python and OpenCV to draw shapes on images and videos
  • Perform image manipulation with OpenCV, including smoothing, blurring, thresholding, and morphological operations.
  • Create Color Histograms with OpenCV
  • Open and Stream video with Python and OpenCV
  • Detect Objects, including corner, edge, and grid detection techniques with OpenCV and Python
  • Create Face Detection Software
  • Segment Images with the Watershed Algorithm
  • Track Objects in Video
  • Use Python and Deep Learning to build image classifiers
  • Work with Tensorflow, Keras, and Python to train on your own custom images.
English [Auto] Welcome back everyone. Now that we have an understanding of the intuition behind optical flow. Let's go ahead and use open Sivi to program our own use cases for optical flow using open civies. Built in functions for it. Let's head over to the Jupiter note book and get started. All right here we are in the notebook where we're going to do is we're going to start off with the Lucas Kanade method for optical flow and that's for those sparse points that we want to track. Essentially just picking a few points and then tracking their flow throughout the video from frame to frame will actually be tagging. So you'll see a little faint line being drawn as the objects move and we're going to choose points based off corners detection later on in future object tracking videos. In this section we'll actually detect the face first and then track that. But right now we'll keep it simple and it will use an object and method that we already know which is just corner detection. So I will start off by importing non-pilots and P and importing to. Then next I'm going to set some parameters for sheets of maasi corner detection. That's from the good features that track paper that we already discussed. And this should feel a little familiar to you. Essentially just a dictionary of corner tracking parameters and every season some default values here so we can detect Max corners. Let's say we're going to detect 10 corners in our image. Just the best quality ones and we can actually define a quality level. So we'll say that the quality level we want here is let's make sure is correct is 0.3. Obviously you can kind of play around with these well say minimum distance is 7 and we'll say block size is also 7. OK. So we have our parameters for the corner detection. Essentially what we're gonna do is on the very first frame of the video detect 10 corners and then track those. All right the next thing I want to do is set up in other dictionary called L.K. programs and these are going to be parameters for the Lucas Kanade optical flow function we call later on we're going to be right here a couple of default values. The first one is going to be the window size and the window size here. You're essentially going to have a tradeoff between smaller windows and larger windows if you have a smaller window you're going to be more sensitive to noise and you may miss larger motions. If the point you're trying to detect are moving really fast that means from one frame to the other the movement was actually really large. And you may not have caught it if you have a smaller window size and if you have a larger one the size you're going to fail to catch those larger motions. However they may not be as sensitive to smaller motions of the points. So you may just think that the point is standing still when it's actually kind of moving in really small harder to detect frames. So again it's kind of a tradeoff there between being too sensitive to noise versus being able to catch those smaller motions. So that's when those eyes will go ahead and use that default value of 200 by 200. Next we're going to text or provide a max level. And what this does is the Luca's Kanade method can actually use this algorithm with what's known as an image pyramid. And if you check out the Wikipedia page on Pyramid for image processing and scroll down here eventually you'll notice that there's an image here that essentially describes what a pyramid is when it comes to image analysis and the Lucas Kanade method what it does is you can actually have it use image permits for analysis. So if you leave max level at 0 it's just going to use the original image. However you can have higher max levels and will provide a max level of two. And what this does is it allows you to find optical flow at various resolutions of the image so you can see Level 1 is at half revolution resolution and level 2. Is that a quarter resolution. And so on. So we use a default value of max level equal to 2. And the next thing we're going to provide is the criteria and the criteria is going to work is will say CB2 term under score criteria underscore e p s and also a pipe operator essentially or will say CB2 term underscore criteria under underscore counts. And then right after that we're going to say 10 and zero point zero of three. So what this does is since you're providing two criteria to perform the Luca's cannot the optical flow we're providing the maximum number of iterations. So we have here we're providing a term criteria count and that's related to 10. And then we're also providing IPX or epsilon and that's going to be 0.03 now more iterations means a more exhaustive search for the points. So that's to do if the counts on how many iterations are you going to be looking for these points in the current frame versus the previous frame. And if you have a smaller Epsilon than that actually means you're going to finish earlier. Essentially what this does is you can play around with these values of criteria count criteria Epsilon. And whichever one essentially happens first you're going to finish earlier. And what it does is it's exchanging speed of your tracking versus the accuracy of your tracking. Again here we flip some good default values for you to play around with but you'll often have to adjust these depending on what your actual video sources. So keep that in mind. Essentially we're doing here is playing around with some values and exchanging speed accuracy. OK. So that is the parameter Dictionnaire we're going to use later on when we actually call the builtin Lucas Kanade function. Let's go ahead and run that and then continue. Now the next step is to actually grab an image from our camera. So a lot of this will actually already be familiar with will say CB2 video capture and you can actually just provide a video file. But in my case I'm going to do it live streaming from the camera so I will see zero. And then we're going to take RTT centrally in the kitting true or false. Is it actually able to capture the video and then we will call this the previous frame of the video so we're going to read the very first frame and treat it as a previous frame. Essentially saying here is the points Here's the previous frame and then the next family use that to see if we can find those points in the most current frame. So we're reading the very first frame and we'll label it as previous frame. And in order to do this we're actually going to also grab a greyscale version of that frame you'll see previous underscore Grey is equal to CB2 CVT color. And we're going to pass in the previous frame and convert it will see color from blue green red because it's open see the two gray and we're going to do is we're actually going to decide what are the points we actually want to track. So here are the points to track. We're going to do here is just grab using the corner tracking parameters here. Some good features attract. So we're going to choose the top 10 corners and track those previous points is equal to CB2 and then a good feature is the track pass and the previous greyscale. We're not going to have a mask for this. And then I can pass in the corner tracking parameters simply by saying to Asterix there and then passing corner tracking parameters. So essentially the laws are to provide a dictionary into a function call like that. OK so you have the points you want to track and then this math that we're creating here is going to be for displaying the actual points and drawing lines. So this mass has more to do with actually drawing lines onto the video then tracking points. So consider this mask just as a way of visualizing. And we'll be using it later on. So we'll see zeros like. And then passen the previous frame. All this does is it creates a number of zeros that has the same shape as the current frame you're looking at. So this creates a matching mass of the previous frame for drawing on later. OK so we have a point subtract we're reading in video data. We have the great skill and we have our mask that we can draw on and later recombine with the actual frame that we're looking at. Now it's time to actually have a for loop that does most of the work will say while true and we'll say. Rhett's common frame is equal to capture read just as we did above. But notice here now this is called frames. This is our current frame. The very first one is the previous frame. So we have a frame and then we actually do the exact same here of creating a grayscale. So we're going to copy this and paste it in a set of previous gray. I'm going to call it just frame gray. So it's the current greyscale frame. And then we're going to pasan frame. So we're taking now the current frame and technically at the very first instance those two will be the same. So a frame gray and then next is going to calculate the optical flow on this grace Gaile frame. And we do that by saying next point's status and e r r for air. And these are the three things that are returned by the built in function calque optical flow. And right now we're using p y r l k which stands for pyramid Luca's Kanade and we need to do here is pasan our previous image the previous greyscale image the current grayscale image. The points that we want to find from the previous frame in the next frame. And that is going to be these previous points appear to start off with later on we're going to reassign them. So there are the previous points and then the next parameter we just label as none. And you can check out the docs here if you want to see all the parameters. So there's previous points. And then what the next points are. But in this case we actually want to figure out what the next points are so we won't provide them instead. That's going to be spit up by the function itself. So we'll say none there and after that we just need all the parameters which have already the fine in L.K. programs. OK. So a lot going on at this function call. But really it's the same stuff that we were talking about in the slides in the previous lecture we pass in the previous frame we pass in the current frame both in grayscale the previous points and then we're not defined next points. We want those to be returned. And then these parameters that we can adjust by simply adjusting the dictionary which I encourage you to play around with OK. So we called the optical flow and now that we've calculated it where we're going to do is we're going to use the returned status array. So this data cerate essentially what it does is it outputs what's known as a status vector where each element of the vector is set to one. If the flow for the corresponding features has been found. Otherwise it's set to zero. So let me show what that looks like and how it can use it. I will create a variable called good underscore new and then say next points where status is equal to 1 and then say good underscore previous where the previous points has that is equal to 1. Essentially matching them up based on the index location or remember the status vector is set to 1. If the flow for the corresponding features has been found essentially this is connecting where the previous points were to the next set of points. Number one is say for I comma and then as a tuple you can say new and previous and then we'll say in a numerate and we're going to zip together the good new points along with the good previous points. And then we're going to calculate the X and Y positions so we can actually draw little markers there so I'll say X new Y new is equal to new and then we're going to say here is call DOT Revelle or ravelled and see how you pronounce it. And what this does is essentially a non-pilot method and you can check out the docs here and it's very similar to flattening out an array. Essentially it's the same as saying reshape. Negative one where you are keeping the current order. So here we can see a array passed in to them mentions after you call Ravell on it. Then it's kind of just flattened out to this single array. And we're going to be using that because we want to use these to draw. So we'll come back a period to Jupiter lab and then we'll do the same operation on the old points. We will say X previous and y previous and then set that equal to the previous and then call Revelle and that. And we're going to be doing is on our umask we'll be drawing some lines will say CB2 thought line and we're going to pasand the mask and go from x to y new and draw a line from that you point to the previous point it's essentially going to draw a little tracking line of the points as they go from frame to frame they don't say X previous to previous they'll choose the color. Let's go ahead and make it green and you can choose a thickness. We're going to give it three thickness. Now on the actual current frame I want to draw dots of where the current point we're tracking is. We'll say frame is equal to CB2 call circle on this. And then we will say frame and now we're only going to draw this on the new points because that's her current location. We'll say new points. Let's go ahead and give this radius of 8 and give it the color red and we'll will have it filled and we'll just say thickness is negative 1. So all this is doing is it's going through the new points and the good previous points. And this is kind of like complex Pyra because it's technically tracking 10 different points because we asked it to track 10 different corners. It can be more or it can be less than that. But keep that in mind. So has a bunch of points to track and then we're essentially using numerate to iterate through those good points and then flatten them out using some them by drawing a line connecting the previous points position to the current point position. So x x y new and then the previous points there and then we're drawing a circle on where the current frame of the point is. Now once you've done that outside of the for loop we're going to display this will say image is equal to to. Now we're going to say and we're going to add on the frame and the mask. So that adds on the frame with the circles and then the mask with the lines then we'll say CB2 and just show this. So we call that tracking and then we'll show that image that has both the frame with the points and the mask of the lights. Add a little bit of functionality here just to make sure that we can escape out of this see wait key. So wait 30 milliseconds in and 0 x F-F and then say if K is equal to 27 you could technically write that all in one line we break. And we've seen this before. Now the most important step here is that we need to update the previous point Senao be the current points. So we just drew our lines in our circles. Now we need to update the current frame to be the previous frame for the next iteration which means I will say previous Gray is equal to frame underscore gray and created as a copy that way don't overwrite anything. And then we'll say the previous points is equal to good new. And we're going to reshape that to negative one and then one comma two since that's going to be the way that it's going to be accepted into this calc optical flow pyramid. Lucas Kanade function the outside of all of this outside of the entire while loop. We'll say CB2 will destroy all the windows. Essentially we're done we hit the escape. And then we'll also release the capture and before we run this we want to fix a minor typo we made up here. We're actually checking for equality not assigning status. Sort of check for status being equal to one not reassigning status so a simple little typo there. Let's go ahead and save this and run it. And what I will recommend is before you actually do shift try to run the cell is look really into your USP camera. So hopefully some of the corners that the techs have to do if your face that we will track a few points on your face. We're going to look into my USP camera do shift enter here let me bring up what it's running. And there we go. So I tracked my eyes and my nostrils and a couple of other corners I'm going to begin moving. And then you should see and this looks really creepy. From my angle we should see the points being tracked as well as the lines moving and being drawn on that mask as we later combine it. And as you can sell eventually if you start moving really fast it may actually lose some of those points. I'm pretty well here but you can actually see I started turning my face and it kind of lost my left eyeball. I didn't really understand how to track that since it disappeared. So keep that in mind if a point disappears. That's optical flow won't be able to track any further. All right. So this was for a sparse set of points. Coming up next we're going to look at density which is simply swapping out that functional line similar code. But we're going to swap a single light out for a dense optical flow calculation. We'll cover that in the next lecture. I'll see you there.