You can learn anything on Udemy. Start today with one of our 10,000+ online courses

MIT 6.00 Intro to Computer Science & Programming, Fall 2008

An introduction to computer science with MIT professor, John Guttag. No programming experience required!
30 reviews
WHAT'S INSIDE
  • Lifetime access to 38 lectures
  • 21+ hours of high quality content
  • A community of 7300+ students learning together!
TAUGHT BY
  • Professor Guttag received a bachelor's degree in English from Brown University in 1971, and a master's degree in applied mathematics from Brown in 1972. In 1975, he received a doctorate in computer science from the University of Toronto. He was a member of the faculty at the University of Southern California from 1975-1978, and joined the MIT faculty in 1979.

    From 1993 to 1998, Professor Guttag served as Associate Department Head for Computer Science of MIT's Electrical Engineering and Computer Science Department. From January of 1999 through August of 2004, Professor Guttag served as Head of that department. EECS, with approximately 1800 students and 125 faculty members, is the largest (and, of course, best) department at MIT.

    Professor Guttag also co-heads the MIT Computer Science and Artificial Intelligence Laboratory's Networks and Mobile Systems Group. This group studies issues related to computer networks, applications of networked and mobile systems, and advanced software-based medical instrumentation and decision systems. Professor Guttag has also done research, published, and lectured in the areas of software defined radios, software engineering, mechanical theorem proving, and hardware verification.

    Professor Guttag currently serves on the technical advisory board of Vanu, Inc., on the Board of Directors of Empirix, and on the Board of Trustees of the MGH Institute of Health Professions. He is also a Fellow of the ACM and a member of the American Academy of Arts and Sciences.
SHARE

MIT 6.00 Intro to Computer Science & Programming, Fall 2008

An introduction to computer science with MIT professor, John Guttag. No programming experience required!
30 reviews

This subject is aimed at students with little or no programming experience. It aims to provide students with an understanding of the role computation can play in solving problems. It also aims to help students, regardless of their major, to feel justifiably confident of their ability to write small programs that allow them to accomplish useful goals. The class will use Python as its base programming language.

While not required for this online course, you can acquire Professor Guttag's most recent textbook here.

Goals of the course; what is computation; introduction to data types, operators, and variables. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    • Over 38 lectures and 21.5 hours of content!

CURRICULUM

  • 1
    Syllabus
    Text
    Course Meeting TimesLectures: 2 sessions / week, 1 hour / sessionRecitations: 1 session / week, 1 hour / sessionPrerequisitesThis subject is aimed at students with
  • SECTION 1:
    Lectures and Assignments
  • 2
    MIT Lecture 1: Introduction and Goals; Data Types, Operators, and Variables
    53:30
    LECTURE TRANSCRIPT

    PROFESSOR: Good morning. Try it again. Good morning.

    STUDENTS: Good morning.

    PROFESSOR: Thank you. This is 6.00, also known as Introduction to Computer Science and Programming. My name is Eric Grimson, I have together Professor John Guttag over here, we're going to be lecturing the course this term. I want to give you a heads up; you're getting some serious firepower this term. John was department head for ten years, felt like a century, and in course six, I'm the current department head in course six. John's been lecturing for thirty years, roughly. All right, I'm the young guy, I've only been lecturing for twenty-five years. You can tell, I have less grey hair than he does. What I'm trying to say to you is, we take this course really seriously. We hope you do as well. But we think it's really important for the department to help everybody learn about computation, and that's what this course is about.

    What I want to do today is three things: I'm going to start-- actually, I shouldn't say start, I'm going to do a little bit of administrivia, the kinds of things you need to know about how we're going to run the course. I want to talk about the goal of the course, what it is you'll be able to do at the end of this course when you get through it, and then I want to begin talking about the concepts and tools of computational thinking, which is what we're primarily going to focus on here. We're going to try and help you learn how to think like a computer scientist, and we're going to begin talking about that towards the end of this lecture and of course throughout the rest of the lectures that carry on.

    Right, let's start with the goals. I'm going to give you goals in two levels. The strategic goals are the following: we want to help prepare freshmen and sophomores who are interested in majoring in course six to get an easy entry into the department, especially for those students who don't have a lot of prior programming experience. If you're in that category, don't panic, you're going to get it. We're going to help you ramp in and you'll certainly be able to start the course six curriculum and do just fine and still finish on target. We don't expect everybody to be a course six major, contrary to popular opinion, so for those are you not in that category, the second thing we want to do is we want to help students who don't plan to major in course six to feel justifiably confident in their ability to write and read small pieces of code.

    For all students, what we want to do is we want to give you an understanding of the role computation can and cannot play in tackling technical problems. So that you will come away with a sense of what you can do, what you can't do, and what kinds of things you should use to tackle complex problems.

    And finally, we want to position all students so that you can easily, if you like, compete for things like your office and summer jobs. Because you'll have an appropriate level of confidence and competence in your ability to do computational problem solving. Those are the strategic goals.

    Now, this course is primarily aimed at students who have little or no prior programming experience. As a consequence, we believe that no student here is under-qualified for this course: you're all MIT students, you're all qualified to be here. But we also hope that there aren't any students here who are over-qualified for this course. And what do I mean by that? If you've done a lot prior programming, this is probably not the best course for you, and if you're in that category, I would please encourage you to talk to John or I after class about what your goals are, what kind of experience you have, and how we might find you a course that better meets your goals.

    Second reason we don't want over-qualified students in the class, it sounds a little nasty, but the second reason is, an over-qualified student, somebody who's, I don't know, programmed for Google for the last five years, is going to have an easy time in this course, but we don't want such a student accidentally intimidating the rest of you. We don't want you to feel inadequate when you're simply inexperienced. And so, it really is a course aimed at students with little or no prior programming experience. And again, if you're not in that category, talk to John or I after class, and we'll help you figure out where you might want to go.

    OK. Those are the top-level goals of the course. Let's talk sort of at a more tactical level, about what do we want you to know in this course. What we want you to be able to do by the time you leave this course? So here are the skills that we would like you to acquire. Right, the first skill we want you to acquire, is we want you to be able to use the basic tools of computational thinking to write small scale programs. I'm going to keep coming back to that idea, but I'm going to call it computational thinking. And that's so you can write small pieces of code. And small is not derogatory here, by the way, it just says the size of things you're going to be able to do.

    Second skill we want you to have at the end of this course is the ability to use a vocabulary of computational tools in order to be able to understand programs written by others. So you're going to be able to write, you're going to be able to read.

    This latter skill, by the way, is incredibly valuable. Because you won't want to do everything from scratch yourself, you want to be able to look at what is being created by somebody else and understand what is inside of there, whether it works correctly and how you can build on it. This is one of the few places where plagiarism is an OK thing. It's not bad to, if you like, learn from the skills of others in order to create something you want to write. Although we'll come back to plagiarism as a bad thing later on.

    Third thing we want you to do, is to understand the fundamental both capabilities and limitations of computations, and the costs associated with them. And that latter statement sounds funny, you don't think of computations having limits, but they do. There're some things that cannot be computed. We want you to understand where those limits are. So you're going to be able to understand abilities and limits.

    And then, finally, the last tactical skill that you're going to get out of this course is you're going to have the ability to map scientific problems into a computational frame. So you're going to be able to take a description of a problem and map it into something computational.

    Now if you think about it, boy, it sounds like grammar school. We're going to teach you to read, we're going to teach you to write, we're going to teach you to understand what you can and cannot do, and most importantly, we're going to try and give you the start of an ability to take a description of a problem from some other domain, and figure out how to map it into that domain of computation so you can do the reading and writing that you want to do.

    OK, in a few minutes we're going to start talking then about what is computation, how are we going to start building those tools, but that's what you should take away, that's what you're going to gain out of this course by the time you're done.

    Now, let me take a sidebar for about five minutes to talk about course administration, the administrivia, things that we're going to do in the course, just so you know what the rules are. Right, so, class is two hours of lecture a week. You obviously know where and you know when, because you're here. Tuesdays and Thursdays at 11:00. One hour of recitation a week, on Fridays, and we'll come back in a second to how you're going to get set up for that. And nine hours a week of outside-the-class work. Those nine hours are going to be primarily working on problem sets, and all the problems sets are going to involve programming in Python, which is the language we're going to be using this term.

    Now, one of the things you're going to see is the first problem sets are pretty easy. Actually, that's probably wrong, John, right? They're very easy. And we're going to ramp up. By the time you get to the end of the term, you're going to be dealing with some fairly complex things, so one of the things you're going to see is, we're going to make heavy use of libraries, or code written by others. It'll allow you to tackle interesting problems I'll have you to write from scratch, but it does mean that this skill here is going to be really valuable. You need to be able to read that code and understand it, as well as write your own.

    OK. Two quizzes. During the term, the dates have already been scheduled. John, I forgot to look them up, I think it's October 2nd and November 4th, it'll be on the course website. My point is, go check the course website, which by the way is right there. If you have, if you know you have a conflict with one of those quiz dates now, please see John or I right away. We'll arrange something ahead of time. But if you-- The reason I'm saying that is, you know, you know that you're getting married that day for example, we will excuse you from the quiz to get married. We'll expect you come right back to do the quiz by the way, but the-- Boy, tough crowd. All right. If you have a conflict, please let us know.

    Second thing is, if you have an MIT documented special need for taking quizzes, please see John or I well in advance. At least two weeks before the quiz. Again, we'll arrange for this, but you need to give us enough warning so that we can deal with that.

    OK, the quizzes are open book. This course is not about memory. It's not how well you can memorize facts: in fact, I think both John and I are a little sensitive to memory tests, given our age, right John? This is not about how you memorize things, it's about how you think. So they're open note, open book. It's really going to test your ability to think.

    The grades for the course will be assigned roughly, and I use the word roughly because we reserve the right to move these numbers around a little bit, but basically in the following percentages: 55% of your grade comes from the problem sets, the other 45% come from the quizzes. And I should've said there's two quizzes and a final exam. I forgot, that final exam during final period. So the quiz percentages are 10%, 15%, and 20%. Which makes up the other 45%.

    OK. Other administrivia. Let me just look through my list here. First problem set, problem set zero, has already been posted. This is a really easy one. We intend it to be a really easy problem set. It's basically to get you to load up Python on your machine and make sure you understand how to interact with it.

    The first problem set will be posted shortly, it's also pretty boring-- somewhat like my lectures but not John's-- and that means, you know, we want you just to get going on things. Don't worry, we're going to make them more interesting as you go along.

    Nonetheless, I want to stress that none of these problems sets are intended to be lethal. We're not using them to weed you out, we're using them to help you learn. So if you run into a problem set that just, you don't get, all right? Seek help. Could be psychiatric help, could be a TA. I recommend the TA. My point being, please come and talk to somebody. The problems are set up so that, if you start down the right path, it should be pretty straight-forward to work it through. If you start down a plausible but incorrect path, you can sometimes find yourself stuck in the weeds somewhere, and we want to bring you back in. So part of the goal here is, this should not be a grueling, exhausting kind of task, it's really something that should be helping you learn the material. If you need help, ask John, myself, or the TAs. That's what we're here for.

    OK. We're going to run primarily a paperless subject, that's why the website is there. Please check it, that's where everything's going to be posted in terms of things you need to know. In particular, please go to it today, you will find a form there that you need to fill out to register for, or sign up for rather, a recitation.

    Recitations are on Friday. Right now, we have them scheduled at 9:00, 10:00, 11:00, 12:00, 1:00, and 2:00. We may drop one of the recitations, just depending on course size, all right? So we reserve the right, unfortunately, to have to move you around. My guess is that 9:00 is not going to be a tremendously popular time, but maybe you'll surprise me. Nonetheless, please go in and sign up. We will let you sign up for whichever recitation makes sense for you. Again, we reserve the right to move people around if we have to, just to balance load, but we want you to find something that fits your schedule rather than ours.

    OK. Other things. There is no required text. If you feel exposed without a text book, you really have to have a textbook, you'll find one recommended-- actually I'm going to reuse that word, John, at least suggest it, on the course website. I don't think either of us are thrilled with the text, it's the best we've probably found for Python, it's OK. If you need it, it's there. But we're going to basically not rely on any specific text.

    Right. Related to that: attendance here is obviously not mandatory. You ain't in high school anymore. I think both of us would love to see your smiling faces, or at least your faces, even if you're not smiling at us every day. Point I want to make about this, though, is that we are going to cover a lot of material that is not in the assigned readings, and we do have assigned readings associated with each one of these lectures. If you choose not to show up today-- or sorry, you did choose to show up today, if you choose not to show up in future days-- we'll understand, but please also understand that the TAs won't have a lot of patience with you if you're asking a question about something that was either covered in the readings, or covered in the lecture and is pretty straight forward. All right? We expect you to behave responsibly and we will as well. All right.

    I think the last thing I want to say is, we will not be handing out class notes. Now this sounds like a draconian measure; let me tell you why. Every study I know of, and I suspect every one John knows, about learning, stresses that students learn best when they take notes. Ironically, even if they never look at them. OK. The process of writing is exercising both halves of your brain, and it's actually helping you learn, and so taking notes is really valuable thing. Therefore we're not going to distribute notes. What we will distribute for most lectures is a handout that's mostly code examples that we're going to do. I don't happen to have one today because we're not going to do a lot of code. We will in future. Those notes are going to make no sense, I'm guessing, outside of the lecture, all right? So it's not just, you can swing by 11:04 and grab a copy and go off and catch some more sleep. What we recommend is you use those notes to take your own annotations to help you understand what's going on, but we're not going to provide class notes. We want you to take your own notes to help you, if you like, spur your own learning process.

    All right. And then finally, I want to stress that John, myself, all of the staff, our job is to help you learn. That's what we're here for. It's what we get excited about. If you're stuck, if you're struggling, if you're not certain about something, please ask. We're not mind readers, we can't tell when you're struggling, other than sort of seeing the expression on your face, we need your help in identifying that. But all of the TAs, many of whom are sitting down in the front row over here, are here to help, so come and ask. At the same time, remember that they're students too. And if you come and ask a question that you could have easily answered by doing the reading, coming to lecture, or using Google, they're going to have less patience. But helping you understand things that really are a conceptual difficulty is what they're here for and what we're here for, so please come and talk to us.

    OK. That takes care of the administrivia preamble. John, things we add?

    PROFESSOR GUTTAG: Two more quick things. This semester, your class is being videotaped for OpenCourseware. If any of you don't want your image recorded and posted on the web, you're supposed to sit in the back three rows.

    PROFESSOR GRIMSON: Ah, thank you. I forgot.

    PROFESSOR GUTTAG: --Because the camera may pan. I think you're all very good-looking and give MIT a good image, so please, feel free to be filmed.

    PROFESSOR GRIMSON: I'll turn around, so if you want to, you know, move to the back, I won't see who moves. Right. Great. Thank you, John.

    PROFESSOR GUTTAG: So that, the other thing I want to mention is, recitations are also very important. We will be covering material in recitations that're not in the lectures, not in the reading, and we do expect you to attend recitations.

    PROFESSOR GRIMSON: Great. Thanks, John. Any questions about the administrivia? I know it's boring, but we need to do it so you know what the ground rules are.

    Good. OK. Let's talk about computation. As I said, our strategic goal, our tactical goals, are to help you think like a computer scientist. Another way of saying it is, we want to give you the skill so that you can make the computer do what you want it to do. And we hope that at the end of the class, every time you're confronted with some technical problem, one of your first instincts is going to be, "How do I write the piece of code that's going to help me solve that?"

    So we want to help you think like a computer scientist. All right. And that, is an interesting statement. What does it mean, to think like a computer scientist? Well, let's see. The primary knowledge you're going to take away from this course is this notion of computational problem solving, this ability to think in computational modes of thought. And unlike in a lot of introductory courses, as a consequence, having the ability to memorize is not going to help you. It's really learning those notions of the tools that you want to use. What in the world does it mean to say computational mode of thought? It sounds like a hifalutin phrase you use when you're trying to persuade a VC to fund you. Right. So to answer this, we really have to ask a different question, a related question; so, what's computation?

    It's like a strange statement, right? What is computation? And part of the reason for putting it up is that I want to, as much as possible, answer that question by separating out the mechanism, which is the computer, from computational thinking. Right. The artifact should not be what's driving this. It should be the notion of, "What does it mean to do computation?"

    Now, to answer that, I'm going to back up one more level. And I'm going to pose what sounds like a philosophy question, which is, "What is knowledge?" And you'll see in about two minutes why I'm going to do this. But I'm going to suggest that I can divide knowledge into at least two categories. OK, and what is knowledge? And the two categories I'm going to divide them into are declarative and imperative knowledge.

    Right. What in the world is declarative knowledge? Think of it as statements of fact. It's assertions of truth. Boy, in this political season, that's a really dangerous phrase to use, right? But it's a statement of fact. I'll stay away from the political comments. Let me give you an example of this. Right. Here's a declarative statement. The square root of x is that y such that y squared equals x, y's positive. You all know that. But what I want you to see here, is that's a statement of fact. It's a definition. It's an axiom. It doesn't help you find square roots. If I say x is 2, I want to know, what's the square root of 2, well if you're enough of a geek, you'll say 1.41529 or whatever the heck it is, but in general, this doesn't help you find the square root. The closest it does is it would let you test. You know, if you're wandering through Harvard Square and you see an out-of-work Harvard grad, they're handing out examples of square roots, they'll give you an example and you can test it to see, is the square root of 2, 1.41529 or whatever. I don't even get laughs at Harvard jokes, John, I'm going to stop in a second here, all right? All right, so what am I trying to say here? It doesn't -- yeah, exactly. We're staying away from that, really quickly, especially with the cameras rolling.

    All right. What am I trying to say? It tells you how you might test something but it doesn't tell you how to.

    And that's what imperative knowledge is. Imperative knowledge is a description of how to deduce something. So let me give you an example of a piece of imperative knowledge. All right, this is actually a very old piece of imperative knowledge for computing square roots, it's attributed to Heron of Alexandria, although I believe that the Babylonians are suspected of knowing it beforehand. But here is a piece of imperative knowledge. All right? I'm going to start with a guess, I'm going to call it g. And then I'm going to say, if g squared is close to x, stop. And return g. It's a good enough answer. Otherwise, I'm going to get a new guess by taking g, x over g, adding them, and dividing by two. Then you take the average of g and x over g. Don't worry about how came about, Heron found this out. But that gives me a new guess, and I'm going to repeat.

    That's a recipe. That's a description of a set of steps. Notice what it has, it has a bunch of nice things that we want to use, right? It's a sequence of specific instructions that I do in order. Along the way I have some tests, and depending on the value of that test, I may change where I am in that sequence of instructions. And it has an end test, something that tells me when I'm done and what the answer is. This tells you how to find square roots. it's how-to knowledge. It's imperative knowledge.

    All right. That's what computation basically is about. We want to have ways of capturing this process. OK, and that leads now to an interesting question, which would be, "How do I build a mechanical process to capture that set of computations?" So I'm going to suggest that there's an easy way to do it-- I realized I did the boards in the wrong order here-- one of the ways I could do it is, you could imagine building a little circuit to do this. If I had a couple of elements of stored values in it, I had some wires to move things around, I had a little thing to do addition, little thing to do division, and a something to do the testing, I could build a little circuit that would actually do this computation.

    OK. That, strange as it sounds, is actually an example of the earliest computers, because the earliest computers were what we call fixed-program computers, meaning that they had a piece of circuitry designed to do a specific computation. And that's what they would do: they would do that specific computation. You've seen these a lot, right? A good example of this: calculator. It's basically an example of a fixed-program computer. It does arithmetic. If you want play video games on it, good luck. If you want to do word processing on it, good luck. It's designed to do a specific thing. It's a fixed-program computer.

    In fact, a lot of the other really interesting early ones similarly have this flavor, to give an example: I never know how to pronounce this, Atanasoff, 1941. One of the earliest computational things was a thing designed by a guy named Atanasoff, and it basically solved linear equations. Handy thing to do if you're doing 1801, all right, or 1806, or whatever you want to do those things in. All it could do, though, was solve those equations.

    One of my favorite examples of an early computer was done by Alan Turing, one of the great computer scientists of all time, called the bombe, which was designed to break codes. It was actually used during WWII to break German Enigma codes. And what it was designed to do, was to solve that specific problem.

    The point I'm trying to make is, fixed-program computers is where we started, but it doesn't really get us to where we'd like to be. We want to capture this idea of problem solving. So let's see how we'd get there. So even within this framework of, given a description of a computation as a set of steps, in the idea that I could build a circuit to do it, let me suggest for you what would be a wonderful circuit to build.

    Suppose you could build a circuit with the following property: the input to this circuit would be any other circuit diagram. Give it a circuit diagram for some computation, you give it to the circuit, and that circuit would wonderfully reconfigure itself to act like the circuits diagram. Which would mean, it could act like a calculator. Or, it could act like Turing's bombe. Or, it could act like a square root machine.

    So what would that circuit look like? You can imagine these tiny little robots wandering around, right? Pulling wires and pulling out components and stacking them together. How would you build a circuit that could take a circuit diagram in and make a machine act like that circuit? Sounds like a neat challenge.

    Let me change the game slightly. Suppose instead, I want a machine that can take a recipe, the description of a sequence of steps, take that as its input, and then that machine will now act like what is described in that recipe. Reconfigure itself, emulate it, however you want to use the words, it's going to change how it does the computation.

    That would be cool. And that exists. It's called an interpreter. It is the basic heart of every computer. What it is doing, is saying, change the game. This is now an example of a stored-program computer. What that means, in a stored-program computer, is that I can provide to the computer a sequence of instructions describing the process I want it to execute. And inside of the machine, and things we'll talk about, there is a process that will allow that sequence to be executed as described in that recipe, so it can behave like any thing that I can describe in one of those recipes.

    All right. That actually seems like a really nice thing to have, and so let me show you what that would basically look like. Inside of a stored-program computer, we would have the following: we have a memory, it's connected to two things; control unit, in what's called an ALU, an arithmetic logic unit, and this can take in input, and spit out output, and inside this stored-program computer, excuse me, you have the following: you have a sequence of instructions. And these all get stored in there. Notice the difference. The recipe, the sequence of instructions, is actually getting read in, and it's treated just like data. It's inside the memory of the machine, which means we have access to it, we can change it, we can use it to build new pieces of code, as well as we can interpret it. One other piece that goes into this computer-- I never remember where to put the PC, John, control? ALU? Separate? I'll put it separate-- you have a thing called a program counter.

    And here's the basis of the computation. That program counter points to some location in memory, typically to the first instruction in the sequence. And those instructions, by the way, are very simple: they're things like, take the value out of two places in memory, and run them through the multiplier in here, a little piece of circuitry, and stick them back into someplace in memory. Or take this value out of memory, run it through some other simple operation, stick it back in memory. Having executed this instruction, that counter goes up by one and we move to the next one. We execute that instruction, we move to the next one. Oh yeah, it looks a whole lot like that.

    Some of those instructions will involve tests: they'll say, is something true? And if the test is true, it will change the value of this program counter to point to some other place in the memory, some other point in that sequence of instructions, and you'll keep processing. Eventually you'll hopefully stop, and a value gets spit out, and you're done.

    That's the heart of a computer. Now that's a slight misstatement. The process to control it is intriguing and interesting, but the heart of the computer is simply this notion that we build our descriptions, our recipes, on a sequence of primitive instructions. And then we have a flow of control. And that flow of control is what I just described. It's moving through a sequence of instructions, occasionally changing where we are as we move around.

    OK. The thing I want you to take away from this, then, is to think of this as, this is, if you like, a recipe. And that's really what a program is. It's a sequence of instructions. Now, one of things I left hanging is, I said, OK, you build it out of primitives. So one of the questions is, well, what are the right primitives to use? And one of the things that was useful here is, that we actually know that the set of primitives that you want to use is very straight-forward.

    OK, but before I do that, let me drive home this idea of why this is a recipe. Assuming I have a set of primitive instructions that I can describe everything on, I want to know what can I build. Well, I'm going to do the same analogy to a real recipe. So, real recipe. I don't know. Separate six eggs. Do something. Beat until the-- sorry, beat the whites until they're stiff. Do something until an end test is true. Take the yolks and mix them in with the sugar and water-- No. Sugar and flour I guess is probably what I want, sugar and water is not going to do anything interesting for me here-- mix them into something else. Do a sequence of things.

    A traditional recipe actually is based on a small set of primitives, and a good chef with, or good cook, I should say, with that set of primitives, can create an unbounded number of great dishes. Same thing holds true in programming. Right. Given a fixed set of primitives, all right, a good programmer can program anything. And by that, I mean anything that can be described in one of these process, you can capture in that set of primitives.

    All right, the question is, as I started to say, is, "What are the right primitives?" So there's a little bit of, a little piece of history here, if you like. In 1936, that same guy, Alan Turing, showed that with six simple primitives, anything that could be described in a mechanical process, it's actually algorithmically, could be programmed just using those six primitives.

    Think about that for a second. That's an incredible statement. It says, with six primitives, I can rule the world. With six primitives, I can program anything. A couple of really interesting consequences of that, by the way, one of them is, it says, anything you can do in one programming language, you can do in another programming language. And there is no programming language that is better-- well actually, that's not quite true, there are some better at doing certain kinds of things-- but there's nothing that you can do in C that you can't do in Fortran. It's called Turing compatibility. Anything you can do with one, you can do with another, it's based on that fundamental result.

    OK. Now, fortunately we're not going to start with Turing's six primitives, this would be really painful programming, because they're down at the level of, "take this value and write it onto this tape." First of all, we don't have tapes anymore in computers, and even if we did, you don't want to be programming at that level. What we're going to see with programming language is that we're going to use higher-level abstracts. A broader set of primitives, but nonetheless the same fundamental thing holds. With those six primitives, you can do it.

    OK. So where are we here? What we're saying is, in order to do computation, we want to describe recipes, we want to describe this sequence of steps built on some primitives, and we want to describe the flow of control that goes through those sequence of steps as we carry on.

    So the last thing we need before we can start talking about real programming is, we need to describe those recipes. All right, And to describe the recipes, we're going to want a language. We need to know not only what are the primitives, but how do we make things meaningful in that language. Language. There we go. All right. Now, it turns out there are-- I don't know, John, hundreds? Thousands? Of programming languages? At least hundreds-- of programming languages around.

    PROFESSOR JOHN GUTTAG: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: True. Thank you. You know, they all have, you know, their pluses and minuses. I have to admit, in my career here, I think I've taught in at least three languages, I suspect you've taught more, five or six, John? Both of us have probably programmed in more than those number of languages, at least programmed that many, since we taught in those languages.

    One of the things you want to realize is, there is no best language. At least I would argue that, I think John would agree. We might both agree we have our own nominees for worst language, there are some of those. There is no best language. All right? They all are describing different things. Having said that, some of them are better suited for some things than others.

    Anybody here heard of MATLAB Maybe programmed in MATLAB? It's great for doing things with vectors and matrices and things that are easily captured in that framework. But there's some things that are a real pain to do in MATLAB. So MATLAB's great for that kind of thing.

    C is a great language for programming things that control data networks, for example.

    I happen to be, and John teases me about this regularly, I'm an old-time Lisp programmer, and that's how I was trained. And I happen to like Lisp and Scheme, it's a great language when you're trying to deal with problems where you have arbitrarily structured data sets. It's particularly good at that.

    So the point I want to make here is that there's no particularly best language. What we're going to do is simply use a language that helps us understand. So in this course, the language we're going to use is Python. Which is a pretty new language, it's growing in popularity, it has a lot of the elements of some other languages because it's more recent, it inherits things from it's pregenitors, if you like.

    But one of the things I want to stress is, this course is not about Python. Strange statement. You do need to know how to use it, but it's not about the details of, where do the semi-colons go in Python. All right? It's about using it to think.

    And what you should take away from this course is having learned how to design recipes, how to structure recipes, how to do things in modes in Python. Those same tools easily transfer to any other language. You can pick up another language in a week, couple of weeks at most, once you know how to do Python.

    OK. In order to talk about Python and languages, I want to do one last thing to set the stage for what we're going to do here, and that's to talk about the different dimensions of a language. And there're three I want to deal with.

    The first one is, whether this is a high-level or low-level language. That basically says, how close are you the guts of the machine? A low-level language, we used to call this assembly programming, you're down at the level of, your primitives are literally moving pieces of data from one location of memory to another, through a very simple operation. A high-level language, the designer has created a much richer set of primitive things. In a high-level language, square root might simply be a primitive that you can use, rather than you having to go over and code it. And there're trade-offs between both.

    Second dimension is, whether this is a general versus a targeted language. And by that I mean, do the set of primitives support a broad range of applications, or is it really aimed at a very specific set of applications? I'd argue that MATLAB is basically a targeted language, it's targeted at matrices and vectors and things like that.

    And the third one I want to point out is, whether this is an interpreted versus a compiled language. What that basically says is the following: in an interpreted language, you take what's called the source code, the thing you write, it may go through a simple checker but it basically goes to the interpreter, that thing inside the machine that's going to control the flow of going through each one of the instructions, and give you an output. So the interpreter is simply operating directly on your code at run time.

    In a compiled language, you have an intermediate step, in which you take the source code, it runs through what's called a checker or a compiler or both, and it creates what's called object code. And that does two things: one, it helps catch bugs in your code, and secondly it often converts it into a more efficient sequence of instructions before you actually go off and run it. All right?

    And there's trade-offs between both. I mean, an interpreted language is often easier to debug, because you can still see your raw code there, but it's not always as fast. A compiled language is usually much faster in terms of its execution. And it's one of the things you may want to trade off.

    Right. In the case of Python, it's a high-level language. I would argue, I think John would agree with me, it's basically a general-purpose language. It happens to be better suited for manipulating strings than numbers, for example, but it's really a general-purpose language. And it's primarily-- I shouldn't say primarily, it is an interpreted language. OK?

    As a consequence, it's not as good as helping debug, but it does let you-- sorry, that's the wrong way of saying-- it's not as good at catching some things before you run them, it is easier at some times in debugging as you go along on the fly.

    OK. So what does Python look like? In order to talk about Python-- actually, I'm going to do it this way-- we need to talk about how to write things in Python. Again, you have to let me back up slightly and set the stage.

    Our goal is to build recipes. You're all going to be great chefs by the time you're done here. All right? Our goal is to take problems and break them down into these computational steps, these sequence of instructions that'll allow us to capture that process.

    To do that, we need to describe: not only, what are the primitives, but how do we capture things legally in that language, and interact with the computer? And so for that, we need a language. We're about to start talking about the elements of the language, but to do that, we also need to separate out one last piece of distinction. Just like with a natural language, we're going to separate out syntax versus semantics.

    So what's syntax? Syntax basically says, what are the legal expressions in this language? Boy, my handwriting is atrocious, isn't it? There's a English sequence of words. It's not since syntactically correct, right? It's not a sentence. There's no verb in there anywhere, it's just a sequence of nouns. Same thing in our languages. We have to describe how do you put together legally formed expressions. OK? And as we add constructs to the language, we're going to talk about.

    Second thing we want to talk about very briefly as we go along is the semantics of the language. And here we're going to break out two pieces; static semantics and full semantics. Static semantics basically says which programs are meaningful. Which expressions make sense. Here's an English sentence. It's syntactically correct. Right? Noun phrase, verb, noun phrase. I'm not certain it's meaningful, unless you are in the habit of giving your furniture personal names.

    What's the point? Again, you can have things that are syntactically legal but not semantically meaningful, and static semantics is going to be a way of helping us decide what expressions, what pieces of code, actually have real meaning to it. All right?

    The last piece of it is, in addition to having static semantics, we have sort of full semantics. Which is, what does the program mean? Or, said a different way, what's going to happen when I run it? That's the meaning of the expression. That's what you want. All right? You want to know, what's the meaning of this piece of code? When I run it, what's going to happen? That's what I want to build.

    The reason for pulling this out is, what you're going to see is, that in most languages, and certainly in Python-- we got lots of help here-- all right, Python comes built-in with something that will check your static, sorry, your syntax for you. And in fact, as a sidebar, if you turn in a problem set that is not syntactically correct, there's a simple button that you push that will check your syntax. If you've turned in a program that's not syntactically correct, the TAs give you a zero. Because it said you didn't even take the time to make sure the syntax is correct. The system will help you find it. In Python, it'll find it, I think one bug at a time, right John? It finds one syntax error at a time, so you have to be a little patient to do it, but you can check that the syntax is right.

    You're going to see that we get some help here on the static semantics, and I'm going to do an example in a second, meaning that the system, some languages are better than others on it, but it will try and help you catch some things that are not semantically correct statically. In the case of Python, it does that I think all at run time. I'm looking to you again, John, I think there's no pre-time checks. Its-- sorry?

    PROFESSOR JOHN GUTTAG: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: There is some. OK. Most of them, I think though, are primarily caught at run time, and that's a little bit of a pain because you don't see it until you go and run the code, and there are some, actually we're going to see an example I think in a second where you find it, but you do get some help there.

    The problem is, things that you catch here are actually the least worrisome bugs. They're easy to spot, you can't run the program with them there, so you're not going to get weird answers. Not everything is going to get caught in static semantics checking. Some things are going to slide through, and that's actually a bother. It's a problem. Because it says, your program will still give you a value, but it may not be what you intended, and you can't always tell, and that may propagate it's way down through a whole bunch of other computations before it causes some catastrophic failure. So actually, the problem with static semantics is you'd like it to catch everything, you don't always get it. Sadly we don't get much help here. Which is where we'd like it. But that's part of your job.

    OK. What happens if you actually have something that's both syntactically correct, and appears to have correct static semantics, and you run it? It could run and give you the right answer, it could crash, it could loop forever, it could run and apparently give you the right answer. And you're not always going to be able to tell. Well, you'll know when it crashes, that doesn't help you very much, but you can't always tell whether something's stuck in an infinite loop or whether it's simply taking a long time to compute. You'd love to have a system that spots that for you, but it's not possible.

    And so to deal with this last one, you need to develop style. All right? Meaning, we're going to try to help you with how to develop good programming style, but you need to write in a way in which it is going to be easy for you to spot the places that cause those semantic bugs to occur.

    All right. If that sounds like a really long preamble, it is. Let's start with Python. But again, my goal here is to let you see what computation's about, why we need to do it, I'm going to remind you one last time, our goal is to be able to have a set of primitives that we combine into complex expressions, which we can then abstract to treat as primitives, and we want to use that sequence of instructions in this flow of control computing, in order to deduce new information. That imperative knowledge that we talked about right there. So I'm going to start today, we have about five or ten minutes left, I think, in order-- sorry, five minutes left-- in order to do this with some beginnings of Python, and we're going to pick this up obviously, next time, so; simple parts of Python.

    In order to create any kinds of expressions, we're going to need values. Primitive data elements. And in Python, we have two to start with; we have numbers, and we have strings. Numbers is what you'd expect. There's a number. There's another number. All right? Strings are captured in Python with an open quote and some sequence of characters followed by a closed quote.

    Associated with every data type in Python is a type, which identifies the kind of thing it is. Some of these are obvious. Strings are just a type on their own. But for numbers, for example, we can have a variety of types. So this is something that we would call an integer, or an INT. And this is something we would call a floating point, or a float. Or if you want to think of it as a real number. And there's some others that we can see.

    We're going to build up this taxonomy if you like, but the reason it's relevant is, associated with each one of those types is a set of operators that expect certain types of input in order to do their job. And given those types of input, will get back output.

    All right. In order to deal with this, let me show you an example, and I hope that comes up, great. What I have here is a Python shell, and I'm going to just show you some simple examples of how we start building expressions. And this'll lead into what you're going to see next time as well as what you're going to do tomorrow. So. Starting with the shell, I can type in expressions.

    Actually, let me back up and do this in video. I can type in a number, I get back a number, I can type in a string, I get back the string. Strings, by the way, can have spaces in them, they can have other characters, it's simply a sequence of things, and notice, by the way, that the string five-- sorry, the string's digit five digit two is different than the number 52. The quotes are around them to make that distinction. We're going to see why in a second.

    What I'm doing, by the way, here is I'm simply typing in expressions to that interpreter. It's using its set of rules to deduce the value and print them back out. Things I might like to do in here is, I might like to do combinations of things with these. So we have associated with simple things, a set of operations. So for numbers, we have the things you'd expect, the arithmetics. And let me show you some examples of that.

    And actually, I'm going to do one other distinction here. What I typed in, things like-- well, let me start this way-- there's an expression. And in Python the expression is, operand, operator, operand, when we're doing simple expressions like this, and if I give it to the interpreter, it gives me back exactly what you'd expect, which is that value. OK?

    The distinction I'm going to make is, that's an expression. The interpreter is going to get a value for it. When we start building up code, we're going to use commands. Or statements. Which are actually things that take in a value and ask the computer to do something with it. So I can similarly do this, which is going to look strange because it's going to give me the same value back out, but it actually did a slightly different thing.

    And notice, by the way, when I typed it how print showed up in a different color? That's the Python saying, that is a command, that is a specific command to get the value of the expression and print it back out. When we start writing code, you're going to see that difference, but for now, don't worry about it, I just want to plant that idea.

    OK. Once we've got that, we can certainly, though, do things like this. Notice the quotes around it. And it treats it as a string, it's simply getting me back the value of that string, 52 times 7, rather than the value of it. Now, once we've got that, we can start doing things. And I'm going to use print here-- if I could type, in order to just to get into that, I can't type, here we go-- in order to get into the habit. I can print out a string. I can print out-- Ah!-- Here's a first example of something that caught one of my things. This is a static semantic error.

    So what went on here? I gave it an expression that had an operand in there. It expected arithmetic types. But I gave two strings. And so it's complaining at me, saying, you can't do this. I don't know how to take two strings and multiply them together. Unfortunately-- now John you may disagree with me on this one-- unfortunately in Python you can, however, do things like this. What do you figure that's going to do? Look legal? The string three times the number three? Well it happens to give me three threes in a row.

    I hate this. I'm sorry, John, I hate this. Because this is overloading that multiplication operator with two different tasks. It's saying, if you give me two numbers, I'll do the right thing. If you give me a number and a string, I'm going to concatenate them together, it's really different operations, but nonetheless, it's what it's going to do.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: There you go. You know, there will be a rebuttal phase a little later on, just like with the political debates, and he likes it as a feature, I don't like it, you can tell he's not a Lisp programmer and I am.

    All right. I want to do just a couple more quick examples. Here's another one. Ah-ha! Give you an example of a syntax error. Because 52A doesn't make sense. And you might say, wait a minute, isn't that a string, and the answer's no, I didn't say it's a string by putting quotes around it. And notice how the machine responds differently to it. In this case it says, this is a syntax error, and it's actually highlighting where it came from so I can go back and fix it.

    All right. Let's do a couple of other simple examples. All right? I can do multiplication. I've already seen that. I can do addition. Three plus five. I can take something to a power, double star, just take three to the fifth power. I can do division, right? Whoa. Right? Three divided by five is zero? Maybe in Bush econom-- no, I'm not going to do any political comments today, I will not say that, all right?

    What happened? Well, this is one of the places where you have to be careful. It's doing integer division. So, three divided by five is zero, with a remainder of three. So this is the correct answer. If I wanted to get full, real division, I should make one of them a float. And yes, you can look at that and say, well is that right? Well, up to some level of accuracy, yeah, that's .6 is what I'd like to get out.

    All right. I can do other things. In a particular, I have similar operations on strings. OK, I can certainly print out strings, but I can actually add strings together, and just as you saw, I can multiply strings, you can kind of guess what this is going to do. It is going to merge them together into one thing. I want-- I know I'm running you slightly over, I want to do one last example, it's, I also want to be able to do, have variables to store things. And to do that, in this it says, if I have a value, I want to keep it around, to do that, I can do things like this.

    What does that statement do? It says, create a name for a variable-- which I just did there, in fact, let me type it in-- mystring, with an equal sign, which is saying, assign or bind to that name the value of the following expression. As a consequence, I can now refer to that just by its name. If I get the value of mystring, there it is, or if I say, take mystring and add to it the string, mylastname, and print it back out.

    So this is the first start of this. What have we done? We've got values, numbers and strings. We have operations to associate with them. I just threw a couple up here. You're going to get a chance to explore them, and you'll see not only are there the standard numerics for strings, there are things like length or plus or other things you can do with them. And once I have values, I want to get a hold of them so I can give them names. And that's what I just did when I bound that. I said, use the name mystring to be bound to or have the value of Eric, so I can refer to it anywhere else that I want to use it.

    And I apologize for taking you over, we'll come back to this next time, please go to the website to sign up for recitation for tomorrow.
  • 3
    Problem Set #0: Printing Your Name
    3 pages
    A very simple program: entering and printing your name
  • 4
    MIT Lecture 2: Branching, Conditionals, and Iteration
    50:49
    Operators and operands; statements; branching, conditionals, and iteration. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR JIM ERICSON: OK, to work. A word of warning: fasten your seat belts. Or, another way of saying it is, I'm going to open up the fire hose a little bit today.

    Last lecture, you might have thought this was a SHASS class, it's not like a philosophy class, and it was important to set the stage for what we're going to talk about, but we talked about very high level things. The notion of recipes, the notion of computation, why you want to do this, what you're going to learn.

    Today we're going to dive into the nitty-gritty, the nuts and bolts of the basics of computation, and in particular, what I'm going to do today is, I'm going to talk about operators and operands, which we did a little bit real last time, in particular how to create expressions, I'm going to talk about statements as the key building blocks for writing code, and I'm going to introduce simple sets of programs, in particular I'm going to talk about branching, conditionals, and iteration. So, a lot to do. OK? So, let me jump straight to it.

    At the end of last lecture, we started introducing some of the pieces you want to do. And I want to remind you of our goal. We're trying to describe processes. We want to have things that deduce new kinds of information. So we want to write programs to do that.

    If we're going to write programs, we need at least two things: we need some representation for fundamental data. And we saw last time two examples of that. And the second thing we're going to need, is we're going to need a way to give instructions to the computer to manipulate that data. We need to give it a description of the recipe.

    In terms of primitive data, what we saw were two kinds: Right? Numbers and strings. A little later on in the lecture we're going to introduce a third kind of value, but what we're going to see throughout the term is, no matter how complex a data structure we create, and we're going to create a variety of data structures, fundamentally all of them have their basis, their atomic level if you like, are going to be some combinations of numbers, of strings, and the third type, which are booleans, which I'm going to introduce a little later on in this lecture.

    And that kind of makes sense right? Numbers are there to do numeric things, strings are our fundamental way of representing textual information. And so we're going to see how to combine those things as we go along.

    Second thing we saw was, we saw that associated with every primitive value was a type. And these are kind of obvious, right? Strings are strings. For numbers, we had some variations; we had integers, we had floats. We'll introduce a few more as we go along. But those types are important, because they tell us something about what we want to do when we want to put them together.

    OK, but nonetheless, I want to stress we have both a value, yeah, and a type. All right. Once we have them, we want to start making combinations out of them. We want to put pieces together. And for that, we combine things in expressions. And what we saw as expressions are formed of operands and operators. And the simple things we did were the sort of things you'd expect from numerical things.

    Now I want to stress one other nuance here. Which is, and we're going to do some examples of this, initially we just typed in expressions into the interpreter; that is, directly into Python. And as I suggested last time, the interpreter is actually a program inside of the machine that is basically following the rules we're describing here to deduce the value and print it up. And if we type directly into the interpreter, it essentially does an eval and a print. It evaluates, and it prints. Most of the time, we're going to be doing expressions inside of some piece of code, inside of a script, which is the Python word for program. In there, I want to make this distinction, this nuance: the evaluator is still going to be taking those expressions and using its rules to get a value, but it's not going to print them back out.

    Why? Because typically, you're doing that to use it somewhere else in the program. It's going to be stored away in a variable. It's going to be stuck in a data structure. It's going to be used for a side effect. So, inside of code, or inside of a script, there's no print, unless we make it explicit. And that's a little bit down in the weeds, it's a detail, but one I want to stress. You need to, if you want something to be printed out inside your code, you need to tell the machine to do that.

    OK. So let's do some simple examples. We've already seen somebody's. I just want to remind you, if I wanted to, for example, type in an expression like that, notice the syntactical form, it's an expression, a number, followed by an operand, followed by another expression. And of course I get out the value I'd like there.

    Yes sir. Oh, you don't like leaning that far to the left? OK, if you're a Republican I'll be happy to shift this over a little bit. Wow, John, I got a laugh for a political joke, I'm in big trouble. That better? Oh damn, all right, I'll have to do it even more. OK, here we go, here we go, you see, I'm doing it down here, I can't see it, does that-- ah, I hear his sighs of relief, OK, good. There we go. Better. All right.

    One of the other things we showed last time is that operators are overloaded. And this is where you heard John and I disagree. I don't happen to like this, but he thinks it's an ok thing. In a particular-- if we, whoa, we don't do that, we do this-- that is, give a combination of a number multiplication in a string, this will in fact give us back a new string with that many replicas, if you like, of the string concatenated together. All right? And if you want to do other things, for example, we can take two strings and add-- whoops, sorry-- and add them together, we will get out, again, a concatenation of that string. And these will, we'll let you work through the variations, but these are the simple expressions we can use. Now, sometimes things get a little interesting. All right? What's the value of that expression? What do you think should happen if I evaluate that expression? Somebody with a hand up, so I can see it. What's going to happen?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR JIM ERICSON: An error? Why?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR JIM ERICSON: Great. OK. That means, let's check it. It certainly is. We bribe people. So I, ah, by the way, John's a Yankees fan, he throws like Johnny Damon, I'm a Red Sox fan, so we'll see if I, how about that? And I almost hit John along the way, great. My third right, exactly, what can I say? All right, so we're into bribing you as we go along here, and all right? You'll be badly overweight by the end of the term.

    Right, it's a syntactic error, because it doesn't know how to deal with this. But there's an important thing going on here, if I in fact wanted to combine those into a string, I should have told the machine to do that, and I can do that, by explicitly saying, take that, which is a number, and convert it into a string, and then-- bleah, I keep doing that-- then add it to that string.

    OK, so there's an important point here. We've got what's called type conversion. That is, if I want to combine two things together in a particular way, I need to make sure that I give it the kind of operand it expects. So STR, which I just typed up there, takes in parens, some input, and it converts it into a string, so that now I can use that where I was expecting a string. John.

    PROFESSOR JOHN GUTTAG: You've got a static semantic error in your syntax.

    PROFESSOR JIM ERICSON: Thank you. And I was going to come to that in a second, but thank you, John, for pointing it out.

    All right. Why is it a static semantic error? The syntax is OK in the sense of, it is an operand, an operator, an operand, so syntactically it's OK. The semantics was what caused the problem, because the operator was expecting a particular kind of structure there.

    There's a second thing going on here that I want to highlight, because it's really important. Yes indeed. OK, there we go. The second thing I want to highlight is, that what's going on, is that Python is doing some type checking. It caught the error, because it checked the types of the operands before it applied things, and it says, I'm going to stop.

    Now, you might have said, gee, why didn't it just assume that I wanted to in fact treat these as strings, and combine them together? Sounds like a reasonable thing to do. But it's a dangerous thing. Because in doing that, Python would then have a value that it could pass on into some other part of a computation, and if it wasn't what I wanted, I might be a long ways downstream in the computation before I actually hit some result that makes no sense. And tracing back where it came from can be really hard. So I actually want to have type checking as much as I can early on.

    And in fact, under type checking, different languages sometimes fall on a spectrum from weak to strong typing. Which basically says, how much type checking do they do? Now, you're going to hear John and I go back and forth a lot, as I said I'm an old time-- well I'm certainly old time, but I'm also an old time Lisp programmer. I love Lisp, but Lisp is certainly in the category of a very weakly typed language. It does not check the types of its arguments at all. Python is, I wouldn't say completely strong, but it's much closer to the strong end of the spectrum. It's going to do a lot of type checking for you before it actually passes things back. Nonetheless, I'm also going to argue that it's probably not as strongly typed as we might like.

    So, for example, there's an expression. Now, less than is just, if you haven't used it before, it's just the operator you'd expect, it's comparing two things and it's going to return either true or false depending on whether the first argument is less than the second argument. What's going to happen here? Again, I need a hand so I can know where to throw candy. I've also got on my reading glasses on, I can't see anything. Anybody. TAs don't count, they get their own candy. When it, yep.

    STUDENT: [INAUDIBLE]

    PROFESSOR JIM ERICSON: Good question. Sounds like a reasonable guess, right? How in the world am I going to compare a string to a number? So, see how good my aim is, ah, not bad. All right. A good quest-- sorry, a good thought, but in fact, son of a gun. Or as my younger son would say, fudge knuckle. Yeah. All right? So, what in the world's going on here? This is a place-- I don't know about you, John, I think this is actually really not good, because right, what this is doing is, it's allowing-- sorry, let me back up and say it-- it's got an overload on the less-than that allows you to compare basically the lexicographic ordering, or this sequence of ordering of symbols, including numbers inside of the machine. And this, in my mind, should have been an error. Why in the world would you want to compare that?

    Just to give you an example of that, for instance, I can do the following: all right, the number four is less than the string three, whereas the string four, oops, is not less than the string three. And this is a place where it's comparing strings and numbers in a strange way.

    So why am I showing you this? Partly to show you that it's kind of weird, but also to tell you that one of the things you want to do is exercise what I'm going to call some type discipline. Meaning, when you write code, you want to get into the habit of A, checking out operators or procedures to see what they do under different circumstances, either check them out or read the specifications of it, and two, when you write your own code, you want to be disciplined about what types of arguments or operands you apply to operators. Because this is something that could certainly have screwed you up if you didn't realize it did it, and you need to have that discipline to make sure it's doing the right thing.

    OK. One of the other things you're going to see is that some of the operators have odd meanings. And again, I think we looked-- Yup?

    STUDENT: So, the string A is less than three, is false because they're comparing like ASCII values?

    PROFESSOR JIM ERICSON: Yes. I mean, I'm sorry. The answer is, I don't know if it's ASCII. John, do you know, are they doing ASCII encoding inside of here? I'm assuming so. Right. So, in case you didn't understand what the the question was, basically every symbol gets translated into a particular encoding, a string of bit, if you like, inside the machine, there's a particular one called ASCII, which is, if you like, an ordering of that, and that's what the machine's actually comparing inside of here, which is why in under ASCII encoding the numbers are going to appear after the characters, and you get the strange kind of thing going on.

    All right. I want a couple of other things, just to quickly remind you, and one of them is, remember, the operators do look at the types, so division, for example nine divided by five is one, because this is integer division, that is, it's the largest number of integer multiples of five to go into nine, and there would be a remainder associated with it, which is in fact four. And again, you've got to be careful about how you use the operators.

    Right, having done that, we can certainly get to more complicated things, so for example, suppose I look at that expression. Three plus four times five. All right. Now. There are two possible values here, I think. One is 23, the other's 35. Because this could be three plus four, times five, or it could be three, plus four times five. And of course, you know, when you look at code it doesn't pause in between them. But what I do? I just separated, do I do the addition first or do the multiplication first? Anybody know what happens In this case? Yeah, way up, oh God I'm going to have a hell of time throwing up there, way up at the back.

    STUDENT: Standard order of operations, I guess take the multiplication first, and add the three.

    PROFESSOR JIM ERICSON: Right. I'm going to try, if I don't make it, you know, just get somebody to pass back, whoa! I just hit somebody in the head. Thank you. Please pass it back to that guy. If you want candy, sit much closer down, and that way we can film you as well as we go along. Right.

    So the point is, there is something here called operator precedence, which is what the gentleman said. I'm not going to say much more about it, but basically what it says is, with all other things being equal, things like exponentiation are done before you do multiplication or division, which are done before you do things like addition and subtraction. And so, in fact, if I wanted the other version of it, in fact, if I do this right, it's going to give me 23 because it does the multiplication first, if I wanted the other version of it, I need to tell it that, by using, excuse me, parentheses. And in general, what I would say is, when in doubt, use parens. OK.

    Now, that just gives us expressions. We can start having complex expressions, you can imagine we can have things are lots of parens, and all sorts of things in it. Yes, question.

    STUDENT: What does it mean, the operator used, when you were calculating the remainder between nine and five?

    PROFESSOR JIM ERICSON: It's the percent sign. If you can't read it, I guess I'm going to have to blow that font up, aren't I, next time around. Yeah, it's a percent, so this percent sign will give you the remainder.

    OK. Second thing I need to do, though, is I need to, when I get those values, I want to hang on to them. I'd like to give them a name, so I can refer to them in other places. And so we saw that as well, the last piece we had here is the ability to create variables, which have their own values, and that's done using an assignment statement. So in particular, that is an assignment statement. It says, take the name x and create a binding for that name to the value of the sub-expression and in fact to do this, to stress a point, let's do that. It's not just a number, it's any expression. What Python will do, is it will evaluate that expression using the kinds of rules we talked about, and then it creates a binding for x to that value. And I want to stress this, we're going to come back to it later on in the term, so the way I'd like you to think about it for now, is that somewhere in the machine, there's a big space that contains all possible values. Right. It's a slight lie, it doesn't have all possible values, but you get the idea. It has, if you like, intellectually, all possible values. And when I create a binding, I'm taking a variable name, in this case x, stored somewhere in a table, and I'm creating a link or a pointer from that name to that value. This is a nuance. It's going to make a lot more sense later on when we introduce mutation into our language, but I want you to start thinking of it that way. Don't think of it as a specific box into which we're putting things; think of it as a link to a value.

    I could have, for example, another assignment statement, and that creates a binding from y into that same value, and one of the things as a conservist I can do is, I could have a statement like, let z be bound to the value of x. And I said it deliberately that way. That statement says, get the value of x, which is this link, and give z a pointer to the same place. To the value, not to x. OK, and we'll just plant that idea, we're going to come back to later on, as we carry on.

    OK. So if we have variables, one of the questions we can ask is, what's the type of the variable. And the answer is, it inherits it from its value. OK. Yes. So if somewhere in my code, I have that statement, that assignment statement, x now is a variable whose value is an integer. Unfortunately, at least in my mind, in Python, these variable bindings are dynamic, or the type, rather, is dynamic. Meaning, it changes depending on what the current value is. Or said a different way, if somewhere later on in the program I do this, x now has changed its type from INT to string. Now why should you care? OK, my view is, I don't like it. Especially in the presence of operator overload. Because I might have written some code in which I'm expecting that particular variable to have an integer value. If somewhere later on in the code it shifts to string, I might not be manipulating that and getting actual values out, but not what I wanted, and it's going to be really hard for me to chase it back. So one of the things I would like to suggest is that you develop some good style here, and in particular, don't change types arbitrarily. I can't spell today. Meaning, sometimes you need to do this, but in general there's-- at least in my view and I don't, John, would you agree?-- you just don't want to do this. You don't want to make those changes. It just leads to trouble down the road.

    OK. Now, last thing about variables, and then we're going to start pushing on this, is where can you use them? And the answer is, you can use a variable anywhere you can use the value. So, any place it's legal to use the value.

    OK. Now. This is just sort of bringing us back up to speed and adding a few more details in. What we really want to do now though is start using this stuff. So, operands. Let us take expressions, get values out, we can store them away in places, but ultimately we want to do something with them, so we need to now start talking about what are the things we can do inside of Python, or any programming language, to manipulate them. And for that, we're going to have statements.

    Statements are basically, if you want to think about it, legal, and I was about to use the word expression except I've misused that elsewhere, so legal commands that Python can interpret. You've already seen a couple of them. Print, assignment, certainly two obvious statements, they're commands to do something. Assignment is binding a name to a value, print is saying put it back out in the screen. Obviously if you have print as a way of putting things out, we expect to have ways of getting input in, we're going to see an example that in the second. And as we go through the next few lectures, we're going to add in more and more of these statements.

    But let's look at what we could do with this, OK? And to do this, I'm going to use some code that I've already typed in. So I'm hoping you can read that, and it's also in your handout. This is a little file I created, all right, and I'm going to start with a sequence of these things and walk them along, again I invite you to put comments on that handout so that you can follow what we're going to do. All right?

    So let's look at the first part of this. Right now, this is just a text file. OK. And I've highlighted in blue up there one of the pieces I'm going to start with. And what do I have? I have a sequence of commands; I've got an assignment statement, I've got another assignment statement, I've got a print statement, I've got an input statement, which we'll come back to in a second. And I want to basically try and use these things to do something with them.

    Second thing I want to note is, the little hash mark or the pound sign, that's identifying a comment. So what's a comment? It's words to you, or to the reader of the code, that are telling you what's going on inside of this code. OK?

    Now, these comments, frankly, are brain-damaged, or computationally challenged if you prefer. Meaning, why in the world do I have to tell the reader that I'm binding x to the value three? All right? I'm putting them in there to make a point. In general, good programming style says you put in comments that are going to be valuable in helping you as a reader understand what's going on inside of the code. It could be, what's the intuition behind this piece of code. It could be, preconditions I want to have on input. It could be, explanations of specific things you're doing. But you need to have those comments there.

    Now, this becomes a little bit of one of those motherhood and apple pie kinds of lectures. You know, your mother always told you to eat brussels sprouts because it was good for you. Well this is a brussels sprouts comment. everybody goes yeah, yeah, yeah, comments, of course. Of course we're going to do comments. And they never do. So my challenge to you, and I know Professor Guttag can do this, my challenge to you is, a year from now, come back and look at code you wrote here. Can you still understand what it was you were trying to do? I don't know, John, if you'd agree, right? If you can read the code a year later, even code you wrote yourself, it's a good sign that you put good comments in, right?

    Second good piece of style here is choice of variable names. These are lousy. Deliberately. OK? I'm just using simple things like x and y and z because I want to make it, just get through the lecture if you like. But in general, the choice of variable name is a great way of commenting your code. Use variable names that make sense. That little problem set zero that you did. You read in a couple of values, you probably stored them away. My bet is, you used simple names like x and y. A much better name would have been firstname, lastname, as the name of the variable to tell you what you were trying to capture there.

    OK. The other piece I want to say about variable names is, once I have that choice of variable name, I can use it, but in fact there are a few things that I can't use in terms of variable names. So, these are an important way of documenting, but there're some things excluded. And in particular, there are some key words that Python is going to use that have to be excluded. Let me highlight that. As I said, right now that's just text file. I'm going to save this away-- yeah, not that way, I'm going to save this away-- with the subscript, or the suffix rather, py to make it a Python file. Yeah, and I know it's already there but I'm going to do it, and I get some wonderful colors. But these are important, OK? So notice what I have up there now. Comments appear in red. I can see those. There's a keyword, which I'm going to highlight right up here, print, which is in, I don't know what that color is, orange? There's a function in purple, there's a string in green, and in black I have the assignment statements.

    That print is a keyword. It's a command to Python to do something. As a consequence, I can't use it as a variable name. All right, think about it for a second. If I wanted to use print as a variable name, how do I get the system to decide gee, do I want print as a value for something, or do I want print as a command? So there's a sequence of these that are blocked out, and I-- John, I think there are what, twenty-eight? Something like that, TAs, is that right? Twenty-eight keywords that are blocked? We'll find them as we go along--

    OK. Having done this now, I can simply go ahead and run this, and in fact if I go up here to run, you'll see I've got both an option to check the module, though in this case I'm just going to run it. Oh, notice what happened. It ran through that sequence of instructions, in particular it bound x to the value three, and then it took x times x, got the value of x multipied by x, which of course is nine, bound that to the value of x, and then it printed out the value, and now it's sitting here waiting for an input. You notice what it did, it printed out that little, right up here I'd said enter a number and that's what it's printed out, so I can enter a number and it prints it out. Great.

    Let's run it again. Actually for that, I can just use, if I'm lucky, function F5, which didn't work, so let me try it again., here we go. We're going to run that module. OK.

    Whoa. What happened? I said enter a number. I didn't. I gave it a string. And it still took it. And printed it up. Well, this is one of the places where I want to come back to that highlighting of what do things do? Even though my statement said enter a number, in particular, raw input here simply takes in a set of characters and treats it as a string. And then prints it back out. So if in fact I wanted to make sure this was a number, I should have done something like either try and convert it to a number, which of course failed here, or put in a check to say where it is. So it's a way of reminding you, I've got to be careful about the types of things that I put in.

    OK. This is still boring, so let's step on the accelerator. What I have now is the following: I can write expressions, do combinations of things to get out values, I can store them away, I can print them up. But literally all I can do at this stage is write what we would call a straight-line program, that is, a program in which we execute in which we execute the sequence of instructions one by one. Simply walk down that list. That's what we just did there, right? We just walked through that list. This is boring. In fact, you can do some nice things to prove what is the class of functions you can compute with straight-line programs, and what you'd see if you did that is, it's not particularly interesting.

    OK. Let's go back and think about our recipes. What we use as our motivation here. Even in real recipes, you have things like, if needed, add sugar. That's a decision. That's a conditional. That's a branch. That says, if something is true, do something. Otherwise, do something different. So to really add to this, we need to have branching programs. What I mean by that is, a branching program is something that can change the order of instructions based on some test. And that test is usually a value of a variable. OK. And these get a whole lot more interesting.

    So let's look at a little example, and this is going to, excuse me, both allow us introduce the syntax as well as what we want to have as the flow of control inside of here. So let me go back up here, and I'm going to comment out that region, and let's uncomment this region. I want to write a little piece of code. It's going to print out even or odd, depending on whether the value I put in, which is x in this case, is even or odd.

    Think about that. That says, if this thing has some particular value, I want to do one thing; otherwise, I want to do something different. And let's look at the syntax of this. This is the first of the conditionals that we're going to see. Notice the format. I'm going to go up there. The first statement right here, that's just an assignment statement, I'm giving some value to x. We could make it something different. And then, notice the structure here. The next three statements.

    First of all, IF is a keyword. which makes sense. It is followed, as you can see there, by some expression, followed by a colon. And in fact, that colon is important, so let me stress this over here. The colon is important It's defining the beginning of a block of instructions. Yes sir.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR JIM ERICSON: Based on a test. Usually the value of a variable.

    OK, so let me go back to where I am. I'm looking at that piece of code. What that colon is saying is, I'm about to begin a sequence of instructions that I want to treat as a block. So it identifies a block of instructions. It's, and in particular, the colon is the start, and the carriage return is the end.

    Now what in the world does that mean? I'm doing a lot of words here, let me try and say this a little bit better. That code says the following: the IF says, I've got an expression, I'm going to evaluate it. If that value is true, I want to do a set of things. And that set of things is identified by the sequence of commands that are indented in, this one right here, following the colon but before I get back to the same place in terms of the indentation. If that test is not true, I want to skip this instruction, and there's a second keyword else, followed by a colon, and that tells me the thing I want to do in the case that it's false.

    So in fact if I run this, ah, and save it, and it prints out odd. So, what happened here? Well, let's look at the code. Right? x is initially bound to fifteen. I get to the IF. The IF says, evaluate that next expression. In that next expression, I'm actually taking advantage of the fact that I'm doing integer multiplication and division here. Right, that divide is, if x is an integer and two is an integer, what's it going to do? If x was even, x divided by two is going to be actually the half of x, right? If x is odd, that integer division is going to give me the number of multiples of two, that go into x, plus a remainder, which I'm going to throw away. In either case, I take that value and multiply back by two, if it was even I get back the original number, if it was odd, I'm not going to get back the original number, so I can just check to see if they're the same.

    OK, so a little nuance that I'm using there. So, the first thing that IF does, bleah that IF says is, evaluate that expression and if it's true, do the next thing, the thing after the colon. In this case it's not true, so it's going to skip down and evaluate the thing printed up the odd.

    OK. What-- yes.

    STUDENT: [INAUDIBLE]

    PROFESSOR JIM ERICSON: Thank you. I was hoping somebody would ask that question. The question was, if you didn't hear, why do I have two equal signs? It's like I'm doing this, right? Anybody have a se--%uFFFD why don't I just use an equal sign? I want to know if something's equal to something. Yeah.

    STUDENT: [INAUDIBLE]

    PROFESSOR JIM ERICSON: Absolutely. The equal sign is going to bind-- Nice catch. John, this is so much fun, throwing candy. I've got to say, we've got to do this more often-- Right. Let me, let me get to the point. What does an equal sign do? It is an assignment. It says, take this thing on the left and use it as a name to bind to the value on the right. It's not what I want here. Having already chosen to use equal as an assignment, I need something else to do comparison. And that's why I use double equals. Those two equal signs are saying, is this thing equal to, in value, the thing on the other side?

    OK. Now, having done that, again I want to stress this idea and I'm going to write it out one more time, that there's a particular format here. So we have if, and that is followed by, I'm going to use angle braces here just to indicates something goes in here, some test followed by a colon. That is followed by a block of instructions. And we have an ELSE, followed by a colon in some other block of instructions. And I want you to get used to this, that colon is important, it identifies the start, and then the set of indented things identify all the things at the same level, and when we reset back to the previous level, that's when we go back to where we were.

    OK. Now, that's a nice simple little test, let's look at a couple of other examples to get a sense of what this will do. OK, let me comment this out, and let's look at this next little piece of code. All right. I'm binding a z to be some value, and then I'm going to run this. Well, let's just run it and see what it does.

    Nothing. OK, so why? Well, let's look at it. I'm doing a test there to say, if the string x is less than the value of b, and x does not appear before b as strings, then I was going to do, oh, a couple of things, because they're at the same block level. Given that that wasn't true, it did nothing.

    Now, wait a minute, you say, where's the ELSE clause? And the answer is, I don't need one. All right, if this is purely a test of, if this is true do this otherwise I don't care, I don't need the ELSE clause in there to identify it. All right?

    Second thing I want to look at is, suppose I compare that the one below it. Oops, that I don't want to do. Comment that out, and let's uncomment this. Yeah, I've still got a binding for z and I'm giving it the same test, but notice now I've got the two same commands but they have different indentation. In this case, in fact I do get a different behavior.

    Why? Because that block identifies a set of things that I'm going to do if the test is true. If the test was not true, notice that that last command for print Mon is now back at the same level as the IF, so what this says is the IF does the test, having done the test, it decides I'm not going to do anything in the block below it, I'm going to skip down therefore to the next instruction at the same level as the IF, which gets me to the second print statement.

    OK. So now we're seeing some of these variations, let's see what else can we do here. So let me just to try something a little more interesting, and then we'll get to writing some simple programs. So I'm going to comment those out, and let's go down to this piece of code, and uncomment it. Ooh yes, that was brilliant. Let's try this again. And uncomment that, and uncomment it again. Right, so here's a little piece of code that's going to print out the smallest value of three. And notice what this is showing is that the IFs can be nested. All right, it's so if I looked at it, it's going to say that IF x is y-- sorry, IF x is less than y, THEN check to see IF x is less than z, and if that's true, print out x is the smallest. And notice the structure of it, if it's not true I'm going to go to that next ELSE, and print out that z is smallest. If the first test wasn't true, I'm going to skip that whole block and just go down and print out that y was smallest. So notice the nesting, I can flow my way through how those tests are actually going to take place.

    All right, so let's run this and see what happens. Great. y is smallest. OK. Is that code correct? Is that a tentative hand back there? Yeah.

    STUDENT: Let me compare y to [INAUDIBLE]

    PROFESSOR JIM ERICSON: Yeah, it's not doing all of the comparisons. All right, and let's just check this out, because I want to make a point of this, let's go back and do the following. Let's take y, change it to thirteen, let's run it, hmm.

    So what did I miss here? Two important points. First one, when I write a piece of code, especially code that has branches in it, when I design test cases for that piece of code, I should try and have a specific test case for each possible path through the code. And by just doing that, I just spotted, there's a bug here. And the bug was in my thinking, I did not look for all of the tests. So the way I can fix that, is, let me comment that out, and keep doing that, comment that out, let's uncomment this, notice the structure here. I now have multiple tests. So actually, let's just run it and then we'll talk about what it does.

    I run this, yeah, I have a syntax error, yes indeed, because I forgot to comment that one out, all right, and cue, we'll try it again. Ah-ha!

    And let's quickly look at the structure of this. This now has, gee, a funny thing, it says IF x is less than y AND x is less than z, then do something. And then it has a strange thing called ELIF, which is simply short for else/if in a second test. So the way to think about this in terms of flow is, it starts with that if and it says, check both of those things.

    And that fact that both of those things is the fact that we're using Boolean combination here. It is to say, we can take any logical expressions, combine them together with AND, OR, or NOT to make a complex expression, and use the value of that expression as my test. And that's literally what I've done there, right, I've got x less than y, that's a test, it returns a Boolean, which by the way is the other type, at least I would include here, it has only two values, which are true and false, and what that code says, if x is less than y, AND, logically, whatever I put up there, x is less than z, then the combination is true, and therefore I'm going to do something.

    So AND is if both arguments are true it's true, OR is if either argument it's true it's true, NOT is if the argument is not true it's true, and then the last piece, as I said is, I can now have a sequence of things I want to do. So if this is true do something else, otherwise test and see if this is true, do something else, as many as I like followed by the end. And ELSE, it says, here's what I want to do.

    OK. Now. Having added this in, I have branching instructions. I have simple branching programs. These are still awfully simple. OK? And they're awfully simple because, all I can do now, is decide whether to execute some piece of code or another. Said a different way, in the case of the straight-line programs, how long would it take to run a program? Well, basically, however many instructions I have, because I've got to do each one in order. With simple branching, how long is it going to take to run a piece of code? Well at most, I'm going to execute each instruction once. All right? Because the IFs are saying, if it's true do this, otherwise skip on it.

    Therefore, for simple branching programs, the length of time, the complexity the code, is what we would call constant. That is, it's at most the length of the actual number of instructions. It doesn't depend on the input. Real simple programs.

    Let's take another simple example. Suppose I want to compute the average age of all the MIT faculty. There's about a thousand of us. However I do that, I know that should inherently take more time than it takes to compute the average age of all the EECS faculty. There's only 125 of us. And that should take more time than what it takes to compute the average of John's and my ages, instructors in 600, because there's only two of us.

    All right, those pieces of code inherently involved something that does depend on the size of the input, or on the particular input. And that is a preface to an idea of computational complexity we're going to come back to. One of the things we want to help you do is identify the different classes of algorithms, what their costs are, and how you can map problems into the most efficient class to do the computation.

    OK. Now. Think for a second about computing the average age of the faculty. You can already kind of see what I want to do. I, somehow if, I want to walk through some sequence of data structures, gathering up or doing the same thing, adding ages in until I get a total age and then divide by the number faculty.

    How do I write a piece of code for that? Well, let's go back up to our original starting point of recipes. And I'm sure you don't remember, but one of the things I had in my recipe, is beat egg whites until stiff. OK. That until is an important word. It's actually defining a test. Let me rephrase it into garbled English that'll lead more naturally into what I want to do. While the egg whites are not stiff, beat them. That is a different kind of structure. It has a test in it, which is that while, while something is true, do something, but I want to keep doing it.

    And so for that, we need to add one last thing. Which is iteration. Or loops. We're going to see variations of this, we're going to see a variation of it called recursion, a little later on, but for now we're just going to talk about how do we do iterations. And I want to show you an example of this, to lead to both the syntax, and to the semantics. And let me comment that out, and let's go to this one.

    All right. What does this piece of code say? Not, what does it do, but what does it say. Well, the first three statements are just assignment statements. I'm binding x, y, and iters left to some values. And then notice the structure, I got a keyword WHILE, there's that color identifying it, and in parentheses I have a test. I'm expecting the value that test to be a Boolean. Followed by a colon. The colon's identifying a block of code. And what this is saying is, gee. Check to see if the variable iters left has a value greater than zero. If it does, then execute each of the instructions in that block.

    So I'm going to have an assignment of y, I'm going to have an assignment of iters left, I've got a comment that I had borrowed in order to do some debugging, and then what do I want it to do? I want it to go back around to the test. Once again, say, is that test true? If it is true, execute the sequence of instructions. So in fact we can block this out and see what it does. If I make a little chart here, I've got x, y, and iters left. x starts off as I think I set it up as, here I can't even read it, is x is three, y is zero, iters left is three. I can hand-simulate it. It says, is the value of iters left greater than zero? Yes it is. So, execute those two instructions. It says, take value of y and value of x, add them together, and create that as the new value of y. All right. That's the assigned statement. It says, take iters left, subtract one from it, and bind that as the new value of iters left.

    Having reached the end of the block, go back up and check the test. Is iters left greater than zero? Yes it is. So, evaluate the same sequence of instructions again. y plus x is six, that's my new value of y, two minus one is one, that's my new value of iters left, go back up. Is iters left greater than zero? Yes it is. So once more, thank God I didn't take 47 as an example, x plus y, subtract one from iters left, go back up to the test. Is iters left's value greater than zero? No, it is not. Therefore, skip the rest of that block of code and go to the next instruction, which is, ah, print out y.

    In fact, if we test this, son of a gun. Got a simple square procedure,. Right, It's just squaring an integer, is what it's doing.

    But notice the structure. Now I have the ability to create a loop, that is, to reuse the same pieces of code over and over again as I go around. And this adds, now, a lot of power to the kinds of code I can write.

    Notice some other things I want to highlight on this. All right? The first one is, that test has to involve-- shouldn't have to, but almost always is going to involve-- the value of some variable. What if I don't change the value of that variable inside of the code? Or, another way of saying it is, what if I did this?

    Comment it up. What happens if I run this sucker?

    STUDENT: [INAUDIBLE]

    PROFESSOR JIM ERICSON: Yeah. It'll go forever. Absolutely, right? It's going to loop into an infinite loop-- I think I can hit this close, ah, no I can't, boy what a terrible aim-- All right, what has [UNINTELLIGIBLE PHRASE] try again, the point I'm trying to make here-- thank God we're at the end of this lecture, my tongue is falling apart-- the point I'm trying to make is, that test needs to involve some loop variable that is changing. Otherwise the test is always going to be true, we're going to go off here, but this would loop forever if I did that.

    All right. Second question: or maybe a better way of saying this, and the general format you're likely to see here is, a test involving a variable name, which must be initialized outside of the loop, and which interior to the loop gets changed, so that the test is going to change. Here's the second question. What value of inputs, what values of x will this run correctly for? Probably should be integers, right? Otherwise, this is going to be doing something strange, but all integers?

    All right, suppose I do this. It's my last example. Yeah, how about that, right? We knew this was trying to do squaring, so intellectually we know we can square -4, it ought to be 16, but what happens here? Double fudge knuckle. All right? It's going to run through the loop, accumulating the answers, but because I'm subtracting, it's just going to keep making x more and more negative as it goes along, again it's off into an infinite loop. Which is a way of reminding you that I need to think as I write the code about what are my expectations from the input, and how might I enforce them. In this case, I probably want to make sure I use absolute value of x before I carry it on. Yes ma'am.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR JIM ERICSON: You're absolutely right, because I bind iters left to, um, yeah. Yes. Thank you. Boy, two candy for you. You caught me making an error. Yes. The point is, it's not going to work, and she caught both of them, impressive, it's not going to work because iters left is already negative, it's just going to skip the whole loop, and I'm in trouble, so thank you for catching that.

    All right. I was going to do one more example, but I've run you up to the end of the time. I'll leave the example on the handout, it shows you another version that we'll come back to next time. The key thing to notice is, I now have the ability to create iterations, which extends well I can do. And we'll see you next time.
  • 5
    Assignment 1: Computing prime numbers, product of primes
    4 pages
  • 6
    MIT Lecture 3: Common Code Patterns: Iterative Programs
    51:00
    Common code patterns: iterative programs. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR ERIC GRIMSON: All right, I'm going to start today by talking about, so what have we been doing? What have we actually done over the last few lectures? And I want to suggest that what we've done is, we've outlined a lot of the basic elements of programming. A lot of the basic elements we're going to need to write code. And I want to just highlight it for you because we're going to come back and look at it.

    So I'm going to suggest that we've looked at three different kinds of things. We've talked about data, we've talked about operations, and we've talked about commands or statements. All right?

    Data's what we expect. It's our way of representing fundamentally the kinds of information we want to move around. And here, I'm going to suggest we've seen numbers, we've seen strings, and I'm going to add Booleans here as well. They're a third kind of value that we saw when we started talking about conditions.

    We saw, associated with that primitive data, we have ways of taking data in and creating new kinds of data out, or new versions of data out, so we have operations. Things like addition and multiplication, which we saw not only apply to numbers, but we can use them on things like strings and we're going to come back to them again. Can't use them on Booleans, they have a different set of things. They do things like AND, and OR. And of course there's a bunch of other ones in there, I'm not going to put them all up, but we're building up a little collection, if you like, of those operations.

    And then the main thing we've done is, we've talked about commands. So I'm going to suggest we've seen now four different things. We've seen assignment, how to bind a name to a value. We've seen input and output. Print for output, for example, and raw input for input. We've seen conditionals, or said another way, branches, ways of changing the flow of control through that sequence of instructions we're building up. And the last thing we added were loop mechanisms. And here we saw, wow. It's the first example we've seen.

    So what've we done so far? Now, interestingly, this set of instructions was actually quite powerful, and we're going to come back to that later on, in terms of what we can do with it, but what we've really done is, given that basis, we're now talking about, how do we write common patterns of code, how do we write things that solve particular kinds of problems. So what I want you to do, is to keep in mind, those are the bases, we ought to be able to do a lot with that bases, but what we're really interested in is not filling out a whole bunch of other things in here, but how do we put them together into common templates. And we're going to do that today.

    Second thing we've been doing, I want to highlight for you is, we've along the way, mostly just verbally rather than writing it down, but we've been talking about good style. Good programming style. All right? Things that we ought to do, as you put these pieces together, in order to give you really good code. And you should be collecting those together.

    Give you some examples. What have we talked about? We've talked about things like using comments to highlight what you're doing in the code, to make it easier to debug. We talked about type discipline, the notion that you should check the types of operands before you apply operators to them, to make sure that they're what the code is expecting. We talked about descriptive use of good variable names, as a way, in essence, of documenting your code. The fourth one we talked about was this idea of testing all possible branches through a piece of code, if it's got conditionals in it, to make sure that every possible input is going to give you an output that you actually want to see.

    So, you know, you can start writing your own, kind of, Miss Manners book, if you like, I mean, are what are good programming, you know-- I wonder what you'd call them, John, good programming hygiene? Good programming style? Good programming practices?-- Things that you want to do to write good code.

    OK. What we're going to do today is, we're going to start now building up, beyond just these pieces, although they're valuable, to start creating two things: one, common patterns of code that tackle certain classes of problems, and secondly we're going to talk about tools you can use to help understand those pieces of things.

    OK. So last time around, we talked about, or introduced if you like, iterative programs. And I want to generalize that for a second, because we're going to come back and use this a lot. And I want to do a very high-level description of what goes into an iterative program, or how I would think about this, all right? And I know if John disagrees with me he'll tell me, but this is my way of thinking about it.

    If I want to try and decide how to tackle a problem in an iterative matter, here the steps I'm going to go through. First thing I'm going to do, is I'm going to choose a variable that's going to count. What I meant-- what in the world do I mean by that? I'm thinking about a problem, I'm going to show you an example in a second, first thing I'm going to do is say, what is the thing that's going to change every time I run through the same set of code? What is counting my way through this process?

    Now I'm putting count in double quotes, not to make it a string, but to say, this is count generically. It could be counting one by one through the integers, it could also be taking a collection of data and going through them one by one. It could be doing counting in some other mechanism. But what's the variable I want to use?

    Second thing I do, I need to initialize it. And I need to initialize it outside of the loop. That is, where do I want to start? And I need to make sure I have a command that sets that up.

    The third thing I'm going to do, is I need to set up the right end test. How do I know when I'm done with the loop? And obviously, that ought to involve the variable in some way, or it's not going to make a lot of sense, so this includes the variable, since that's the thing that's changing.

    All right. The fourth thing I'm going to do, is I'm going to then construct the block of code. And I want to remind you, that block of code is a set of instructions, the same set of instructions that are going to be done each time through the loop. All that's going to change, is the value the variable or the value of some data structures. And remind you that inside of here, I'd better be changing the variable. All right, if that variable that's counting is not changing, I'm going to be stuck in an infinite loop, so I ought to [UNINTELLIGIBLE PHRASE] that , right, expect somewhere in there, a change of that variable. All right? And then the last thing I want to do, is just decide, you know, what do I do when I'm done.

    OK. I know. It looks boring. But it's a structure of the things I want to think about when I go through trying to take a problem and mapping it into a iterative program. Those are the things I want to see if I go along.

    All right. So let me give you an example. I'm given an integer that's a perfect square, and I want to write a little piece of code that's going to find the square root of it. All right, so I'm cheating a little, I know it's a perfect square, somebody's given it to me, we'll come back in a second to generalizing it, so what would the steps be that I'd use to walk through it?

    Well if you think about these steps, here's an easy way to do it. Let's start at 1. Let's call x the thing I'm trying to find the square root of. Let's start at 1. Square it. If it's not greater than x, take 2. Square it. If it's not greater than x, take 3. Square it. And keep going, until the square of one of those integers is greater than or equal to-- sorry, just greater than x. OK, why am I doing that? When I get greater than x, I've gone past the place where I want to be. And obviously, when I get to something whose square is equal to x, I've got the answer I want, and I kick it out.

    So who knows what I've done? I've identified the thing I'm going to use to count, something some variable is going to just count the integers, I've identified the end test, which is when that square is bigger than the thing I'm looking for, I've identified basically what I want to do inside the loop, which is simply keep changing that variable, and I didn't say what I want to do when I'm done, essentially print out the answer.

    OK, so how can I code this up? Well, you might think, let's just jump in and write some code, I don't want to quite do that though, because I want to show you another tool that's valuable for thinking about how to structure the code, and that is a something called a flow chart. Now. People of Professor Guttag's and my age, unfortunately remember flow charts back-- as they say, on the Simpsons, back in the day, back in the 1960's, John, right?-- really good programmers had these wonderful little plastic stencils, I tried to find one, I couldn't find it It's a little stencil with little cut-out shapes on it, that you used to draw flow charts, I'm going to show you in a second, and you tucked it right in here next to your pocket protector with all your pens in it, you know, so, your belt was also about this high, and your glasses were this thick, you know, we have a few of those nerds left, we mostly keep them in the museum, but that was what you did with the flow chart.

    Despite making a bad joke about it, we're going to do the same thing here. We're going to do the same thing here, we're going to chart out a little bit of what should go into actually making this thing work. So here's a simple flow chart that I'm going to use to capture what I just described. And I'm going to, again, I'm actually going to do it the way they used to do it, and draw a rectangle with rounded corners, that's my starting point, and then what did I say to do? I said I need to identify a variable, I'm going to give it a name, let's just call ANS, for answer, and I need to initialize it, so I'm going to come down, and in a square box, I'm going to initialize ANS to 0.

    And now I want to run through the loop. What's the first thing I do in a loop? I test an end test. So the flow chart says, and the tradition was to do this in a diamond shape, I'm going to check if ANS times ANS-- oh, which way did I want to do this-- is less than or equal to x. Now that's a test. There are two possibilities. If the answer is yes, then I'm still looking for the answer, what do I want to do? Well, I don't have to do anything other than change the counter. So I'm going to go to ANS is ANS plus 1, and I'm going to do it again. Eventually, if I've done this right, that test is no-- and I wonderfully ran out of room here-- in which case, I'm going to go to a print statement, which was always done in a trapezoid, and print out ANS. I should have put a box below it that says stop.

    OK? Wow. And notice what I got here. Actually, this is a useful tool for visualizing how I'm trying to put it together, because it lets me see where the loop is, right there, it lets me see the end test, it lets me make sure that I'm in fact initializing the variable and I'm checking the right things as I go along. And the idea of this flow chart is, if you start, you know, a little ball bearing here, it's going to roll down, setting up an assignment statement, and then, depending on here, it's like there's a pair of flippers in there, it does the test, it sets the ball this way to change it to ANS plus 1, and comes back around, eventually it's going to drop through and print out the answer.

    The reason I'm going to show you this flow chart, I'm going to do one other example in a second, but I want to show you a comparison. Remember last time, we wrote this simple piece of code to print out even or odd. If, you know, x, it was in fact, even or odd. So let me show you what a flow chart for that would look like, because I want to make a comparison point here.

    If I were to do a flow chart for that one, I'd do the following. It reminds you, that the test here was, we took x if that's what we were looking for, it did integer division by 2, multiplied it by 2, and we check to see if that was the same as x. If the answer is yes, then we did a print of even. If the answer was no, we did a print of odd, and we then carried on. Again, wow.

    But there's an important point here. Remember last time, I said that there's different kinds of complexity in our code, and I suggested for simple branching programs, the amount of time it takes to run that program is, in essence, bounded by the number of instructions, because you only execute each instruction at most once. It didn't depend on the size of the input. And you can see that there.

    I start off, either I take this path and carry on, or I take that path and carry on, but each box, if you like, gets touched exactly once.

    On the other hand, look at this one. This depends now on the size of x. All right? Because what am I going to do? I'm going to come down and say, is ANS squared less than or equal to x? If it is, I'm going to go around, and execute that statement, check it again, and go around and execute that. So I'm going to cycle around that loop there enough times to get to the answer, and that number of times is going to depend on the input, so as I change the input, I'm going to change the complexity of the code.

    Now this happens to be what we would call a linear process, because the number of times I go around the loop is directly related to the size of the argument. If I double the argument, I'm going to double the number of times I go around the loop. If I increase it by five, I'm going to increase by five the number of times I go around the loop.

    We'll see later on, there are classes of computation that are inherently much more complex. We hate them, because they're costly, but they're sometimes inherently that way. But you can see the comparison between these two.

    OK. Now, having done that, let's build this code. Yeah, if my machine will come back up, there we go. So, I'm going to now go ahead and write a little piece of code, and I put it here and I hope you can actually see these better this time, let me uncomment that region. All right. So, there's basically an encapsulation of that code, right? It says-- what, look at this, where am I, right here-- I've got some value for x initially, I'm going to set ANS to 0, just like there, and there's my loop, there's the test, which is right like that, is ANS squared less than or equal to x, if it is, there's the block corresponding to the loop, change ANS, and eventually when I'm done with all this thing, I'm just going to print ANS out.

    OK. All right, let me show you one other tool that I want to use. Which is, I've written that piece of code, I ought to check it. Well, I could just run it, but another useful thing to do is, I'm, especially as I want to debug these things, is to simulate that code. And I'm going to do this because, as Professor Guttag noticed to me, students seem reluctant to do this. I guess it's not macho enough, John, to just, you know, you know, go off and do things by hand, you ought to just run them, but it's a valuable tool to get into, so let me do that here.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: I'm doing such a great job. I've got to say, when my, I've got two sons, now aged eighteen and twenty, they used to think I had the coolest job in the world because I came home covered in chalk. Now they have a different opinion that you can probably figure out.

    All right. Simulate the code. What I mean by that is, pick a simple set of values, and let's walk through it to see what happens. And this is useful because it's going to allow me to A, make sure that I've got something that's going to terminate, it's going to let me make sure that in fact I'm doing the right kinds of updates. I could do this, by the way, by running the code and putting print statements in various places as well, but the hand simulation is valuable, so let me just start it.

    What do I have here? I need the variable, ANS, I need x, and I need ANS times ANS, ANS times ANS. Right. Those are the three things that are involved in this computation. and I pick something reasonably simple. The ANS starts at 0. I set up x, I think, to be 16 there. So what does the loop say? I can either look at my flow chart, or I can look at the code. If I look at the flow chart, it says, I'm at this point. Look at ANS squared. Is it less than or equal to-- sorry, first of all, ANS squared is 0, is it less than or equal to x, yes. So what do I do? Change ANS. X doesn't change. Back around to the test. What's ANS squared? It's 1. Is it less than or equal to 16? Sure. Run the loop again. ANS becomes 2. X stays 16. ANS squared is 4. Is that less than or equal to 16? Yes. Aren't you glad I didn't pick x equals 500? All right. ANS goes up by 0. ANS squared is nine. Still less than or equal to 16. ANS goes to 4. X stays the same. 4 squared is 16. Is 16 less than or equal to 16? Yes. So ANS goes to five. ANS squared becomes 25. Ah! That is now no longer true here, so I print out 5. Right. Sure. Square root of 16 is 5. It's Bush economics. OK? I know. I'm not supposed to make bad jokes like that.

    What happened? Yeah.

    STUDENT: It doesn't stop at the right place.

    PROFESSOR ERIC GRIMSON: It doesn't stop at the right place. Thank you. Exactly. Right? My bug here is right there. Ah, let me find my cursor. I probably want that. Right? I want less than, rather than less than or equal to. This is an easy bug to come up with. But imagine, if you don't do the test, you're going to get answers that don't make any sense. And in fact, if we just go ahead and run this now, hopefully we get out-- oops, sorry, I'm going to have to change this quickly, I still have some things uncommented at the bottom, yeah, there they are, I don't think we need that yet, all right, we will comment those out.

    OK. So. Why did I do it? It's a simple example, I agree, but notice what I just did. It allowed me to highlight, is the code doing the right thing? I spotted an error here, I could have spotted it by running it on different test sets, and using prints things, another way of doing it, but this idea of at least simulating it on simple examples lets you check a couple of important questions.

    And in fact, now let me ask those two questions about this piece of code. First question is, for what values of integers-- we're going to assume integers-- but for what values of x does this code terminate? And the second question is, for what values of x does it give me back the right answer?

    All right, first question. What values of x does it terminate? Again, assume x is an integer. Well, break it down into pieces. Suppose x is positive. Does it terminate? Sure. All right? Because ANS starts out as 0, so ANS squared is 0, and each time through the loop, ANS is increasing. That means, at some point, in some finite number of steps, ANS squared has got to get bigger than x if x is positive. So for positive integers, it terminates. And it probably, I think we can deduce, returns the right answer here.

    Right. X is negative. X is -16. Does this code terminate? Boy, I feel like Arnold Schwarzenegger. Does this terminate? Somebody.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: Ah, thank you, so it does terminate, right? You're sitting too far back, let me try-- oh, too far!-- Sorry. Come get me one later if you can't find it.

    Yes, it stops at the first step, right? Let's look at it. It says, if answer, sorry, imagine x is -16, ANS is 0, is less than -16, no. So what does it do? It prints out 0.

    Ah! So that now answers my second question, it does terminate, but does it give me the right answer? No. Right? It gives me an answer, and imagine I'm using this somewhere else, you know, it's going to go off and say, gee, the square root of -16 is 0. Well, it really should be a, you know, an imaginary number, but this is not a valuable thing to have come back.

    So that's the second thing I've just highlighted here, is that I now have the ability to check whether it does the right thing.

    And those are two things that you'd like to do with every looping construct you write: you'd like to be able to assure yourself that they will always terminate, and then the second thing you'd like to do, is to assure yourself that it does give you back a reasonable answer.

    We started to talk about ways to do the former. It's looking at the end test. It's looking at the kinds of conditions you're going to put in. For the latter, this is a place where running test cases would do a good job of helping with that. Nonetheless, having done that, let's look at a better way to write this. Which is right here, it is also, I think, on your sheet, I'm going to uncomment that, and comment this one out, yeah. All right?

    So let's look at this code for a second. Notice what this does. Certainly the heart of it, right in here, is still the same thing. But notice what this does. The first thing it does is, it says, let's check and make sure x is greater than or equal to 0. If it isn't, notice what's going to happen. None of that block is going to get executed, and it's going to come down here and print out a useful piece of information, which says, hey, you gave me a negative number. I don't know how to do this.

    If it is, in fact, positive, then we're going to go in here, but now notice what we're doing here. There is the basic thing we did before, right? We're checking the end test and incrementing, actually I was going to, I commented that out for a reason you'll see in a second, but I, normally I would keep this on, which would let me, at each step, see what it's doing. If I ran this, it would print out each step. Which is helping me make sure that it's incrementing the right way.

    OK, once it gets to the end of that, what's it going to do? It's going to come down here and, oh. What's that doing? Well, I cheated when I started. I said, somebody's giving me a perfect square, I'm looking for the square root of it. But suppose I gave this thing 15, and asked it to run. It'd still give me an answer. It just would not be the answer I'm looking for.

    So now, in this case, this code is going to, when we get here, check, and if you haven't seen that strange thing there, that exclamation point in computer-ese called a bang, it says if ANS star ANS is not equal to x, all right? What's that say, it says, I've already gotten to the end of the loop, I'm now past where I wanted to be, and I'm going to check to make sure that, in fact, this really is a perfect square. If it isn't, print out something says, ah, you gave me something that wasn't a perfect square. And only if that is true, am I going to print out the answer.

    It's the same computation. But this is a nice way of writing it, often called defensive programming. And I think we have lots of variations on it-- I don't know about John, what your favorite is, for the definition of defensive programming-- for me it says, make sure that I'm going through all possible paths through the code, make sure I'm printing out, or returning if you like, useful information for each style, sorry, for each path through the code, make sure that for all possible inputs there is a path through the code, or a way to get through the code, that does not cause an error or infinite loop. What else would you add, John?

    PROFESSOR JOHN GUTTAG: Well, we'll come back to this later in the term, and talk in some detail about particular techniques. The basic idea of defensive programming is, to assume that A, if you're getting inputs from a user, they won't necessarily give you the input you've asked for, so if you ask for a positive number, don't count on them giving you one, and B, if you're using a piece of a program written by a programmer who is not perfect, perhaps yourself, there could be mistakes in that program, and so you write your program under the assumption that, not only might the user make a mistake, other parts of your program might make a mistake, and you just put in lots of different tests under the assumption that you'd rather catch that something has gone wrong, then have it go wrong and not know it. And we'll talk later in the term about dozens of different tricks, but the main thing to keep in mind is the general principle that people are dumb. And will make mistakes. And therefore, you write your programs so that catastrophes don't occur when those mistakes are made.

    PROFESSOR ERIC GRIMSON: Good. As John said, we're going to come back to it. But that's what, basically the goal here. And you saw me put my hands up when I said stupid programmer? I've certainly written code that has this problem, I've tried to use my own code that has this problem, and good to us, right, good hygiene, I'm going to use that word again here, of getting into the habit of writing defensive code up front, it's part of that collection of things that you ought to do, is a great thing to do.

    I stress it in particular because, I know you're all going to get into this stage; you've got a problem set due in a couple of hours, you're still writing the code, you don't want to waste time, and I'm going to use quotes on "waste time", doing those extra things to do the defensive programming, you just want to get the darn thing done. It's a bad habit to get into, because when you come back to it, it may haunt you later on down the road. So really get into that notion of trying to be defensive as you program.

    OK. The other thing I want to say here, is that this style of program we just wrote, is actually a very common one. And we're going to give it a nice little name, often referred to as exhaustive enumeration.

    What does that mean? It says, I'm literally walking through all possible values of some parameter, some element of the computation, testing everything until I find the right answer. All right, so it's, you know, again, I can even write that down, essentially saying, try all reasonable values until you find the solution. And you might say, well, wait a minute, isn't that going to be really expensive? And the answer is, yeah, I guess, if you want to search, you know, all the pages on Google, one by one, yes, probably, it's going to take a while. But there are an awful lot of computations for which this is the right way to do it. You just want to exhaustively go through things.

    And just to give you a sense of that, let me show you an example. I'm going to change this, all right? Nice big number. You know, computers are fast these days. I can make this even bigger, it's going to do it fairly quickly, so it really is quick to do this. It doesn't mean that exhaustive enumeration is a bad idea, it is often the right idea to use.

    So we've seen one example of this, this idea of walking through all the integers looking for the square root. Let's look at some other examples, in order to try and see other ways in which we could do it.

    OK. In particular, let's go over to here, and let me show you a second example. And let me comment that out. Here's another problem that I'd like to solve. Suppose I want to find all the divisors of some integer, I want to figure out what all the divisors are that go evenly into it. Again, same kind of reasoning says, given some value x, I happened to pick a small one here, what's an easy way to do this? Well, let's just start at one. That's my variable I'm going to change and check. Does it divide evenly into x? If it does, print it out. Move on to the next one, print it out. So again, I can do the same kind of thing here, you can see that, in fact, let's just run it to make sure it does the right thing, OK? In fact, if I go back to the code, what did I decide to do here? I say, starting with an initialization of I, there's my first step, as equal to 1, I'm going to walk through a little loop where I check, as long-- first of all, as long as I is less than x, so there's my end test, I'm going to do something. And in this case, the something is, I'm going to look to see if I divides x evenly. So I'll remind you of that amp-- sorry, that percent sign there, that says if x divided by I has a remainder, because this gives me back the remainder, if that's equal to 0, print something out. And there's my nice increment. Simple little piece of code. Notice again, exactly the same form: I picked the thing I wanted to vary, I initialized it outside the loop, I have a test to see when I'm done, and then I've got a set of instructions I'm doing every time inside the loop. In this case, it's doing the check on the remainder and printing them out. And when I'm done with the whole thing, before I end the increment of the variable, you know, when I'm done, I'm just not returning anything out. OK. So now you've seen two simple examples. Let me generalize this. In this case, my incrementer was just adding 1 to an integer, it's a pretty straightforward thing to do. But you can imagine thinking about this a little differently. If I somehow had a collection, an ordered collection of all the integers, from 1 to 10, I could imagine doing the same thing, where now what I'm doing is, I'm starting with the first element of that collection, doing something, going to the next element, doing something, going to the next element, doing something, I'm just walking through the sequence of elements. Right? And I haven't said yet, how do I get that collection, but you could certainly conceptualize that, if I had that collection, that would be nice thing to do. That is a more common pattern. That is basically saying, given some collection of data, I want to have again a looping mechanism, where now my process is, walk through this, the collection, one element at a time. And for that, we have a particular construct, called a FOR loop. It's going to do exactly that for us. It's going to be more general than this, and we're going to come back to that, in fact, Professor Guttag's going to pick this up in a couple of lectures, but we can talk right now about the basic form. The form of a FOR loop says, FOR, and I'm going to put little angle braces in here again, to say, for some variable, like a name I want to get to it, in some collection, and then I have a block of code. And what it's saying semantically is, using that variable as my placeholder, have it walk through this collection, starting at the first thing, execute that code, then the next thing, execute that code, and so on. One of the advantages of this is, that I don't have to worry about explicitly updating my variable. That happens for me automatically. And that's very nice, because this allows me to be sure that my FOR loop is going to terminate. And because, as long as this collection is finite, this thing is just going to walk through. All right? So, if I show you, for example, I'm going to comment this one out in the usual manner, and let's look at uncommenting that, there is the same piece of code. Now, I slung something by you, or snuck something by you, which is, I hadn't said how to generate the set of integers from 1 to 10. So, range is a built-in Python function. I'm going to come back to it in a second. For now, just think of it as saying, it gives you all the integers from 1 up to, but not including, x. OK. But now you can see the form. This now says, OK, let I start as the first thing in there, which is 1, and then do exactly as I did before, the same thing, but notice I don't need to say how to increment it. It's happening automatically for me. OK. In fact, if I run it, it does the same thing, which is what I would expect. OK. Now, the advantage of the FOR, as I said, is that it has, then, if you like, a cleaner way of reading it. I don't have to worry about, do I initialize it, did I forget to initialize it outside the loop, it happens automatically just by the syntax of it, right there, that's going to start with the first element. I don't have to worry about, did I remember to put the incrementer in, it's going to automatically walk it's way through there. Second advantage of the FOR is, that right now, we're thinking about it just as a sequence of integers. We could imagine it's just counting its way through. But we're going to see, very shortly, that in fact those collections could be arbitrary. We're going to have other ways of building them, but it could be a collection of all the primes. Hm. There's an interesting thing to do. It could be a collection of, ah, you know, I don't know, batting averages of somebody or other. It could be arbitrary collections that you've come up with in other ways. The FOR is, again, going to let you walk through that thing. So it does not have to be something that could be described procedurally, such as add 1 just to the previous element. It could be any arbitrary collection. And if I were to use that again, I'd just put it on your handout, I could go back and rewrite that thing that I had previously for finding the square roots of the perfect squares, just using the FOR loop. OK. What I want to do, though, is go on to-- or, sorry, go back to-- my divisor example. [UNINTELLIGIBLE PHRASE] OK. Try again. I've got a number, I want to find the divisors. Right now, what my code is doing is, it's printing them up for me, which is useful. But imagine I actually wanted to gather them together. I wanted to collect them, so I could do something with them. I might want to add them up. Might want to multiply them together. Might want to do, I don't know, something else with them, find common divisors, of things by looking at them. I need, in fact, a way to make explicit, what I can't do that with range, is I need a way to collect things together. And that's going to be the first of our more compound data structures, and we have exactly such a structure, and it's called a tuple. This is an ordered sequence of elements. Now, I'm going to actually add something to it that's going to make sense in a little while, or in a couple of lectures, which is, it is immutable. Meaning, I cannot change it, and we'll see why that's important later on. But for now, tuple is this ordered sequence of structures. OK. And how do I create them? Well, the representation is, following a square bracket, followed by a sequence of elements, separated by commas, followed by a closed square bracket. And that is literally what I said, it is an ordered sequence of elements, you can see where they are. OK? So, let me do a little example of this. If I go back over here, let's define-- er, can't type-- I can look at the value of test, it's an ordered sequence. I need to get elements out of it. So again, I have a way of doing that. In particular, I can ask for the zeroth element of test. OK, notice, I'm putting a square bracket around it, and it gives me-- I know this sounds confusing, but this is a long tradition, it gives me-- ah, yes.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: Sorry?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: I created a list here? Ah, thank you. I'm glad you guys are on top of it. You're saying I want that. Is that right, John? Yes? OK. Sorry. You're going to see why this was a mistake in a little while. I did not want to make a list, I wanted to create a tuple thank you for catching it. I want parens, not square brackets there. You'll also see in a little while why both of these things would work this way, but it's not what I wanted. OK? So I guess I should go back, and let me do this correctly this way. Again, I can look at test, and I guess test now if I want to get the element out-- angle bracket or square bracket? I still want square bracket, that's what I thought-- OK. Now I can go back to where I was, which is a strange piece of history, which is, we start counting at 0. So the-- I hate to say it this way, the first element of this tuple is at position 0, or index 0, OK?-- so I can get the zeroth one out, I can get, if I do 2, I get the third thing out, because it goes 0, 1, 2-- notice, however, if I do something that tries to go outside the length of the tuple it complains, which is right. Tuples? also have another nice structure, which is, I can go the other direction, which is, if I want to get the last element of that tuple I give it a negative index. So, imagine, you think of it as, is it starting right, just before the beginning of the thing, if I give it a it's going to take the first one, if I give it a 1, it's going to take the next one, but I can go the other direction, if I give it a -1, it picks up the last element of the tuple.

    And again, I can go -2, go back. So this is what we would call selection. We can do things like foo of to get out the particular element. I can also pick up pieces of that tuple. Again I want to show you the format here. If I give it this strange expression, this is saying I want to get the piece of the tuple starting at index 1, it's going to be the second element, and going up to but not including index 3. And it gives me back that piece. Actually a copy of that piece of the tuple. This is called slicing. And then just to complete this. Two other nice things you can do with slices are you can get the beginning or the end of tuple. So, for example, if I say TEST and I don't give it a start but I give it an end, then it gives me all the elements up to that point. And I can obviously do the other direction which is I can say skip to index 2 and all the remaining pieces. This lets me slice out, if you like, the front part or back part or a middle part of the tuple as I go along.

    What in the world does that have to do with my divisor example? Well, actually, before I do that let me in fact fill in a piece here. Which is remember I said range we could think of conceptually as a tuple -- or sorry as a sequence of these things. In fact it gives me back, now I hate this, it's actually a list it's not a tuple. But for now think of it as giving you back an explicit version of that representation of all those elements. You'll see why I'm going to make that distinction in a couple of lectures.

    All right. What does this have to do with my divisor example? This says I can make tuples, but imagine now going back to my divisor example and I want to gather up the elements as I go along. I ought to be able to do that by in fact just adding the pieces in. And that's what I'm going to do over here. Which is, let me comment that out, let me uncomment that. And I guess I need the same thing here, right? I need parens not, thank you. You can tell I'm an old time list packer. I really do love these things. And is that right, John? OK, so my apologies that your handout is wrong. I did not think to check about the difference between these things.

    Nonetheless, having done that, let's look at what I'm going to do. I now want to run a loop where I need to collect things together. I'm going to give a name to that. And what you see there is I'm going to call divisors initially an empty tuple, something has nothing in it. Right here. And then I'm going to run through the same loop as before, going through this set of things, doing the check. Now what I'd like to do, every time I find a divisor I'd like to gather it together. So I'm going to create a tuple of one element, the value of i. And then, ah, cool. Here's that addition operation that's badly overloaded. This is why Professor Guttag likes and I don't. Because given that this is a tuple and that's a tuple, I can just add them together. That is concatenate them, if you like, one on the end of it. And if I keep doing that, when I'm done divisor will be a collection of things. So let me just run it. All right. This is what I get for trying to --

    STUDENT There should be a comment after the i in parentheses.

    PROFESSOR ERIC GRIMSON: Thank you. Right there. All right, we'll try this again. OK. And there are the set of devices. Thank you. Who did that? Somebody gets, no? Yours? Thank you. Nice catch too by the way. All right, so now that you can see that I can screw up programming, which I just did. But we fixed it on the fly. Thank you. What have we done? We've now got a way of collecting things together, right? And this is the first version of something we'd like to use. Now that I've gotten that bound as a name, I could go in and do things with that. I could go in and say give me the fourth divisor, give me the second through fifth divisor. Again as I suggested if I've got two integers and I want to find common divisors I could take those two lists and walk through them. I shouldn't say list, those two tuples, and walk through them to find the pieces that match up.

    So I've got a way now of gathering data together. The last thing I want to do is to say all right, now that we've got this idea of being able to collect things into collections, we've got the ability now to use looping structures as we did before but we can walk down then doing things to them, where else might we have this need to do things with looping structures? And I'm going to suggest you've already seen it. What's a string? Well at some level it is an ordered sequence of characters. Right? Now it is not represented this same way. You don't see strings inside these open parens and closed parens. You don't see strings with commas between them, but it has the same kind of property. It is in ordered sequence of characters. We'd like to do the same thing with strings. That is we'd like to be able to get pieces of them out. We'd like to be able add them together or concatenate them together. We'd like to be able to slice them. And in fact we can.

    So strings also support things like selection, slicing, and a set of other parameters, other properties. And let's just look at that. Again if I go back here, let me comment this out. Right here are a pair of strings that I've set up, s 1 and s 2. Let me just run these. We can go back over here. So I can see the value of s 1, it's a string. I can do things like s 1 and s 2. As we saw before, it simply concatenates them together and gives me back a longer string. But I can also ask for parts of this. So I can, for example, say give me the first element of string 1, s 1. Ah, that's exactly what we would have thought if this was represented as an ordered sequence of things. Again I should have said first, index 0, the first one. I can similarly go in and say I'd like all the things between index 2 and index 4. And again, remember what that does. Index 2 says start a 0. 1, 2. So a, b, c. And then it goes up to but not including index 4 so it gets c and d and then it stops. I can similarly, just as I did with the tuples, I can ask for everything up to some point or I can ask for everything starting at some point and carrying on.

    Now what you're seeing here then is the beginning of complex data structures. And the nice thing is that there's a shared behavior there. Just as I can have tuples as an ordered collection of things, strings behave as an ordered collection of things. So I can start thinking about doing manipulation on strings. I can concatenate them together, I can find pieces inside of them, I could actually do things with them. And let me show you just a simple little example of something I might want to do. Suppose I take, I better comment this one out or it's going to spit it out. Let me comment that out. Suppose I take a number. I'd like to add up all the digits inside of the number. I can use the tools I've just described in order to capture that.

    So what would I want to do? I'd like to somehow walk down each of the digits one at a time and add them up. Ah, that's a looping mechanism, right? I need to have some way of walking through them. An easy way to do it would be inside of a FOR. And what would I like to do? Well I need to take that number and I'm going to turn it into a string. So notice what I'm going to do right here. I take that number and convert it into a string. That's an example of that type conversion we did earlier on. By doing that it makes it possible for me to treat it as an ordered sequence of characters. And so what's the loop going to do? It's going to say FOR c, which was my name for the character in that string. That means starting at the first one, I'm going to do something to it. And what am I'm going to do? I'm going to take that character, convert it back into an integer, and add it into some digits. And I've done a little short hand here, which is I should have said some digits is equal to some digits plus this. But that little short hand there is doing exactly the same thing. It is adding that value into some digits and putting it back or signing it back into some digits. And I'll walk through that loop and when I'm done I can print out the total thing does. And if I do that, I get out what I would expect.

    So what have I done? We've now generalized the idea of iteration into this little pattern. Again as I said this is my version of it, but you can see, every one of the examples we've used so far has that pattern to it. Figure out what I'm trying to walk through. What's the collection of things I'm trying to walk through. Figure out what I want to do at each stage. Figure out what the end test is. Figure out what I'm going to do at the end of it. I can write it explicitly. I can write it inside of a FOR loop. And we've started to add, and we'll see a lot more of this, examples of collections of structures so that we don't just have to do something that can be easily described as walking through a set of things but can actually be a collection that you walk through.

    The last thing I want to point out to you is, I started out with this list. I haven't added anything to the list, right? I mean I've got a different kind of looping mechanism. I guess I should say that's not quite true. I've added the ability to have more complex data structures here. But I dropped a hint in the first lecture about what you could computer with things. In fact if you think for a second about that list, you could ask what can I compute with just that set of constructs? And the answer is basically anything. This is an example of what is referred to frequently as being a Turing complete language. That is to say with just those set of constructs, anything you can describe algorithmically you can compute with that set of constructs. So there's good news and bad news. The good news is it sounds like we're done. Class is cancelled until final exam because this is all you need to know, right? The bad news is of course that's not true. The real issue is to figure out how to build constructs out of this that tackle particular problems, but the fundamental basics of computation are just captured in that set of mechanisms. All right, we'll see you next time.
  • 7
    Assignment 2: Diophantine equations
    4 pages
  • 8
    MIT Lecture 4: Abstraction through Functions; Introduction to Recursion
    51:27
    Decomposition and abstraction through functions; introduction to recursion. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR ERIC GRIMSON: As I've done in the previous lectures, let me set the stage for what we've been doing, so we can use that to talk about what we're going to do today.

    So far, we have the following in our language. Right, we have assignment. We have conditionals. We have input/output. And we have looping constructs. These are things like FOR and WHILE loops. And of course we've got some data to go with that.

    One of the things we said last time was, with that set of things, the ability to give values-- sorry, to give names to values-- the ability to make decisions, the ability to loop as a function of that, the ability get things in and out, we said that that actually gave us a language we said was Turing-complete. And that meant, in English, that this was enough to write any program.

    Now that's a slight lie-- or actually in these days of political debates, a slight misspeaking, a wonderful word-- it is technically correct. It is enough to allow us to write any program, but it's not enough to allow us to easily write any program.

    And so I joked, badly, I'll agree, at the end of last lecture, that we can just stop now, go straight to the final exam, because this is all you need to know.

    The point is, yes it's enough to start with, but we want to add things to this that let us problem solve well. And one of the things to think about is, even though I've got all of that, let's think about what I could do, if I wanted to write a piece of code.

    Right now, you've got to write it in one file. It's a long sequence of instructions, it starts at the beginning, walks through, may jump around a little bit, but eventually comes down at the end. It's okay for the things you're doing for the early problem sets. Ten lines of code. Twenty lines of code.

    Imagine instead, you're writing code that's a hundred thousand lines, a million lines, of code. You don't want to write it in this form. All right, and the reason you don't want to do that is, well, several.

    First of all, it's really hard to figure out where everything is, where everything goes, making sure I'm in the right place. To use an ancient expression from the 1970's, which only John and I will appreciate, it's really hard to grok what that code is doing, to understand what it's trying to make happen.

    And the reason that that's the case is, what we don't have, are two important things. We don't have decomposition, and we don't have abstraction. And that's what we're going to add today.

    So what does that mean? Those are fancy terms. Decomposition is a way of putting structure onto the code. It's a way of breaking the code up into modules. Modules that makes sense on their own. Modules that we can reuse in multiple places. Modules that, if you like, isolate components of the process.

    And abstraction is related to that, abstraction is going to let us suppress details. It's going to let us bury away the specifics of something, and treat that computation like a black box. And by black box, I mean, literally behaves like a mysterious little black box. You put some inputs in, it has a contract that says if you put the right kind of inputs in you'll get a specific output coming out, but you don't have to know what's inside of that box. And that abstraction is really important.

    Again, imagine if I'm a writing a piece of code. I want to just use it, I shouldn't have to worry about what variables I use inside of it, I have shouldn't have to worry about where that is in the code, I should be able to just abstract it away. And that's what we want to add today, are those two things.

    Now, our mechanism for doing that-- or at least one mechanism, I shouldn't say the only one-- one mechanism for doing that is going to be to add functions to our language. Now, the point of a function is that it's going to provide both of these things, so the first thing it's going to do is, it's going to let us break up into modules.

    Second thing they're going to do is let us suppress detail. And in essence what that does is, the functions, and we're going to look at a bunch of examples in a second, these functions are going to give us a way to, in some or in one way of thinking about it is to create new primitives. And I'm going to put those in quotes, it's a generalization.

    What do I mean by that? The idea of a function is, that I'm going to capture a common pattern of computation. Computing square root. I'm going to capture it in a piece of code, I'm going to be able to refer to it by a name, and I'm going to suppress the details, meaning inside of that computation, you don't need to know what it does. You just need to know, if I give it the right kind of input, it'll give me back an input that satisfies the contract that I set up.

    And that in essence says, I've just created the equivalent of a new primitive. Same way that I have multiplication or division as a primitive, functions are going to give me, or somebody else who wrote them for me as part of a library, a new primitive that I'm going to be able to use. And that gives me a lot of power in terms of what I want to have inside of the language. OK.

    So, let's look at an example. To try to see what we're going to do with this. Before I do that though, let me try and give you an analogy to keep this in mind of why we want to basically build these abstractions and what we need in order to have them work together.

    So here's the supposed to say silly analogy. You can tell my jokes are always bad. John's are much better, by the way, which is why Thursday will be a much better lay-- a better lecture.

    But here's the example. You've been hired by PBS to produce a nice thirteen-hour documentary, or drama, that's going to run. And, you know, you start by saying, OK, thirteen hours, I'm going to break it up into thirteen different chunks. I'm going to assign each chunk to a different writer. And they're going to go off and write that element, that hour's worth of stuff. You can imagine what you get: each hours worth of drama, if you like, may be great, but it may have absolutely nothing to do with the other twelve hours. And unless, you know, you've been hired to do Pirandello's Six Characters In Search Of An Author, this is not a great thing, because you get something that is really confusing.

    Now, what's the point of the analogy? What do I need for those writers to all interact together? I need a specification. I need a contract that says, here's what I want in terms of things that you're going to take as input, to begin your part of the drama, here's what you're going to produce at the output, and the details of what they do inside are up to them.

    An idea of abstraction, that idea of specification, is exactly what we want to use inside of our functions. We won't make you write dramas like Pirandello, but we're going to try make you at least write good code. And that's we're going to try and do.

    All right. Let's set the stage for it. Up on the screen, I've got-- I commented it out, but I've got a piece of code that you've seen before, right up here. OK? What is that? It's the piece of code we wrote for computing square roots, square roots of actually perfect squares. [UNINTELLIGIBLE]

    Just to remind you what it does, we bound x to some value, we set up an initial variable called ANS or answer, and then we run through a little loop. All right, we're-- well actually, I should say that better, we first check to see, is x greater than or equal to zero, if it's not, then we come down here and we print something out, otherwise we run through a little loop to get the answer, and then we check it and we spit something out. It does the computation, that's fine.

    Suppose I want to compute square roots a lot of places in a big chunk of code. Right now, I have to take that piece of code and replicate it everywhere I want in my larger file. And I've got to worry about, is somebody else using ANS, answer, as a variable, in which case I've got to be really careful. Is somebody else using x as a variable? I've got to deal with a lot of those details.

    I want to abstract that. And the abstraction you see, right here. I'm going to highlight it for a second so you can see it. I want you to look at it on the handout as well. This is the creation of a function.

    And I want to describe both the syntax, what we're doing, and then the semantics of how do we use it and what does that mean. So.

    Here's the syntax of the function. First of all, we have a keyword. def. Definition or define, depending on which sort of piece of history you come from. This is a keyword to Python that says, when it reads this in the file, it says, I'm creating a definition. I'm creating a function.

    And that's follow-- so this is, let me say this is a keyboard-- that is followed immediately by a name. And this equates to that. In this case, sqrt, square root. I'm saying, this is the name I'm going to give to this function. This is the name to which I'm going to refer when I want to use this function. All right?

    And notice, immediately after that name, we have an open and close paren with another variable name inside of that. And this defines formal parameters of this function. Yup.

    PROFESSOR JOHN GUTTAG: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: It does indeed, thank you. This is me being a Scheme hacker, not a Python hacker. Yes, def has to be lowercase or won't recognize it. Thank you, John. OK. def's the keyword. I'm creating a function. sqrt-- again, I'm being careful about case-sensitive, I'm using all lowercase here, followed by an open paren, and I said, formal parameters. We'll see there could be more than one there. We're going to come back to what they mean in a second, but for now, think of them as, or think of x, in this case, as the place holder. This place holder is saying, if you give me a value for x, inside the body of this function I'm going to use that value everywhere I see x. Question.

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Ah, we're going to come back to this in a second. But the question was, do I always need an input? I can have functions with no parameters, that's fine, I will still need the open and close paren there to identify that I have no parameters. We're going to see an example in a second.

    Good question. Actually, I've got to get rid of this candy, so since it was a good question, here you go. Nice catch. Almost. Sorry. OK. No, I'm not. I'm sorry. I thought you had it, and then I've got the wrong glasses on and I realized you didn't, so I will, ah come back to that later.

    What are we doing here? We got definition, we got name, we got a set of formal parameters. Right. If you look at the rest of that code, gee, it looks a lot like what I had elsewhere. Of what I had outside of it, right? It's running through a similar set of loops. So in some sets, as long as x has the value I want, it ought to do the right thing.

    However, there's a couple of other changes there that we want to highlight. In particular, notice-- let me highlight it for you, if I can find it with the wrong glasses on-- we've got these return commands. So return is another keyword. And it basically says, when you get to this point in the computation, stop the computation. Literally, return the control from this function, and take the value of the next expression, and return that as the value of the whole computation.

    Now, the one that we're most interested in is the one where, in fact, it gets out ANS, so you see down here in the code, there's a spot where it's going to return the value of ANS, which is what we want, right? That's the thing that holds the value that we intended to have.

    But there's another couple of places in that code where it's got this funky-looking thing, return none, and notice none's in a different color. None is a special value, and it has the following slightly-odd behavior: it is a value, we can return it-- God bless-- but what none says is, there is no value coming back from this computation. So when it is returned, and we'll see this in a second, to the interpreter, it doesn't print.

    OK. It simply doesn't print anything. Nonetheless, it is actually a value and we can use it, for example, to do comparisons. If we want to know, did this function return a value or not, rather than reserving, say, -1 or some other special thing which you might want to use some other ways, it literally returns this very special value that says, there is no actual value return from this computation.

    OK. Note, by the way, if I chase through each possible path, like there's some IFs in here, there's some places to go, at least in this piece of code, every possible path through this code ends in a return. And that's a good programming discipline, to make sure that happens.

    There's an exception, which we'll see in a second, but I'll highlight, which is, if we get to the end of the procedure, there's sort of an implicit return there. In fact, a return of none, in that case. It comes out of it.

    But if I look at this, right? If I come into this code, I'm going to check this branch first, if it's not true, ah, there's a return at the end of that branch. If it is true, I do that, and then I've got a second test. If it's true, I return, otherwise a return. So there's a return branch on every possible path through the code. And that's valuable, it's something you want to think about as your right your own.

    OK. What do I do to use this, in particular? How do I invoke this? OK, so I'm going to invoke a function by passing in values for the parameters. And in this case, that literally means typing sqrt, with some value inside the parens.

    OK. Now, let's just try this out to see what happens. I'm going to make sure I've got it there, so if I type, for example, sqrt of 16, ah-ha! What did it do? Well, let's talk about what it did. What this invocation does, is the following: it binds, and I'm going to say this specifically for this example, rather than general, it binds x to 16. Just as you would have done with an assignment statement up in the top level thing.

    But this binding is local. Meaning it only holds within the confines of the code of this procedure. Relative to that, think of that as creating what we, I'm going to call a new environment. Relative to that, it does all the other execution we would do, including, notice the first instruction there is to the set up a binding for ANS.

    So answer, or ANS, is also bound only locally. Meaning, inside the confines of this environment of this procedure. ANS starts off with a value of and now we just run through that loop just like we did before. Writing Increments it slowly, checking to see if ANS squared is bigger than x, and when it gets to that point, it checks to see, is it actually a perfect square or not, and it returns it. And once it returns it, it returns a value from that return, that in this case is just printed out.

    All right. Now I want to say a couple of things about these local bindings. I'm going to repeat this a second time, because it's important. These local bindings do not affect any global bindings.

    What does that mean? Let me show you a little example, and then we'll come back to this. I've got a little function here. See, I've defined f of x to be a function that takes a value of x in, changes x to x+1, and then just returns the value. OK. So it's just adding 1 to x.

    But I want you to see now what happens if I use this. Let's bind x to the value of 3. It's creating a binding for x in this global environment. This is what the interpreter sees. All right? In fact, if I look at x, its value is 3. Let's bind z eh let's bind z to the-- if I could type it would help-- say, f of 3. OK? So the value is z is 4, it's what I expect, right? Locally x got bound to 3, I added 1 to it, whoop-dee-doo, I get back a 4. But what's the value of x? It's still 3.

    The way to think of this is, again, I've got multiple scopes, or multiple frames, or if we're going to come back to those terms, I'm going to use the word environment, because I'm an old-time Lisp hacker, multiple environments in which there are bindings.

    So let me spell this out in just a little bit more detail. What this is saying is the following. When I'm talking to the interpreter, when I'm typing things in as I just did, to that Python environment, I'm getting what I'm going to call global bindings.

    I'm going to draw a little chart here. Think of this as the, as the world of the interpreter, in that I've got things like x bound to the value of 3.

    When I call or invoke a function, think of it as creating a local table. Inside that local table, I bind the formal parameter, which is what I do I did 16 right to some value. This x only gets seen by sqrt. Inside of there, I can bind other things, like ANS gets locally bound to 0, and then it increments around and eventually we return that value out. When I get to a return from sqrt, some value is returned back to the interpreter, and that table goes away. But that table does not affect any bindings for other instances of the variable like x for ANS.

    OK. Let's look at a couple of examples, just to sort of stress that. And one of the things I wanted to show is, OK. Again, I can now use a function just as if it was a primitive, so this is just an assignment and I going to take test to be the value of that, of course nothing gets printed because that was an assignment statement. All right? So if I called sqrt alone, that return value is done, but in this case I bound it to test, so I can go look at test, and there it is.

    What happens if I do that? OK. If you look at the code, it printed out, it's not a perfect square, which is what I wanted, but now, what's the value of test? OK, I bound test to something, if I look at it, it doesn't print anything, but-- if I could type-- I can ask, is test bound to that special name none? The answer is yes.

    Boy, this seems like a nuance, right? But it's a valuable thing. It says, in each case, I return some useful value from this procedure. I can check it, so if this was part of some other computation, I want to know, did it find a perfect square or not? I don't have to go read what it printed out in the screen. This has returned a value that I can use. Because I could do a test to say, is this a return value? If it's not, I'll do something else with it. So the binding is still there, it simply doesn't print it out.

    OK. What do we have out of this? Simple, seems like, at least addition. We've added this notion of a function.

    I've highlighted some of the key things we got here, right? We have that def keyword, we've got a name, we've got a list-- or I shouldn't say a word list, we have a collection of formal parameters that we're going to use-- we have a body, and the body looks just like the normal instructions we'd use, although by the way, we ought to be able to use functions inside the body, which we're going to do in a second, and then we're going to simply return some values out of this.

    Now I started by saying, build these functions. I'm trying to get both decomposition and abstraction. Well, you hopefully can see the decomposition, right? I now have a module.

    OK, let me set the stage. Imagine I wanted to do sqrt, or square root-- no, I'm going to use sqrt, that's the name I'm using here-- square root a hundred different places in some piece of code. Without function, I'd have to copy that piece of code everywhere. Now I got one just simple thing, and I simply have isolated that module inside of that function.

    What about abstraction? Well, I've got part of what I want for abstraction. Abstraction, again, says I'm going to suppress details.

    Now that I've written sqrt, I can just use it anywhere I want in the code. You've got to rely on the fact that I wrote it correctly, but you can basically suppress the details of how it's used.

    There's one more piece that we'd like to get out of that, and that is-- you may have been wondering, what's with the funky stuttering here of three double-quotes in a row. All right? And that is a specification. Which is a really valuable thing to have.

    So what is the specification going to do? It is my place, as a programmer, to write information to the user. This is me writing one hour of that episode of Pirandello and telling the other authors, here's what I'm assuming as you use it. So it's up to me to do it right, but if I do it, I'm going to specify, what does this function do? What does it expect as input, and any other information I want to pass on.

    And notice, by the way, if I do that, I'm going to come down here, and I type sqrt and open the paren, ah-ha! It shows me what the creator, in this case actually I stole this from John so what Professor Guttag put up as his specification for this piece of code.

    Now, it's not guaranteed it's right, right? You're trusting the programmer did it right, but this now tells you something.

    What is this? This is a wonderful piece of abstraction. It is saying, you don't need to know squat about what's inside the body of this function. You don't have to worry about the parameter names, because they're going to be preserved, you don't need to worry about how I'm doing it, this tells you how you can use this, in order to use it correctly. Of course, I can then close it off, and off we go.

    All right, so that notion of abstraction and I was going to come back-- we're going to come back to multiple times during the term-- and it's not just abstraction, it's the idea of a specification.

    And just to look ahead a little bit, you could easily imagine that I might want to not just put a statement in there, what the specs are, I might want to put some constraints. Some specific things to check for, to make sure that you're calling the code right. And it becomes a powerful way of reasoning about the code, a powerful way of using the code, so those notions of specs are really important.

    Look, part of the reason I'm flaming at you is, something like square root, it seems dumb to write specs on it. Everybody knows what this is going to do. But you want to get into that discipline of good hygiene, good style. You want to write the specs so that everybody does in fact know what this piece of code is doing, and you're writing it each time around.

    OK. Now that we've got functions, let's see what we can do as a problem-solving tool using them. In a particular, I've already said I want to get this notion of modularity, it's a module I can isolate, and I want to get the notion of abstracting away the details, let's see how we can actually use that to actually write some reasonably interesting pieces of code, but in particular, to see how we can use it to capture the ideas of decomposition and abstraction.

    So I'm going to shift gears. Start with a simple problem. Boy, we're suddenly be transported to Nebraska. Or where I grew up, Saskatchewan. All right, we've got a farm air problem. I got a farmer, walks out into his yard, one morning. This farmer has a bunch of pigs in a punch-- it's been a long day-- a bunch of pigs and a bunch of chickens. And he walks out into the farmyard and he observes 20 heads and 56 legs. And for sake of argument, there are no amputees among the chickens and the pigs.

    And the question is, so how many pigs does he have, and how many chickens does he have? Wow. What a deep problem, right? But you're going to see why we're going to use this in a second. So you know how to solve this, this is a fifth-grade problem, right?

    And what's the way to solve this? System of linear equations. What are the equations here? Well, I could say, you know, the number of pigs plus the number of chickens equals 20, right? Because we've got 20 heads. And then what else do I have? Four times the number of pigs plus two times the number of chickens, assuming they're not next to a nuclear reactor, is 56.

    And then how would you solve this? Well, it's, you sort of know how you'd do it if this was grammar school right? You'd pull out your pencil and paper, you can do it as a matrix inversion if you know how to do that, or you can just simply do substitution of one equation into another to solve it.

    That's certainly one way to do it, but for computers that's not necessarily the easiest way. So another way of solving it is to do something we already saw last time, which is basically, why not simply enumerate all possible examples and check them? You could say, I could have zero chickens and 20 pigs, does that work? I've got one chicken and nineteen pigs, does that work? I've got two chickens and eighteen pigs, you get the idea.

    So I'm going to solve this by enumerate and check, which is an example of what's called a brute-force algorithm. Meaning, I'm just going to write a little loop that does that. All right, so let's go back to our code. That's right, let me pull this over a little bit, so I can see it. And what I'd like you to look at, I'm going to highlight it just for a second here, is those two pieces of code. OK?

    Let's start with solve. OK. Here's the idea of solve. I'm going to have it take in as input how many legs I got, how many heads do I have, and I just want to write a little loop.

    OK. I know how to do that, right? Write a little loop, all I'm going to do, is run a FOR loop here. I'm going to let the number of chickens be in this range. Remember what range does, it gives me a set or a collection or a tuple of integers from up to 1 - is the last value, so it's going to give me everything from up to the total number of heads.

    Knowing that, I'm going to say, OK, how many pigs are there, well that's just how we're, however many I had total, minus that amount, and then I can see, how many legs does that give, and then I can check, that the number of legs that I would get for that solution, is it even equal to the number of legs I started with, ah! Interesting. A return.

    In particular, I'm going to return a tuple. So, a pair or collection of those two values. If it isn't, then I'm going to go back around the loop, and notice what happens. If I get all the way around the loop, that is, all the way through that FOR loop and I never find a path that takes me through here, then the last thing I'm going to do is return a pair or a tuple with a special simple number none twice.

    Yep. Are you telling me I want parens there and not, and not braces? All right. I hate this language, because I always want to have parens. Every time you see a square bracket, put a paren in. All right? Thank you, Christy. I'll get it eventually .

    Having done that, right, notice what I've got. First of all, two parameters. It's OK. All it says is, when I call this, I need to pass in two parameters for this to work. All right? Now, if I want to use that, I'm going to use a second piece of code here, called Barnyard. I'm going to read in a couple of values, convert them into integers, and then I'm going to use solve to get a solution out.

    And what do I know about solve? It is going to give me back a tuple a collection of two things, and so check out the syntax. I can give two names, which will get bound to the two parts of that return tuple.

    OK, pigs will be the first part, chickens will be the second part. OK, and then once I've got that, well, notice: I can then check to see, did I return that special symbol none? Is the first part. That says, I took the branch through here that eventually got to the end and said, there wasn't a solution, in which case I'm

    going to print out, there ain't no solution, otherwise I'll print out the pieces. All right, let's check it out. Ah, what did I say? Twenty and 56, Right? OK, notice the form. I've got two parameters, they're separated by a comma. Ah, right. Sorry? Yeah, but I see-- it's legs and heads, but it should not still have--

    Oh, sorry. Thank you. I've been doing the wrong thing. I want Barnyard this way, and if I had looked when I opened the paren, it would have shown me a closed paren with no parameters. Aren't you glad I make mistakes, so you can see how well I can fix from these? All right.

    Now I call that, and it says, tell me how many heads you want, give it a 20, and tell it how many legs you want, give it 56, and it prints out the answers. I know, whoop-dee-doo. But notice what's inside if here.

    First of all, notice the modularity. I've used solve. All right? Right there. I've captured it as a computation. It's buried away, all the details are suppressed. I can use that to return values, which I can then use elsewhere, which I did-- and if I just come back and highlight this-- inside of that computation.

    But I don't have to know, inside of Barnyard, what the values are used inside of solve. I don't know what the names of the variables are, I don't care, I can basically suppress away that detail.

    Second thing we saw is, that using this as a computation, I can return multiple values. Which is actually of real value to me here as I use that. OK. Yeah. Question.

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Ah. The question was, when it returns, how does it distinguish between local and other things? So let me try and answer that.

    Inside of solve, solve creates an environment where inside of that, it has bindings for the parameters it's going to use. All right? Like, number of-- wait, what did we call a were solve-- number of legs and number of heads. OK, those are bound locally. When solve is done, it wraps up, if you like, a value that it returns. Which is that.

    That expression, or that value, or that value, literally gets passed back out of that local environment to the value that comes back out of it. So in particular, what's solved returns is a pair. It could be the pair of none, none, it could be the pair of, you know, whatever the answer was that we put up there. That value comes back out and is now available inside the scope of Barnyard. OK. And Barnyard then uses that. Question?

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Here? So the question is, why is this return on the same level as the FOR? Why do you think?

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: No. Good question. All right?

    So what's going to happen here? If I'm inside this FOR, OK, and I'm running around, if I ever hit a place where this test is true, I'm going to execute that return, that return returns from the entire procedure. OK? So the return comes back from the procedure.

    So the question was, why is this return down at this level, it says, well if I ever execute out of this FOR loop, I get to the end of the FOR loop without hitting that branch that took me through the return, then and only then do I want to actually say, gee, I got to this place, there isn't any value to return, I'm going to return none and none.

    I'm still trying to get rid of this candy, Halloween's coming, where were we? There's one, thank you. I don't think I'm going to make it, I did. Thank you.

    Make sense? The answer is no, I want parens to create tuple and I get really confused about the difference between lists and tuples. For now, the code is working. Yes is the answer, all right? And we're having a difference of opinion as to whether we should use a tuple or a list here, right?

    But the answer is yes, you can. And my real answer is, go try it out, because obviously you can tell I frequently do this the wrong way and the TAs give me a hard time every time. John.

    PROFESSOR JOHN GUTTAG: Is the microphone on?

    PROFESSOR ERIC GRIMSON: Yes.

    PROFESSOR JOHN GUTTAG: As you'll see next week, tuples and lists are very close to the same thing. In almost any place where you can get away with using tuples you can use lists.

    PROFESSOR ERIC GRIMSON: Yes.

    PROFESSOR JOHN GUTTAG: But want to emphasize word is almost, because we'll see a couple of places where if it expects a tuple and you use a list you'll get an error message. But we'll see all that next week.

    PROFESSOR ERIC GRIMSON: Right, when the real pro comes in to pick up the pieces I'm leaving behind for him. OK. Let me pull this back up. What we're doing now is we're building this encapsulation. Now one of the things you notice here by the way is, you know, this in essence just solves the simple problems. Suppose I now add one other piece to this. The farmer is not keeping a great set of things so in addition to pigs, and chickens he raises spiders. I have no idea why. He's making silk I guess. Right? Why am I giving you this example? I want to show you how easy it is to change the code. But, notice, once I've added this I actually have a problem. This is now an under-constrained problem. I have more unknowns than I have equations. So you know from algebra I can't actually solve this. There may be multiple solutions to this. What would I have to do to change my code? And the answer is fortunately not a lot.

    So I'm going to ask you to look now at this set of things, which is solve 1 and Barnyard 1. OK. The change is, well, on Barnyard 1 it looks much the same as it did for Barnyard. Right, I'm going to read in the values of the number of heads and the number of legs. I'm going to use solve 1 as before, but now I'm going to bind out three variables. And then I'm going to do a similar thing to print things out. But would the solver do? Well here what I'd like to do is to run through a couple of loops. Right, how would I solve this problem? You can use the same enumerate and check idea, but now say gee let me pick how many pigs there are. Is that the one I used first? Sorry, let me pick the number of spiders there are. Having chosen the number of spiders, let me pick how many chickens I have. With those two in place, I now know how many pigs I must have and I can run through the same solution. The reason I'm showing you this is this is another very standard structure. I now have two nested loops. One running through a choice for one parameter, another one running through a choice for a second parameter. And then the rest of the solution looks much like before. I'm going to get the total number of legs out. I'm going to check to see if it's right or not. And again I'm going to return either a three tuple there or a three tuple there. It's part of what I want, because I'm going to bind those values out.

    And if I run that example, Barnyard 1, I don't know we'll give it 20 heads, 56 legs; and it find a solution. I ought to be able to run something else. I don't know, give me some numbers. How many heads? Pick an integer, somebody.

    STUDENT: 5.

    PROFESSOR ERIC GRIMSON: 5. All right. How many legs? 10? All right. We got an easy one. Let's just for the heck of it -- I should have found some better examples before I tried this. No mutant spiders here. OK, so what have I done? I just added a little bit more now. I'm now running through a pair of loops. Again notice the encapsulation, that nice abstraction going on, which is what I want. Once I get to this stage though by the way, there might be more than one solution. Because in an under-constrained problem there could be multiple solutions. So suppose I want to capture all of them or print all of them out.

    Well I ought to be able to do that by simply generalizing the loop. And that's what the next piece of code on your a hand out shows you. I'm just going to let you look at this. If you look at solve 2, it's going to run through the same kind of loop, printing out all of the answers. But it's going to keep going. In other words it doesn't just return when it finds one, it's going to run through all of them. All right? Sounds like a reasonable thing to do. Notice one last piece. If I'm going to do that, run through all possible answers, I still want to know, gee, what if there aren't any answers? How do I return that case? And that shows you one other nice little thing we want to do, which is if I look in this code notice I set up a variable up here called Solution Found, initially bound to false. The rest of that code's a pair of loops. Pick the number of spiders. Pick the number of chickens. That sets up the number of pigs. Figure out the legs. See if it's right. If it is right, I'm going to print out the information but I'm also going to change that variable to true. And that allows me then, at the end of that pair of loops when I get down to this point right here, I can check to see did I find any solution and if not in that case print out there is no solution. So this gives you another nice piece which is I can now look for first solution, I can look for all solutions, and I can maintain some internal variables that let me know what I found. A trick that you're going to use a lot as you write your own functions.

    All right, I want to end up with the last 10 minutes with a different variation on how to use functions to think about problems. And that is to introduce the idea of recursion. How many of you have heard the term used before? How may have you heard the term used before in terms of programming languages? Great. For the rest you, don't sweat it. This is a highfalutin term that computer scientists use to try and make them look like they're smarter than they really are. But it is a very handy way of thinking about, not just how to program, but how to break problems down into nice sized chunks. And the idea behind recursion I'm going to describe with a simple example. And then I'm going to show you how we can actually use it. The idea of recursion is that I'm going to take a problem and break it down into a simpler version of the same problem plus some steps that I can execute. I'm go to show you an example of a procedure, sorry a function, in a second. But let me give you actually an analogy. If you look at US law, and you look at the definition of the US legal code that defines the notion of a natural born US citizen. It's actually a wonderful recursive definition. So what's the definition?

    If you're born in the United States you are by definition a natural born US citizen. We call that a base case. It's basically the simplest possible solution to the problem. Now if you were not born in the United States, you may still be, under definition, a natural born US citizen if you're born outside this United States, both of your parents are citizens of the United States and at least one parent has lived in the United States. There's a wonderful legal expression. But notice what that is. It's a recursive definition. How do you know that your parents, at least one of your parents satisfies the definition? Well I've reduced the problem from am I a natural born US citizen to is one of my parents a natural born US citizen? And that may generalize again and it keeps going until you either get back to Adam and Eve, I guess. I don't think they were born in the US as far as I know, or you find somebody who satisfies that definition or you find that none of your parents actually are in that category.

    But that second one is called the inductive step, or the recursive step. And in my words it says break the problem into a simpler version of the same problem and some other steps. And I think this is best illustrated by giving you a simple little piece of code. I use simple advisedly here. This is actually a piece of code that is really easy to think about recursively and is much more difficult to think about in other ways. And the piece of code is suppose I have a spring and I want to know if it's a palindrome. Does it read the same thing left to right as right to left. OK? How would I solve that? If the string has no elements in it it is obviously a palindrome. If the string has one element in it, it's a palindrome. There's the base case. If it's longer than one, what do I want to do? Well I'd like to check the two end points to see are they the same character? And if they are, then oh, I just need to know is everything else in the middle a palindrome?

    I know it sounds simple, but notice what I just did. I just used a recursive definition. I just reduced it to a smaller version of the same problem. That is if I can write code that would solve all instances of smaller size strings, then what I just described will solve the larger size one. And in fact that's exactly what I have. I would like you to look at this piece of code right here called isPalindrome. Notice what it says. I'm going to pass in a string, call it s, binds it locally, and it says the following. It says if this is a string of length or 1, I'm done. I'm going to return the answer true. Otherwise I'm going to check to see is the first and last, there's that - 1 indexing, is the first and last element of the string the same? And if that's true is everything in the string, starting at the first element and removing the last element, a palindrome? Let me remind you. By saying first element remember we start at as the initial indexing point. Wonderful recursive definition.

    OK, let's try it out. Go back over here and we're going to say isPalindrome. How did I actually spell this? Palindrome with a capital P. Only in New York, in Canada we pronounce it Palindrome. When you're teaching it you get to call it your way, I'm going to call it my way. Sorry John, you're absolutely right. OK. Notice by the way, there's that nice speck going on saying put a string here. It's going to return true if it's a PAIL-indrome and false if it's a PAL-indrome. And it says true. Now maybe you're bugged by this. I know you're bugged by my bad humor, but too bad. Maybe you're bugged by this, saying wait a minute, how does this thing stop? This is the kind of definition that your high school geometry teacher would have rapped your knuckles over. You can't define things in terms of themselves. This is an inductive definition. Actually we could prove inductively that it holds, but how do we know it stops? Well notice what the computation is doing. it's looking first to see am I in the base case, which I'm done. If I'm not I'm just going to reduce this to a smaller computation. And as long as that smaller computation reduces to another smaller computation, eventually I ought to get to the place where I'm down in that base case.

    And to see that I've written another version of this, which I'm going to use here, where I'm going to give it a little indentation. I'm going to call this palindrome 1. Sorry about that. Palindrome 1. I'm going to give it a little indentation so that we can see this. OK. Code is right here. And all it's doing is when I'm getting into the different places I'm simply printing out information about where I am. What I want you to see is notice what happened here. OK. I'm calling palindrome with that. It first calls it on that problem. And the code over here says, OK gee, if I'm in the base case do something. I'm not, so come down here check that the two end points a and a are the same and call this again also. Notice what happens. There's this nice unwrapping of the problem. I just doubled the indentation each time so you can see it. So each successive call, notice what's happening. The argument is getting reduced. And we're going another level in. When we get down to this point, we're calling it with just a string of length one. At that point we're in the base case and we can unwrap this computation. We say, ah, that's now true. So I can return true here. Given that that's true and I already checked the two end points, that's true, that's true. And I unwrap the computation to get back.

    You are going to have to go play with this. Rock it if you like to try and see where it goes. But I want to stress again, as long as I do the base case right and my inductive or recursive step reduces it to a smaller version of the same problem, the code will in fact converge and give me out an answer. All right, I want to show you one last example of using recursion because we're going to come back to this. This is a classic example of using recursion. And that is dating from the 1200s and it is due to Fibonacci. Does anyone know the history of what Fibonacci was trying to do? Sorry, let me re-ask that. Fibonacci. Which actually is son of Bonacci which is the name of his father who was apparently a very friendly guy. First of all, does anyone know what a Fibonacci number is? Wow.

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Right, we're going to do that in a second, but the answer is Fibonacci numbers, we define the first two. Which are both defined to be, or I can define them in multiple ways, and 1. And then the next Fibonacci number is the sum of the previous two. And the next number is the sum of the previous two. Do you know the history of this?

    STUDENT: [INAUDIBLE].

    PROFESSOR ERIC GRIMSON: Exactly. Thank you. Bad throw, I'm playing for the Yankees. Sorry John. The answer is Fibonacci actually was actually trying to count rabbits back in the 1200s. The idea was that rabbits could mate after a month, at age one month. And so he said, if you start off with a male and a female, at the end of one month they have an offspring. Let's assume they have two offspring. At the end of the next month let's assume those offspring have offspring. Again a male and female. The question was how many rabbits do you have at the end of a year? At the end of two years? At the end of more than that number of years, and so. We can do this with the following level definition. We're going to let pairs of 0, the number of pairs at month 0, actually it would not be it would be 1. We let the number of pairs at month 1 be 1. And then the number of pairs at month n is the number of pairs at month n - 1 plus the number of pairs at month n - 2. The sum of the previous two. If I write Fibonacci, you see it right there. And the reason I want to show you this is to notice that the recursion can be doubled.

    So this says, given a value x, if it's either or 1, either of those two cases, just return 1. Otherwise break this down into two versions of a simpler problem. Fib of x - 1 and fib of x - 2, and then take the sum of those and return that as the value. Notice if I'm going to have two different sub problems I need to have two base cases here to catch this. And if I only had one it would error out. And as a consequence, I can go off and ask about rabbits. Let's see. At the end of 12 months, not so bad. At the end of two years, we're not looking so good. At the end of three years, we are now in Australia. Overrun with rabbits. In fact I don't think the thing ever comes back, so I'm going to stop it because it really gets hung up here. And I'm going to restart it.

    What's the point of this? Again, now that I can think about things recursively, I can similarly break things down into simpler versions of the same problem. It could be one version. It could be multiple versions. And we're going to come back throughout the term to think about how to code programs that reflect this. The last point I want to make to you is, you've started writing programs that you would think of as being inherently iterative. They're running through a loop. It's a common way of thinking about problems. Some problems are naturally tackled that way. There are other problems that are much more naturally thought of in a recursive fashion. And I would suggest palindrome as a great example of that. That's easy to think about recursively. It's much harder to think about iteratively. And you want to get into the habit of deciding which is the right one for you to use. And with that, we'll see you next time.
  • 9
    Assignment 3: Matching strings: a biological perspective
    6 pages
  • 10
    MIT Lecture 5: Floating Point Numbers, Successive Refinement, Finding Roots
    44:13
    Floating point numbers, successive refinement, finding roots. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR JOHN GUTTAG: Good morning.

    We should start with the confession, for those of you looking at this on OpenCourseWare, that I'm currently lecturing to an empty auditorium. The fifth lecture for 600 this term, we ran into some technical difficulties, which left us with a recording we weren't very satisfied with. So, I'm-- this is a redo, and if you will hear no questions from the audience and that's because there is no audience.

    Nevertheless I will do my best to pretend. I've been told this is a little bit like giving a speech before the US Congress when C-SPAN is the only thing watching.

    OK. Computers are supposed to be good for crunching numbers. And we've looked a little bit at numbers this term, but I now want to get into looking at them in more depth than we've been doing.

    Python has two different kinds of numbers. So far, the only kind we've really paid any attention to is type int. And those were intended to mirror the integers, as we all learned about starting in elementary school. And they're good for things that you can count. Any place you'd use whole numbers.

    Interestingly, Python, unlike some languages, has what are called arbitrary precision integers. By that, we mean, you can make numbers as big as you want them to.

    Let's look at an example. We'll just take a for a variable name, and we'll set a to be two raised to the one-thousandth power. That, by the way, is a really big number. And now what happens if we try and display it? We get a lot of digits. You can see why I'm doing this on the screen instead of writing it on the blackboard.

    I'm not going to ask you whether you believe this is the right answer, trust me, trust Python. I would like you to notice, at the very end of this is the letter L.

    What does that mean? It means long. That's telling us that it's representing these-- this particular integer in what it calls it's internal long format.

    You needn't worry about that. The only thing to say about it is, when you're dealing with long integers, it's a lot less efficient than when you're dealing with smaller numbers. And that's all it's kind of warning you, by printing this L.

    About two billion is the magic number. When you get over two billion, it's now going to deal with long integers, so if, for example, you're trying to deal with the US budget deficit, you will need integers of type L.

    OK. Let's look at another interesting example. Suppose I said, b equal to two raised to the nine hundred ninety-ninth power. I can display b, and it's a different number, considerably smaller, but again, ending in an L.

    And now, what you think I'll get if we try a divided by b? And remember, we're now doing integer division. Well, let's see.

    We get 2L. Well, you'd expect it to be two, because if you think about the meaning of exponentiation, indeed, the difference between raising something to the nine hundred ninety-ninth power and to the one-thousandth power should be, in this case, two, since that's what we're raising to a power.

    Why does it say 2L, right? Two is considerably less than two billion, and that's because once you get L, you stay L. Not particularly important, but kind of worth knowing.

    Well, why am I bothering you with this whole issue of how numbers are represented in the computer? In an ideal world, you would ignore this completely, and just say, numbers do what numbers are supposed to do.

    But as we're about to see, sometimes in Python, and in fact in every programming language, things behave contrary to what your intuition suggests. And I want to spend a little time helping you understand why this happens. So let's look at a different kind of number.

    And now we're going to look at what Python, and almost every other programming language, calls type float. Which is short for floating point. And that's the way that programming languages typically represent what we think of as real numbers.

    So, let's look at an example. I'm going to set the variable x to be 0.1, 1/10, and now we're going to display x.

    Huh? Take a look at this. Why isn't it .1? Why is it 0.1, a whole bunch of zeros, and then this mysterious one appearing at the end?

    Is it because Python just wants to be obnoxious and is making life hard? No, it has to do with the way the numbers are represented inside the computer. Python, like almost every modern programming language, represents numbers using the i triple e floating point standard, and it's i triple e 754.

    Never again will you have to remember that it's 754. I promise not to ask you that question on a quiz.

    But that's what they do. This is a variant of scientific notation. Something you probably learned about in high school, as a way to represent very large numbers.

    Typically, the way we do that, is we represent the numbers in the form of a mantissa and an exponent. So we represent a floating point number as a pair, of a mantissa and an exponent.

    And because computers work in the binary system, it's unlike what you probably learned in high school, where we raise ten to some power. Here we'll always be raising two to some power. Maybe a little later in the term, if we talk about computer architecture, we'll get around to explaining why computers working binary, but for now, just assume that they do and in fact they always have.

    All right. Purists manage to refer to the mantissa as a significant, but I won't do that, because I'm an old guy and it was a mantissa when I first learned about it and I just can't break myself of the habit.

    All right. So how does this work? Well, when we recognize so-- when we represent something, the mantissa is between one and two. Whoops. Strictly less than two, greater than or equal to one.

    The exponent, is in the range, -1022 to +1023. So this lets us represent numbers up to about 10 to the 308th, plus or minus 10 to the 308th, plus or minus. So, quite a large range of numbers.

    Where did these magic things come from? You know, what-- kind of a strange numbers to see here. Well, it has to do with the fact that computers typically have words in them, and the words today in a modern computer are 64 bits. For many years they were 32 bits, before that they were 16 bits, before that they were 8 bits, they've continually grown, but we've been at 64 for a while and I think we'll be stuck at 64 for a while.

    So as we do this, what we do is, we get one bit for the sign-- is it a positive or negative number?-- 11 for the exponent, and that leaves 52 for the mantissa. And that basically tells us how we're storing numbers.

    Hi, are you here for the 600 lecture? There is none today, because we have a quiz this evening. It's now the time that the lecture would normally have started, and a couple of students who forgot that we have a quiz this evening, instead of a lecture, just strolled in, and now strolled out.

    OK. You may never need to know these constants again, but it's worth knowing that they exist, and that basically, this gives us about the equivalent of seventeen decimal digits of precision. So we can represent numbers up to seventeen decimal digits long. This is an important concept to understand, that unlike the long ints where they can grow arbitrarily big, when we're dealing with floating points, if we need something more than seventeen decimal digits, in Python at least, we won't be able to get it. And that's true in many languages.

    Now the good news is, this is an enormous number, and it's highly unlikely that ever in your life, you will need more precision than that.

    All right. Now, let's go back to the 0.1 mystery that we started at, and ask ourselves, why we have a problem representing that number in the computer, hence, we get something funny out from we try and print it back.

    Well, let's look at an easier problem first. Let's look at representing the fraction 1/8. That has a nice representation. That's equal in decimal to 0.125, and we can represent it conveniently in both base 10 and base 2.

    So if you want to represent it in base 10, what is it? What is that equal to? Well, we'll take a mantissa, 1.25, and now we need to multiply it by something that we can represent nicely, and in fact that will be times 10 to the -1. So the exponent would simply be -1, and we have a nice representation.

    Suppose we want to represent it in base 2? What would it be? 1.0 times-- anybody?-- Well, 2 to the -3. So, in binary notation, that would be written as 0.001.

    So you see, 1/8 is kind of a nice number. We can represent it nicely in either base 10 or base 2.

    But how about that pesky fraction 1/10? Well, in base 10, we know how to represent, it's 1 times 10 to the-- 10 to the what?-- 10 to the 1? No.

    But in base 2, it's a problem. There is no finite binary number that exactly represents this decimal fraction. In fact, if we try and find the binary number, what we find is, we get an infinitely repeating series. Zero zero zero one one zero zero one one zero zero, and et cetera. Stop at any finite number of bits, and you get only an approximation to the decimal fraction 1/10.

    So on most computers, if you were to print the decimal value of the binary approximation-- and that's what we're printing here, on this screen, right? We think in decimal, so Python quite nicely for us is printing things in decimal-- it would have to display-- well I'm not going to write it, it's a very long number, lots of digits-- however, in Python, whenever we display something, it uses the built-in function repr, short for representation, that it converts the internal representation in this case of a number, to a string, and then displays that string in this case on the screen. For floats, it rounds it to seventeen digits. There's that magic number seventeen again. Hence, when it rounds it to seventeen digits, we get exactly what you see in the bottom of the screen up there.

    Answer to the mystery, why does it display this? Now why should we care? Well, it's not so much that we care about what gets displayed, but we have to think about the implications, at least sometimes we have to think about the implications, of what this inexact representation of numbers means when we start doing more-or-less complex computations on those numbers.

    So let's look at a little example here. I'll start by starting the variable s to 0.0 . Notice I'm being careful to make it a float. And then for i in range, let's see, let's take 10, we'll increase s by 0.1 .

    All right, we've done that, and now, what happens when I print s? Well, again you don't get what your intuition says you should get. Notice the last two digits, which are eight and nine. Well, what's happening here?

    What's happened, is the error has accumulated. I had a small error when I started, but every time I added it, the error got bigger and it accumulates. Sometimes you can get in trouble in a computation because of that.

    Now what happens, by the way, if I print s? That's kind of an interesting question.

    Notice that it prints one. And why is that? It's because the print command has done a rounding here. It automatically rounds.

    And that's kind of good, but it's also kind of bad, because that means when you're debugging your program, you can get very confused. You say, it says it's one, why am I getting a different answer when I do the computation? And that's because it's not really one inside. So you have to be careful.

    Now mostly, these round-off errors balance each other out. Some floats are slightly higher than they're supposed to be, some are slightly lower, and in most computations it all comes out in the wash and you get the right answer. Truth be told, most of the time, you can avoid worrying about these things. But, as we say in Latin, caveat computor. Sometimes you have to worry a little bit.

    Now there is one thing about floating points about which you should always worry. And that's really the point I want to drive home, and that's about the meaning of double equal.

    Let's look at an example of this. So we've before seen the use of import, so I'm going to import math, it gives me some useful mathematical functions, then I'm going to set the variable a to the square root of two.

    Whoops. Why didn't this work? Because what I should have said is math dot square root of two. Explaining to the interpreter that I want to get the function sqrt from the module math. So now I've got a here, and I can look at what a is, yeah, some approximation to the square root about of two.

    Now here's the interesting question. Suppose I ask about the Boolean a times a equals equals two. Now in my heart, I think, if I've taken the square root of number and then I've multiplied it by itself, I could get the original number back. After all, that's the meaning of square root.

    But by now, you won't be surprised if the answer of this is false, because we know what we've stored is only an approximation to the square root. And that's kind of interesting. So we can see that, by, if I look at a times a, I'll get two point a whole bunch of zeros and then a four at the end.

    So this means, if I've got a test in my program, in some sense it will give me the unexpected answer false. What this tells us, is that it's very risky to ever use the built-in double--equals to compare floating points, and in fact, you should never be testing for equality, you should always be testing for close enough.

    So typically, what you want to do in your program, is ask the following question: is the absolute value of a times a minus 2.0 less than epsilon? If we could easily type Greek, we'd have written it that way, but we can't.

    So that's some small value chosen to be appropriate for the application. Saying, if these two things are within epsilon of each other, then I'm going to treat them as equal.

    And so what I typically do when I'm writing a Python code that's going to deal with floating point numbers, and I do this from time to time, is I introduce a function called almost equal, or near, or pick your favorite word, that does this for me. And wherever I would normally written double x equals y, instead I write, near x,y, and it computes it for me.

    Not a big deal, but keep this in mind, or as soon as you start dealing with numbers, you will get very frustrated in trying to understand what your program does.

    OK. Enough of numbers for a while, I'm sure some of you will find this a relief. I now want to get away from details of floating point, and talk about general methods again, returning to the real theme of the course of solving problems using computers.

    Last week, we looked at the rather silly problem of finding the square root of a perfect square. Well, that's not usually what you need. Let's think about the more useful problem of finding the square root of a real number.

    Well, you've just seen how you do that. You import math and you call sqrt. Let's pretend that we didn't know that trick, or let's pretend it's your job to introduce-- implement, rather-- math. And so, you need to figure out how to implement square root.

    Why might this be a challenge? What are some of the issues?

    And there are several. One is, what we've just seen might not be an exact answer. For example, the square root of two. So we need to worry about that, and clearly the way we're going to solve that, as we'll see, is using a concept similar to epsilon. In fact, we'll even call it epsilon.

    Another problem with the method we looked at last time is, there we were doing exhaustive enumeration. We were enumerating all the possible answers, checking each one, and if it was good, stopping.

    Well, the problem with reals, as opposed to integers, is we can't enumerate all guesses. And that's because the reals are uncountable. If I ask you to enumerate the positive integers, you'll say one, two, three, four, five. If I ask you to enumerate the reals, the positive reals, where do you start? One over a billion, plus who knows?

    Now as we've just seen in fact, since there's a limit to the precision floating point, technically you can enumerate all the floating point numbers. And I say technically, because if you tried to do that, your computation would not terminate any time soon.

    So even though in some, in principle you could enumerate them, in fact you really can't. And so we think of the floating points, like the reals, as being innumerable. Or not innumerable, as to say as being uncountable. So we can't do that.

    So we have to find something clever, because we're now searching a very large space of possible answers. What would, technically you might call a large state space. So we're going to take our previous method of guess and check, and replace it by something called guess, check, and improve.

    Previously, we just generated guesses in some systematic way, but without knowing that we were getting closer to the answer. Think of the original barnyard problem with the chickens and the heads and the legs, we just enumerated possibilities, but we didn't know that one guess was better than the previous guess.

    Now, we're going to find a way to do the enumeration where we have good reason to believe, at least with high probability, that each guess is better than the previous guess.

    This is what's called successive approximation. And that's a very important concept. Many problems are solved computationally using successive approximation.

    Every successive approximation method has the same rough structure. You start with some guess, which would be the initial guess, you then iterate-- and in a minute I'll tell you why I'm doing it this particular way, over some range. I've chosen one hundred, but doesn't have to be one hundred, just some number there-- if f of x, that is to say some some function of my--

    Whoops, I shouldn't have said x. My notes say x, but it's the wrong thing-- if f of x, f of the guess, is close enough, so for example, if when I square guess, I get close enough to the number who's root I'm-- square root I'm looking for, then I'll return the guess.

    If it's not close enough, I'll get a better guess. If I do my, in this case, one hundred iterations, and I've not get-- gotten a guess that's good enough, I'm going to quit with some error. Saying, wow. I thought my method was good enough that a hundred guesses should've gotten me there. If it didn't, I may be wrong.

    I always like to have some limit, so that my program can't spin off into the ether, guessing forever.

    OK. Let's look at an example of that. So here's a successive approximation to the square root. I've called it square root bi. The bi is not a reference to the sexual preferences of the function, but a reference to the fact that this is an example of what's called a bi-section method.

    The basic idea behind any bi-section method is the same, and we'll see lots of examples of this semester, is that you have some linearly-arranged space of possible answers. And it has the property that if I take a guess somewhere, let's say there, I guess that's the answer to the question, if it turns out that's not the answer, I can easily determine whether the answer lies to the left or the right of the guess.

    So if I guess that 89.12 is the square root of a number, and it turns out not to be the square root of the number, I have a way of saying, is 89.12 too big or too small. If it was too big, then I know I'd better guess some number over here. It was too small, then I'd better guess some number over here.

    Why do I call it bi-section? Because I'm dividing it in half, and in general as we'll see, when I know what my space of answers is, I always, as my next guess, choose something half-way along that line. So I made a guess, and let's say was too small, and I know the answer is between here and here, this was too small, I now know that the answer is between here and here, so my next guess will be in the middle.

    The beauty of always guessing in the middle is, at each guess, if it's wrong, I get to throw out half of the state space. So I know how long it's going to take me to search the possibilities in some sense, because I'm getting logarithmically progressed.

    This is exactly what we saw when we looked at recursion in some sense, where we solved the problem by, at each step, solving a smaller problem. The same problem, but on a smaller solution space.

    Now as it happens, I'm not using recursion in this implementation we have up on the screen, I'm doing it iteratively but the idea is the same. So we'll take a quick look at it now, then we'll quit and we'll come back to in the next lecture a little more thoroughly.

    I'm going to warn you right now, that there's a bug in this code, and in the next lecture, we'll see if we can discover what that is.

    So, it takes two arguments; x, the number whose square root we're looking for, and epsilon, how close we need to get. It assumes that x is non-negative, and that epsilon is greater than zero.

    Why do we need to assume that's epsilon is greater than zero? Well, if you made epsilon zero, and then say, we're looking for the square root of two, we know we'll never get an answer. So, we want it to be positive, and then it returns y such that y times y is within epsilon of x. It's near, to use the terminology we used before.

    The next thing we see in the program, is two assert statements. This is because I never trust the people who call my functions to do the right thing. Even though I said I'm going to assume certain things about x and epsilon, I'm actually going to test it. And so, I'm going to assert that x is greater than or equal to zero, and that epsilon is greater than zero.

    What assert does, is it tests the predicate, say x greater than or equal to zero, if it's true, it does nothing, just goes on to the next statement. But if it's false, it prints a message, the string, which is my second argument here, and then the program just stops.

    So rather than my function going off and doing something bizarre, for example running forever, it just stops with a message saying, you called me with arguments you shouldn't have called me with.

    All right, so that's the specification and then my check of the assumptions. The next thing it does, is it looks for a range such that I believe I am assured that my answer lies between the ran-- these values, and I'm going to say, well, my answer will be no smaller than zero, and no bigger than x. Now, is this the tightest possible range? Maybe not, but I'm not too fussy about that. I'm just trying to make sure that I cover the space.

    Then I'll start with a guess, and again I'm not going to worry too much about the guess, I'm going to take low plus high and divide by two, that is to say, choose something in the middle of this space, and then essentially do what we've got here.

    So it's a little bit more involved here, I'm going to set my counter to one, just to keep checking, then say, while the absolute value of the guess squared minus x is greater than epsilon, that is to say, why my guess is not yet good enough, and the counter is not greater than a hundred, I'll get the next guess.

    Notice by the way, I have a print statement here which I've commented out, but I sort of figured that my program would not work correctly the first time, and so, I, when I first typed and put in a print statement, it would let me see what was happening each iteration through this loop, so that if it didn't work, I could get a sense of why not.

    In the next lecture, when we look for the bug in this program, you will see me uncomment out that print statement, but for now, we go to the next thing.

    And we're here, we know the guess wasn't good enough, so I now say, if the guess squared was less than x, then I will change the low bound to be the guess. Otherwise, I'll change the high bound to be the guess. So I move either the low bound or I move the high bound, either way I'm cutting the search space in half each step. I'll get my new guess. I'll increment my counter, and off I go.

    In the happy event that eventually I get a good enough guess, you'll see a-- I'll exit the loop. When I exit the loop, I checked, did I exit it because I exceeded the counter, I didn't have a good-enough guess. If so, I'll print the message iteration count exceeded. Otherwise, I'll print the result and return it.

    Now again, if I were writing a square root function to be used in another program, I probably wouldn't bother printing the result and the number of iterations and all of that, but again, I'm doing that here for, because we want to see what it's doing.

    All right. We'll run it a couple times and then I'll let you out for the day. Let's go do this. All right. We're here.

    Well, notice when I run it, nothing happens. Why did nothing happen? Well, nothing happens, it was just a function. Functions don't do anything until I call them. So let's call it. Let's call square root bi with 40.001 Took only one at-- iteration, that was pretty fast, estimated two as an answer, we're pretty happy with that answer.

    Let's try another example. Let's look at nine. I always like to, by the way, start with questions whose answer I know. We'll try and get a little bit more precise.

    Well, all right. Here it took eighteen iterations. Didn't actually give me the answer three, which we know happens to be the answer, but it gave me something that was within epsilon of three, so it meets the specification, so I should be perfectly happy.

    Let's look at another example. Try a bigger number here.

    All right? So I've looked for the square root of a thousand, here it took twenty-nine iterations, we're kind of creeping up there, gave me an estimate.

    Ah, let's look at our infamous example of two, see what it does here. Worked around.

    Now, we can see it's actually working, and I'm getting answers that we believe are good-enough answers, but we also see that the speed of what we talk about as convergence-- how many iterations it takes, the number of iterations-- is variable, and it seems to be related to at least two things, and we'll see more about this in the next lecture. The size of the number whose square root we're looking for, and the precision to which I want the answer.

    Next lecture, we'll look at a, what's wrong with this one, and I would ask you to between now and the next lecture, think about it, see if you can find the bug yourself, we'll look first for the bug, and then after that, we'll look at a better method of finding the answer.

    Thank you.
  • 11
    MIT Lecture 6: Bisection Methods, Newton/Raphson, Introduction to Lists
    50:11
    Bisection methods, Newton/Raphson, introduction to lists. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR JOHN GUTTAG: All right. That said, let's continue, and if you remember last time, we ended up looking at this thing I called square roots bi. This was using something called a bisection method, which is related to something called binary search, which we'll see lots more of later, to find square roots.

    And the basic idea was that we had some sort of a line, and we knew the answer was somewhere between this point and this point. The line is totally ordered. And what that means, is that anything here is smaller than anything to its right. So the integers are totally ordered, the reals are totally ordered, lots of things are, the rationals are totally ordered.

    And that idea was, we make a guess in the middle, we test it so this is kind of a guess and check, and if the answer was too big, then we knew that we should be looking over here. If it was too small, we knew we should be looking over here, and then we would repeat. So this is very similar, this is a kind of recursive thinking we talked about earlier, where we take our problem and we make it smaller, we solve a smaller problem, et cetera.

    All right. So now, we've got it, I've got the code up for you. I want you to notice the specifications to start. We're assuming that x is greater than or equal to 0, and epsilon is strictly greater than 0, and we're going to return some value y such that y squared is within epsilon of x.

    I'd last time talked about the two assert statements. In some sense, strictly speaking they shouldn't be necessary, because the fact that my specification starts with an assumption, says, hey you, who might call square root, make sure that the things you call me with obey the assumption.

    On the other hand, as I said, never trust a programmer to do the right thing, so we're going to check it. And just in case the assumptions are not true, we're just going to stop dead in our tracks.

    All right. Then we're going to set low to-- low and high, and we're going to perform exactly the process I talked about. And along the way, I'm keeping track of how many iterations, at the end I'll print how many iterations I took, before I return the final guess.

    All right, let's test it. So one of the things I want you to observe here, is that instead of sitting there and typing away a bunch of test cases, I took the trouble to write a function, called test bi in this case. All right, so what that's doing, is it's taking the things I would normally type, and putting them in a function, which I can then call.

    Why is that better than typing them? Why was it worth creating a function to do this? Pardon?

    STUDENT:: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: Then I can I can use it again and again and again. Exactly.

    By putting it in a function, if I find a bug and I change my program, I can just run the function again. The beauty of this is, it keeps me from getting lazy, and not only testing my program and the thing that found the bug, but in all the things that used to work.

    We'll talk more about this later, but it often happens that when you change your program to solve one problem, you break it, and things that used to work don't work. And so what you want to do, and again we'll come back to this later in the term, is something called regression testing. This has nothing to do with linear regression. And that's basically trying to make sure our program has not regressed, as to say, gone backwards in how well it works. And so we always test it on everything.

    All right? So I've created this function, let's give it a shot and see what happens. We'll run test bi. Whoops!

    All right, well let's look at our answers. I first tested it on the square root of 4, and in one iteration it found 2. I like that answer. I then tested it on the square root of 9, and as I mentioned last time, I didn't find 3. I was not crushed. You know, I was not really disappointed, it found something close enough to 3 that I'm happy.

    All right. I tried it on 2, I surely didn't expect a precise and exact answer to that, but I got something, and if you square this, you'll find the answer kept pretty darn close to 2.

    I then tried it on 0.25 One quarter. And what happened was not what I wanted. As you'll see, it crashed.

    It didn't really crash, it found an assert statement. So if you look at the bottom of the function, you'll see that, in fact, I checked for that. I assert the counter is less than or equal to 0. I'm checking that I didn't leave my program because I didn't find an answer. Well, this is a good thing, it's better than my program running forever, but it's a bad thing because I don't have it the square root of 0.25.

    What went wrong here? Well, let's think about it for a second. You look like-- someone looks like they're dying to give an answer. No, you just scratching your head? All right.

    Remember, I said when we do a bisection method, we're assuming the answer lies somewhere between the lower bound and the upper bound. Well, what is the square root of a quarter? It is a half.

    Well, what-- where did I tell my program to look for an answer? Between and x. So the problem was, the answer was over here somewhere, and so I'm never going to find it cleverly searching in this region, right? So the basic idea was fine, but I failed to satisfy the initial condition that the answer had to be between the lower bound and the upper bound. Right?

    And why did I do that? Because I forgot what happens when you look at fractions. So what should I do? Actually I lied, by the way, when I said the answer was over there. Where was the answer? Somebody?

    It was over here. Because the square root of a quarter is not smaller than a quarter, it's bigger than a quarter. Right? A half is strictly greater than a quarter.

    So it wasn't on the region. So how-- what's the fix? Should be a pretty simple fix, in fact we should be able to do it on the fly, here. What should I change? Do I need to change the lower bound? Is the square root ever going to be less than 0? Doesn't need to be, so, what should I do about the upper bound here? Oh, I could cheat and make, OK, the upper bound a half, but that wouldn't be very honest.

    What would be a good thing to do here? Pardon? I could square x, but maybe I should just do something pretty simple here. Suppose-- whoops. Suppose I make it the max of x and 1. Then if I'm looking for the square root of something less than 1, I know it will be in my region, right?

    All right, let's save this, and run it and see what happens. Sure enough, it worked and, did we get-- we got the right answer, 0.5 All right? And by the way, I checked all of my previous ones, and they work too.

    All right. Any questions about bisection search?

    One of the things I want you to notice here is the number iterations is certainly not constant. Yeah, when I will looked at 4, it was a nice number like 1, 9 looked like it took me 18, 2 took me 14, if we try some big numbers it might take even longer. These numbers are small, but sometimes when we look at really harder problems, we got ourselves in a position where we do care about the number of iterations, and we care about something called the speed of convergence.

    Bisection methods were known to the ancient Greeks, and it is believed by many, even to the Babylonians. And as I mentioned last time, this was the state of the art until the 17th century. At which point, things got better. So, let's think about it, and let's think about what we're actually doing when we solve this.

    When we look for something like the square root of x, what we're really doing, is solving an equation. We're looking at the equation f of guess equals the guess squared minus x. Right, that's what that is equal to, and we're trying to solve the equation that f of guess equals 0. Looking for the root of this equation.

    So if we looked at it pictorially, what we've got here is, we're looking at f of x, I've plotted it here, and we're asking where it crosses the x axis. Sorry for the overloading of the word x.

    And I'm looking here at 16. Square root of 16, and my plot basically shows it crosses at 4 and-- well, I think that's minus 4. The perspective is tricky-- and so we're trying to find the roots.

    Now Isaac Newton and/or Joseph Raphson figured out how to do this kind of thing for all differentiable functions. Don't worry about what that means.

    The basic idea is, you take a guess, and you -- whoops -- and you find the tangent of that guess.

    So let's say I guessed 3. I look for the tangent of the curve at 3. All right, so I've got the tangent, and then my next guess is going to be where the tangent crosses the x axis. So instead of dividing it in half, I'm using a different method to find the next guess.

    The utility of this relies upon the observation that, most of the time-- and I want to emphasize this, most of the time, that implies not all of the time-- the tangent line is a good approximation to the curve for values near the solution. And therefore, the x intercept of the tangent will be closer to the right answer than the current guess.

    Is that always true, by the way? Show me a place where that's not true, where the tangent line will be really bad. Yeah. Suppose I choose it right down there, I guess 0. Well, the tangent there will not even have an x intercept. So I'm really going to be dead in the water.

    This is the sort of thing that people who do numerical programming worry about all the time. And there are a lot of a little tricks they use to deal with that, they'll perturb it a little bit, things like that. You should not, at this point, be worrying about those things.

    This method, interestingly enough, is actually the method used in most hand calculators. So if you've got a calculator that has a square root button, it's actually in the calculator running Newton's method. Now I know you thought it was going to do that thing you learned in high school for finding square roots, which I never could quite understand, but no. It uses Newton's method to do it.

    So how do we find the intercept of the tangent, the x intercept? Well this is where derivatives come in. What we know is that the slope of the tangent is given by the first derivative of the function f at the point of the guess. So the slope of the guess is the first derivative. Right. Which dy over dx. Change in y divided by change in x.

    So we can use some algebra, which I won't go through here, and what we would find is that for square root, the derivative, written f prime of the i'th guess is equal to two times the i'th guess. Well, should have left myself a little more room, sorry about that.

    All right? You could work this out. Right? The derivative of the square root is not a complicated thing. Therefore, and here's the key thing we need to keep in mind, we'll know that we can choose guess i plus 1 to be equal to the old guess, guess i, minus whatever the value is of the new guess-- of the old rather, the old guess-- divided by twice the old guess.

    All right, again this is straightforward kind of algebraic manipulations to get here. So let's look at an example.

    Suppose we start looking for the square root of 16 with the guess 3. What's the value of the function f of 3? Well, it's going to be, we looked at our function there, guess squared, 3 times 3 is 9 I think, minus 16, that's what x is in this case, which equals minus 7.

    That being the case, what's my next guess? Well I start with my old guess, 3, minus f of my old guess, which is minus 7, divided by twice my old guess, which is 6, minus the minus, and I get as my new guess 4.1666 or thereabouts. So you can see I've missed, but I am closer. And then I would reiterate this process using that as guess i, and do it again.

    One way to think about this intuitively, if the derivative is very large, the function is changing quickly, and therefore we want to take small steps. All right. If the derivative is small, it's not changing, maybe want to take a larger step, but let's not worry about that, all right?

    Does this method work all the time? Well, we already saw no, if my initial guess is zero, I don't get anywhere. In fact, my program crashes because I end up trying to divide by zero, a really bad thing. Hint: if you implement Newton's method, do not make your first guess zero.

    All right, so let's look at the code for that. All right so-- yeah, how do I get to the code for that? That's interesting.

    All right. So we have that square root NR. NR for Newton Raphson. First thing I want you to observe is its specification is identical to the specification of square root bi. What that's telling me is that if you're a user of this, you don't care how it's implemented, you care what it does. And therefore, it's fine that the specifications are identical, in fact it's a good thing, so that means if someday Professor Grimson invents something that's better than Newton Raphson, we can all re-implement our square root functions and none of the programs that use it will have to change, as long as the specification is the same.

    All right, so, not much to see about this. As I said, the specifications is the same, same assertions, and the-- it's basically the same program as the one we were just looking at, but I'm starting with a different guess, in this case x over 2, well I'm going to, couple of different guesses we can start with, we can experiment with different guesses and see whether we get the same answer, and in fact, if we did, we would see we didn't get this, we got different answers, but correct answers. Actually now, we'll just comment that out. I'm going to compute the difference, just as I did on the board, and off we'll go.

    All right. Now, let's try and compare these things. And what we're going to look at is another procedure, you have the code for these things on your handout so we won't worry, don't need to show you the code, but let's look at how we're going to test it.

    I'm doing a little trick by the way, I'm using raw input in my function here, as a just a way to stop the display. This way I can torture you between tests by asking you questions. Making it stop.

    All right, so, we'll try some things. We'll see what it does. Starting with that, well, let's look at some of the things it will do. Yeah, I'll save it.. It's a little bit annoying, but it makes the font bigger.

    All right, so we've tested it, and we haven't tested it yet, we have tested it but, we haven't seen it, well, you know what I'm going to do? I'm going to tort-- I'm going to make the font smaller so we can see more. Sorry about this. Those of you in the back, feel free to move forward.

    All right. So we've got it, now let's test it. So we're going to do here, we're going to run compare methods. Well we're seeing this famous computers are no damn good.

    All right. So we're going to try it on 2, and at least we'll notice for 2, that the bisection method took eight iterations, the Newton Raphson only took three, so it was more efficient. They came up with slightly different answers, but both answers are within .01 which is what I gave it here for epsilon, so we're OK. So even though they have different answers, they both satisfy the same specification, so we have no problem. All right?

    Try it again, just for fun. I gave it here a different epsilon, and you'll note, we get different answers. Again, that's OK. Notice here, when I asked for a more precise answer, bisection took a lot more iterations, but Newton Raphson took only one extra iteration to get that extra precision in the answer. So we're sort of getting the notion that Newton Raphson maybe is considerably better on harder problems. Which, by the way, it is.

    We'll make it an even harder problem, by making it looking an even smaller epsilon, and again, what you'll see is, Newton Raphson just crept up by one, didn't take it long, and got the better answer, where bisection gets worse and worse. So as you can see, as we escalate the problem difficulty, the difference between the good method and the not quite as good method gets bigger and bigger and bigger. That's an important observation, and as we get to the part of the course, we talk about computational complexity, you'll see that what we really care about is not how efficient the program is on easy problems, but how efficient it is on hard problems.

    All right. Look at another example. All right, here I gave it a big number, 123456789. And again, I don't want to bore you, but you can see what's going on here with this trend.

    So here's an interesting question. You may notice that it's always printing out the same number of digits. Why should this be? If you look at it here, what's going on? Something very peculiar is happening here. We're looking at it, and we're getting some funny answers.

    This gets back to what I talked about before, about some of the precision of floating point numbers. And the thing I'm trying to drive home to you here is perhaps the most important lesson we'll talk about all semester. Which is, answers can be wrong.

    People tend to think, because the computer says it's so, it must be so. That the computer is-- speaks for God. And therefore it's infallible. Maybe it speaks for the Pope. It speaks for something that's infallible. But in fact, it is not. And so, something I find myself repeating over and over again to myself, to my graduate students, is, when you get an answer from the computer, always ask yourself, why do I believe it? Do I think it's the right answer? Because it isn't necessarily.

    So if we look at what we've got here, we've got something rather peculiar, right? What's peculiar about what this computer is now printing for us? Why should I be really suspicious about what I see in the screen here?

    STUDENT: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: Well, not only is it different, it's really different, right? If it were just a little bit different, I could say, all right, I have a different approximation. But when it's this different, something is wrong. Right?

    We'll, later in the term when we get to more detailed numerical things, look at what's wrong. You can run into issues of things like overflow, underflow, with floating point numbers, and when you see a whole bunches of ones, it's particularly a good time to be suspicious. Anyway the only point I'm making here is, paranoia is a healthy human trait.

    All right. We can look at some other things which will work better. And we'll now move on. OK.

    So we've looked at how to solve square root we've, looked at two problems, I've tried to instill in you this sense of paranoia which is so valuable, and now we're going to pull back and return to something much simpler than numbers, and that's Python. All right? Numbers are hard. That's why we teach whole semesters worth of courses in number theory. Python it's easy, which is why we do it in about four weeks.

    All right. I want to return to some non-scalar types. So we've been looking, the last couple of lectures, at floating point numbers and integers. We've looked so far really at two non-scalar types. And those were tuples written with parentheses, and strings.

    The key thing about both of them is that they were immutable. And I responded to at least one email about this issue, someone quite correctly said tuple are immutable, how can I change one? My answer is, you can't change one, but you can create a new one that is almost like the old one but different in a little bit.

    Well now we're going to talk about some mutable types. Things you can change. And we're going to start with one that you, many of you, have already bumped into, perhaps by accident, which are lists. Lists differ from strings in two ways; one way is that it's mutable, the other way is that the values need not be characters. They can be numbers, they can be characters, they can be strings, they can even be other lists.

    So let's look at some examples here. What we'll do, is we'll work on two boards at once. So I could write a statement like, techs, a variable, is equal to the list, written with the square brace, not a parenthesis, MIT, Cal Tech, closed brace. What that basically does, is it takes the variable techs, and it now makes it point to a list with two items in it. One is the string MIT and one is the string Cal Tech.

    So let's look at it. And we'll now run another little test program, show lists, and I printed it, and it prints the list MIT, Cal Tech. Now suppose I introduce a new variable, we'll call it ivys, and we say that is equal to the list Harvard, Yale, Brown. Three of the ivy league colleges. What that does is, I have a new variable, ivys, and it's now pointing to another, what we call object, in Python and Java, and many other languages, think of these things that are sitting there in memory somewhere as objects. And I won't write it all out, I'll just write it's got Harvard as one in it, and then it's got Yale, and then it's got Brown. And I can now print ivys. And it sure enough prints what we expected it to print.

    Now, let's say I have univs, for universities, equals the empty list. That would create something over here called univs, another variable, and it will point to the list, an object that contains nothing in it.

    This is not the same as none. It's it does have a value, it just happens to be the list that has nothing in it.

    And the next thing I'm going to write is univs dot append tex. What is this going to do? It's going to take this list and add to it something else.

    Let's look at the code. I'm going to print it, and let's see what it prints. It's kind of interesting. Whoops. Why did it do that? That's not what I expected. It's going to print that. The reason it printed that is I accidentally had my finger on the control key, which said print the last thing you had.

    Why does it start with square braced square brace? I take it-- yes, go ahead.

    STUDENT: So you're adding a list to a list?

    PROFESSOR JOHN GUTTAG: So I'm adding a list to a list. What have I-- what I've appended to the empty list is not the elements MIT and Cal Tech but the list that contains those elements.

    So I've appended this whole object. Since that object is itself a list, what I get is a list of lists.

    Now I should mention this notation here append is what is in Python called a method. Now we'll hear lots more about methods when we get to classes and inheritance, but really, a method is just a fancy word for a function with different syntax. Think of this as a function that takes two arguments, the first of which is univs and the second of which is techs. And this is just a different syntax for writing that function call.

    Later in the term, we'll see why we have this syntax and why it wasn't just a totally arbitrary brain-dead decision by the designers of Python, and many languages before Python, but in fact is a pretty sensible thing. But for now, think of this as just another way to write a function call. All right, people with me so far?

    Now let's say we wanted as the next thing we'll do, is we're going to append the ivys to univ. Stick another list on it. All right. So we'll do that, and now we get MIT, Cal Tech, followed by that list followed by the list Harvard, Yale, Brown. So now we have a list containing two lists.

    What are we going to try next? Well just to see what we know what we're doing, let's look at this code here. I've written a little for loop, which is going to iterate over all of the elements in the list. So remember, before we wrote things like for i in range 10, which iterated over a list or tuple of numbers, here you can iterate over any list, and so we're going to just going to take the list called univs and iterate over it.

    So the first thing we'll do is, we'll print the element, in this case it will be a list, right? Because it's a list with two lists in it. Then the next thing in the loop, we're going to enter a nested loop, and say for every college in the list e, we're going to print the name of the college. So now if we look what we get-- do you not want to try and execute that?-- it'll first print the list containing MIT and Cal Tech, and then separately the strings MIT and Cal Tech, and then the list containing Harvard, Yale, and Brown, and then the strings Harvard, Yale, and Brown.

    So we're beginning to see this is a pretty powerful notion, these lists, and that we can do a lot of interesting things with them. Suppose I don't want all of this structure, and I want to do what's called flattening the list. Well I can do that by, instead of using the method append, use the concatenation operator. So I can concatenate techs plus ivys and assign that result to univs, and then when I print it you'll notice I just get a list of five strings.

    So plus and append do very different things. Append sticks the list on the end of the list, append flattens it, one level of course. If I had lists of lists of lists, then it would only take out the first level of it. OK, very quiet here. Any questions about any of this? All right. Because we're about to get to the hard part Sigh. All right.

    Let's look at the-- well, suppose I want to, quite understandably, eliminate Harvard. All right, I then get down here, where I'm going to remove it. So this is again another method, this is remove, takes two arguments, the first is ivys, the second is the string Harvard. It's going to search through the list until the first time it finds Harvard and then it's going to yank it away. So what happened here? Did I jump to the wrong place?

    STUDENT: You hit two returns.

    PROFESSOR JOHN GUTTAG: I hit two returns. Pardon?

    STUDENT: You hit two returns. One was at

    STUDENT: Pardo

    PROFESSOR JOHN GUTTAG: This one.

    STUDENT: No, up one.

    PROFESSOR JOHN GUTTAG: Up one.

    STUDENT: Right.

    PROFESSOR JOHN GUTTAG: But why is Harvard there?

    STUDENT: I'm sorry, I didn't write it down.

    PROFESSOR JOHN GUTTAG: Let's look at it again. All right, it's time to interrupt the world, and we'll just type into the shell. Let's see what we get here. All right, so let's just see what we got, we'll print univs. Nope, not defined.

    All right, well let's do a list equals, and we'll put some interesting things in it, we'll put a number in it, because we can put a number, we'll put MIT in it, because we can put strings, we'll put another number in it, 3.3, because we can put floating points, we can put all sorts of things in this list. We can put a list in the list again, as we've seen before. So let's put the list containing the string a, and I'll print out, so now we see something pretty interesting about a list, that we can mix up all sorts of things in it, and that's OK.

    You'll notice I have the string with the number 1, a string with MIT, and then it just a plain old number, not a string, again it didn't quite give me 3.3 for reasons we've talked before, and now it in the list a.

    So, suppose I want to remove something. What should we try and remove from this list? Anybody want to vote? Pardon? All right, someone wants to remove MIT. Sad but true. Now what do we get if we print l? MIT is gone.

    How do I talk about the different pieces of l? Well I can do this. l sub 0-- whoops-- will give me the first element of the list, just as we could do with strings, and I can look at l sub minus 1 to get the last element of the list, so I can do all the strings, all the things that I could do with strings.

    It's extremely powerful, but what we haven't seen yet is mutation. Well, we have seen mutation, right? Because notice that what remove did, it was it actually changed the list. Didn't create a new list. The old l is still there, but it's different than it used to be. So this is very different from what we did with slicing, where we got a new copy of something. Here we took the old one and we just changed it.

    On Thursday, we'll look at why that allows you to do lots of things more conveniently than you can do without mutation.
  • 12
    Assignment 4: Simulating a retirement fund
    6 pages
  • 13
    MIT Lecture 7: Lists and Mutability, Dictionaries, Introduction to Efficiency
    46:22
    Lists and mutability, dictionaries, pseudocode, introduction to efficiency. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR JOHN GUTTAG: OK. I finished up last time talking about lists. And I pointed out that lists are mutable, showed you some examples of mutation. We can look at it here; we looked at append, which added things to lists, we looked at delete, deleting things from a list. You can also assign to a list, or to an element of a list. So ivy sub 1, for example, could be assigned minus 15, and that will actually mutate the list.

    So heretofore, when we wrote assignment, what we always meant, was changing the binding of a variable to a different object. Here, we are overloading the notation to say, no, no, ivys is still bound to the same object, but an element of ivys is bound to a different object.

    If you think about it, that makes sense, because when we have a list, what a list is, is a sequence of objects. And what this says is, is the object named by the expression ivys sub 1, is now bound to the object, if you will, named by the constant minus 15. So we can watch this run here. Idle can-- that's exciting. I hadn't expected that answer. All right, your question.

    STUDENT: [INAUDIBLE] four elements to ivys, and you tell it to change the fifth element of ivys to negative 15, will it add it or [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: Well, I'll tell you how ol-- let's answer that the easy way. We'll start up a shell and we'll try it. All right, we'll just get out of what we were doing here.

    And so, we now have some things, so we, for example, have ivys, I can print ivys , and it's only got three elements but your question probably is just as good for adding the fourth as adding the fifth, so what would happen if we say ivys sub 3-- because that of course is the fourth element, right? Let's find out.

    OK. Because what that does is, it's changing the binding of the name ivys, in this case, sub 1. What it looked at here, with the name ivys sub 3, and said that name doesn't-- isn't bound, right? That isn't there. So it couldn't do it, so instead that's what append is for, is to stick things on to the end of the list. But a very good question.

    So we can see what we did here, and, but of course I can now, if I choose, say something like, ivys sub 1 is assigned minus 15, and now if I print ivys, there it is.

    And again, this points out something I wanted to me-- I mentioned last time, list can be heterogeneous, in the sense that the elements can be multiple different types. As you see here, some of the elements are strings and some of the elements are integers.

    Let's look at another example. Let's suppose, we'll start with the list, I'll call it l 1. This, by the way, is a really bad thing I just did. What was-- what's really bad about calling a list l 1?

    STUDENT: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: Is it l 1, or is it 11, or is it l l? It's a bad habit to get into when you write programs, so I never use lowercase L except when I'm spelling the word where it's obvious, because otherwise I get all sorts of crazy things going on.

    All right, so let's make it the list 123. All right? Now, I'll say L 2 equals L 1. Now I'll print L 2. Kind of what you'd guess, but here's the interesting question: if I say L 1 is assigned 0, L 1 sub is assigned 4, I'll print L 1.

    That's what you expect, but what's going to happen if I print L 2? 423 as well, and that's because what happened is I had this model, which we looked at last time, where I had the list L 1, which was bound to an object, and then the assignment L 2 gets L 1, bound the name L 2 to the same object, so when I mutated this object, which I reached through the name L 1 to make that 4, since this name was bound to the same object, when I print it, I got 423.

    So that's the key thing to-- to realize; that what the assignment did was have two separate paths to the same object. So I could get to that object either through this path or through that path, it didn't matter which path I use to modify it, I would see it when I looked at the other. Yes.

    STUDENT: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: So the question, if I said a is assigned 2, b is assigned a, and then a is assigned 3. Is that your question?

    So the question is, a is assigned 1, b is assigned a, a is assigned 2, and then if I print b, I'll get 1. Because these are not mutable, this is going to be assigned to an object in the store, so we'll draw the picture over here, that we had initially a is bound to an object with 1 in it, and then b got bound to the same object, but then when I did the assignment, what that did was it broke this connection, and now had a assigned to a different object, with the number, in this case, 2 in it. Whereas the list assignment you see here did not rebind the object l 1, it changed this. OK?

    Now formally I could have had this pointing off to another object containing 4, but that just seemed excessive, right? But you see the difference. Great question, and a very important thing to understand, and that's why I'm belaboring this point, since this is where people tend to get pretty confused, and this is why mutation is very important to understand. Yeah.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR JOHN GUTTAG: I'm just assuming it'll be a great question.

    STUDENT: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: Exactly. So if-- very good question-- so, for example, we can just do it here.

    The question was, suppose I now type L 1 equals the empty list. I can print L 1, and I can print L 2, because again, that's analogous to this example, where I just swung the binding of the identifier. So this is important, it's a little bit subtle, but if you don't really understand this deeply, you'll find yourself getting confused a lot. All right?

    OK. Let me move on, and I want to talk about one more type. By the way, if you look at the handout from last time, you'll see that there's some other examples of mutation, including a function that does mutation. It's kind of interesting, but I don't think we need-- think we've probably done enough here that I hope it now make sense.

    That one type I want to talk about still is dictionaries. Like lists, dictionaries are mutable, like lists, they can be heterogeneous, but unlike lists, they're not ordered. The elements in them don't have an order, and furthermore, we have generalized the indexing. So lists and strings, we can only get at elements by numbers, by integers, really.

    Here what we use is, think of every element of the dictionary as a key value pair, where the keys are used as the indices. So we can have an example, let's look at it. So, if you look at the function show dics here, you'll see I've declared a variable called e to f, ah, think of that as English to French, and I've defined a dictionary to do translations. And so, we see that the string one corresponds the-- the key one corresponds to the value un the key soccer corresponds to the French word football, et cetera. It's kind of bizarre, but the French call soccer football. And then I can index in it. So if I print e to f of soccer, it will print the string football.

    So you can imagine that this is a very powerful mechanism. So let's look what happens when I run-- start to run this.

    All right. So, it says not defined-- and why did it say not defined, there's an interesting question. Let's just make sure we get this right, and we start the show up again-- All right, so, I run it, and sure enough, it shows football. What happens if I go e to f of 0? I get a key error. Because, remember, these things are not ordered. There is no 0th element. is not a key of this particular object.

    Now I could have made a key, keys don't have to be strings, but as it happened, I didn't. So let's comment that out, so we don't get stuck again. Where we were before, I've printed it here, you might be a little surprised of the order. Why is soccer first? Because the order of this doesn't matter. That's why it's using set braces, so don't worry about that.

    The next thing I'm doing is-- so that's that, and then-- I'm now going to create another one, n to s, for numbers to strings, where my keys are numbers, in this case the number 1 corresponds to the word one, and interestingly enough, I'm also going to have the word one corresponding to the number 1. I can use anything I want for keys, I can use anything I want for values. And now if we look at this, we see, I can get this.

    All right. So these are extremely valuable. I can do lots of things with these, and you'll see that as we get to future assignments, we'll make heavy use of dictionaries. Yeah. Question.

    STUDENT: [INAUDIBLE]

    PROFESSOR JOHN GUTTAG: You can, but you don't know what order you'll get them in. What you can do is you can iterate keys, which gives you the keys in the dictionary, and then you can choose them, but there's no guarantee in the order in which you get keys.

    Now you might wonder, why do we have dictionaries? It would be pretty easy to implement them with lists, because you could have a list where each element of the list was a key value pair, and if I wanted to find the value corresponding to a key, I could say for e in the list, if the first element of e is the key, then I get the value, otherwise I look at the next element in the list.

    So adding dictionaries, as Professor Grimson said with so many other things, doesn't give you any more computational power. It gives you a lot of expressive convenience, you can write the programs much more cleanly, but most importantly, it's fast.

    Because if you did what I suggested with the list, the time to look up the key would be linear in the length of the list. You'd have to look at each element until you found the key.

    Dictionaries are implemented using a magic technique called hashing, which we'll look at a little bit later in the term, which allows us to retrieve keys in constant time. So it doesn't matter how big the dictionary is, you can instantaneously retrieve the value associated with the key. Extremely powerful.

    Not in the next problems set but in the problem set after that, we'll be exploiting that facility of dictionaries. All right. Any questions about this? If not, I will turn the podium over to Professor Grimson.

    PROFESSOR ERIC GRIMSON: I've stolen it. This is like tag team wrestling, right? Professor Guttag has you on the ropes, I get to finish you off. Try this again.

    OK. We wanted to finish up that section, we're now going to start on a new section, and I want to try and do one and a half things in the remaining time. I'm going to introduce one topic that we're going to deal with fairly quickly, and then we tackle the second topic, it's going to start today, and we're going to carry on. So let me tell the two things I want to do.

    I want to talk a little bit about how you use the things we've been building in terms of functions to help you structure and organize your code. It's a valuable tool that you want to have as a programmer.

    And then we're going to turn to the question of efficiency. How do we measure efficiency of algorithms? Which is going to be a really important thing that we want to deal with, and we'll start it today, it's undoubtedly going to take us a couple more lectures to finish it off.

    Right, so how do you use the idea of functions to organize code? We've been doing it implicitly, ever since we introduced functions. I want to make it a little more explicit, and I want to show you a tool for doing that. And I think the easy way to do is-- is to do it with an example.

    So let's take a really simple example. I want to compute the length of the hypotenuse of a right triangle. And yeah, I know you know how to do it, but let's think about what might happen if I wanted to do that.

    And in particular, if I think about that problem-- actually I want to do this-- if I think about that problem, I'm going to write a little piece of pseudo code. Just to think about how I would break that problem up.

    Pseudo code. Now, you're all linguistic majors, pseudo means false, this sounds like code that ain't going to run, and that's not the intent of the term. When I say pseudo code, what I mean is, I'm going to write a description of the steps, but not in a particular programming language. I'm going to simply write a description of what do I want to do.

    So if I were to solve this problem, here's the way I would do it. I would say, first thing I want to do, is I want to input a value for the base as a float. Need to get the base in. Second thing I want to do, I need to get the height, so I'm going to input a value for the height, also as a float, a floating point.

    OK. I get the two values in, what do I need to do, well, you sort of know that, right? I want to then do, I need to find the square root-- b squared plus h squared, right? The base plus the height, that's the thing I want for the hypotenuse-- and I'm going to save that as a float in hyp, for hypotenuse. And then finally I need to print something out, using the value in hyp.

    OK. Whoop-dee-doo, right? Come on. We know how to do this. But notice what I did.

    First of all, I've used the notion of modularity. I've listed a sequence of modules, the things that I want to do.

    Second thing to notice, is that little piece of pseudo code is telling me things about values. I need to have a float. I need to have another float here, it's giving me some information.

    Third thing to notice is, there's a flow of control. The order which these things are going to happen.

    And the fourth thing to notice is, I've used abstraction. I've said nothing about how I'm going to make square root. I'm using it as an abstraction, saying I'm going to have square root from somewhere, maybe I'll build it myself, maybe somebody gives it to me as part of a library, so I'm burying the details inside of it.

    I know this is a simple example, but when you mature as a programmer, one of the first things you should do when you sit down to tackle some problem is write something like this pseudo code. I know Professor Guttag does it all the time.

    I know, for a lot of you, it's like, OK, I got a heavy problem. Let's see, def Foobar open paren, a bunch of parameters.

    Wrong way to start. Start by thinking about what are the sequences.

    This also, by the way, in some sense, gives me the beginnings of my comments for what the structure of my code is going to be.

    OK. If we do that, if you look at the handout then, I can now start implementing this. I wanted to show you that, so, first thing I'm going to do is say, all right, I know I'm going to need square root in here, so I'm going to, in fact, import math.

    That's a little different from other import statements. This says I'm going to get the entire math library and bring it in so I can use it. And then, what's the first thing I wanted to do? I need to get a value for base as a float. Well OK, and that sounds like I'm going to need to do input of something, you can see that statement there, it's-- got the wrong glasses on but right there-- I'm going to do an input with a little message, and I'm going to store it in base.

    But here's where I'm going to practice a little bit of defensive programming. I can't rely on Professor Guttag if I give this-- if this code to him, I can't rely on him to type in a float. Actually I can, because he's a smart guy, but in general, I can't rely on the user--

    PROFESSOR JOHN GUTTAG: I wouldn't do it right to see if you did.

    PROFESSOR ERIC GRIMSON: Actually, he's right, you know. He would not do it, just to see if I'm doing it right. I can't rely on the user. I want to make sure I get a float in it, so how do I do that? Well, here's one nice little trick.

    First of all, having read in that value, I can check to see, is it of the right type? Now, this is not the nicest way to do it but it'll work. I can look at the type of the value of base and compare it to the type of an actual float and see, are they the same? Is this a real or a float?

    If it is, I'm done. How do I go back if it isn't? Well, I'm going to create a little infinite loop. Not normally a good idea. I set up a variable here, called input OK. Initially it's false, because I have no input. And then I run a loop in which I read something in, I check to see if it's the right type, if it is, I change that variable to say it's now the correct type, which means the next time through the loop, I'm going to say I'm all set and I'm going to bounce out.

    But if it is not, it's going to print out a message here saying, you screwed up, somewhat politely, and it's going to go back around. So it'll just cycle until I get something of the right type. Nice way of doing it.

    Right, what's the second thing I do? Well, I get the same sort of thing to read in the height, once I have that I'm going to take base squared plus height squared, and there's a form that we've just seen once before, and it's going to repeat it, that is math.SQRT and it says the following: it says, take from the math library the function called sqrt.

    OK. We're going to come back to this when we get to objects, it's basically picking up that object and it's applying that, putting that value into hype, and then just printing something out.

    And again, if I just run this, just to show that it's going to do the right thing, it says enter base, I'm obnoxious, it says oops, wasn't a float, so we'll be nice about it, and I enter a height, and it prints out what I expected. I just concatenated those strings together, by the way, at the end.

    All right. Notice what I did. OK. I went from this description, it gives me [UNINTELLIGIBLE] some information. I need to have a particular type. I made sure I had the particular type. I've used some abstraction to suppress some details here.

    Now if you look at that list, there is actually something I didn't seem to check, which is, I said I wanted a float stored in hyp.

    How do I know I've got a float in hyp? Well I'm relying on the contract, if you like, that the manufacturer of square root put together, which is, if I know I'm giving it two floats, which I do because I make sure they're floats, the contract, if you like, of square root says I'll give you back a float. So I can guarantee I've got something of the right type.

    OK. I know this is boring as whatever. But there's an important point here. Having now used this pseudo code to line things up, I can start putting some additional structure on this. And in particular, I'm sure you're looking at this going-- will look at it if we look at the right piece-- going, wait a minute. This chunk of code and this chunk of code, they're really doing the same thing.

    And this is something I want to use. If I look at those two pieces of computation, I can see a pattern there. It's an obvious pattern of what I'm doing. And in particular, I can then ask the following question, which is, what's different between those two pieces of code?

    And I suggest two things, right? One is, what's the thing I print out when I ask for the input? The second thing is, what do I print out if I actually don't get the right input in? And so the only two differences are, right there, and there versus here and here.

    So this is a good place to think about, OK, let me capture that. Let me write a function, in fact the literal thing I would do is to say, identify the things that change, give each of them a variable name because I want to refer to them, and then write a function that captures the rest of that computation just with those variable names inside. And in fact, if you look down-- and I'm just going to highlight this portion, I'm not going to run it-- but if you look down here, that's exactly what that does.

    I happen to have it commented out, right? What does it do? It has height, it says, I've got two names of things: the request message and the error message. The body of that function looks exactly like the computation up above, except I'm simply using those in place of the specific message I had before. And then the only other difference is obviously, it's a function I need to return a value. So when I'm done, I'm going to give the value back out. All right?

    And that then let's me get to, basically, this code. Having done that, I simply call base with get float, I call height with get float, and do the rest of the work.

    All right. What's the point of doing this? Well, notice again. What have I done? I've captured a module inside of a function.

    And even though it's a simple little thing here, there's some a couple of really nice advantages to this. All right? First one is there's less code to read. It's easier to debug. I don't have as much to deal with. But the more important thing is, I've now separated out implementation from functionality, or implementation from use.

    What does that mean? It means anybody using that little function get float doesn't have to worry about what's inside of it. So for example, I decide I want to change the message I print out, I don't have to change the function, I just pass in a different parameter.

    Well if I-- you know, with [UNINTELLIGIBLE PHRASE sorry, let me say it differently. I don't need to worry about how checking is done, it's handled inside of that function. If I decide there's a better way to get input, and there is, then I can make it to change what I don't have to change the code that uses the input.

    So, if you like, I've built a separation between the user and the implementer. And that's exactly one of the reasons why I want to have the functions, because I've separated those out.

    Another way of saying it is, anything that uses get float doesn't care what the details are inside or shouldn't, and if I change that definition, I don't have to change anything elsewhere in my code, whereas if I just have the raw code in there, I have to go off and do it.

    Right, so the things we want you to take away from this are, get into the habit of using pseudo code when you sit down to start a problem, write out what are the steps. I will tell you that a good programmer, at least in my mind, may actually go back and modify the pseudo code as they realize they're missing things, but it's easier to do that when you're looking at a simple set of steps, than when you're in the middle of a pile of code.

    And get into the habit of using it to help you define what is the flow of control. What are the basic modules, what information needs to be passed between those modules in order to make the code work.

    OK. That was the short topic. I will come back to this some more and you're going to get lots of practice with this. What I want to do is to start talking about a different topic. Which is efficiency.

    And this is going to sound like a weird topic, we're going to see why it's of value in a second. I want to talk about efficiency, and we're going to, or at least I'm going to, at times also refer to this as orders of growth, for reasons that you'll see over the next few minutes.

    Now, efficiency is obviously an important consideration when you're designing code, although I have to admit, at least for me, I usually want to at least start initially with code that works, and then worry about how I might go back and come up with more efficient implementation. I like to have something I can rely on, but it is an important issue.

    And our goal over the next couple of lectures, is basically to give you a sense of this. So we're not going to turn you into an expert on computational efficiency. That's, there are whole courses on that, there's some great courses here on that, it takes some mathematical sophistication, we're going to push that off a little bit.

    But what we-- what we do want to do, is to give you some intuition about how to approach questions of efficiency. We want you to have a sense of why some programs complete almost before you're done typing it. Some programs run overnight. Some programs won't stop until I'm old and gray. Some programs won't stop until you're old and gray. And these are really different efficiencies, and we want to give you a sense of how do you reason about those different kinds of programs.

    And part of it is we want you to learn how to have a catalog, if you like, of different classes of algorithms, so that when you get a problem, you try and map it into an appropriate class, and use the leverage, if you like, of that class of algorithms.

    Now. It's a quick sidebar, I've got to say, I'm sure talking about efficiency to folks like you probably seems really strange. I mean, you grew up in an age when computers were blazingly fast, and have tons of memory, so why in the world do you care about efficiency?

    Some of us were not so lucky. So I'll admit, my first computer I program was a PDP6, only Professor Guttag even knows what PDP stands for, it was made by Digital Equipment Company, which does not exist anymore, is now long gone. It had, I know, this is old guy stories, but it had 160k of memory. Yeah. 160k. 160 kilobits of memory. I mean, your flash cards have more than that, right? It had a processor speed of one megahertz. It did a million operations per second.

    So let's think about it. This sucker, what's it got in there? That Air Mac, it's, see, it's got, its go-- my Air Mac, I don't know about John's, his is probably better, mine has 1.8 gigahertz speed. That's 1800 times faster. But the real one that blows me away is, it has 2 gig of memory inside of it. That's 12 thousand times more memory. Oh, and by the way? The PDP6, it was in a rack about this tall. From the floor, not from the table.

    All right, so you didn't grow up in the late 1800s like I did, you don't have to worry about this sort of stuff, right? But a point I'm trying to make is, it sounds like anymore computers have gotten so blazingly fast, why should you worry about it?

    Let me give you one other anecdote that I can't resist. This is the kind of thing you can use at cocktail parties to impress your friends from Harvard. OK. Imagine I have a little lamp, a little goose-- one of those little gooseneck lamps, I'd put it on the table here, I'd put the height about a f-- about a foot off the table. And if I was really good, I could hit, or time it so that when I hurt-- yeah, try again. When I turn this on switch on in the lamp, at exactly the same time, I'm going to hit a key on my computer and start it running.

    OK. In the length of time it takes for the light to get from that bulb to the table, this machine processes two operations. Oh come on, that's amazing. Two operations. You know, you can do the simple numbers, right? [UNINTELLIGIBLE PHRASE]

    Light travels basically a foot in a nanosecond. Simple rule of thumb. Now, the nanosecond is what, 10 to the minus 9 seconds. This thing does 2 gig worth of operations. A gig is 10 to the 9, so it does two operations in the length of time it takes light to get from one foot off the table down to the table. That's amazing.

    So why in the world do you care about efficiency? Well the problem is that the problems grow faster than the computers speed up. I'll give you two examples.

    I happen to work in medical imaging. Actually, so does Professor Guttag. In my in my area of research, it's common for us to want to process about 100 images a second in order to get real time display. Each image has about a million elements in it. I've got to process about a half a gig of data a second in order to get anything out of it.

    Second example. Maybe one that'll hit a little more home to you. I'm sure you all use Google, I'm sure it's a verb in your vocabulary, right? Now, Google processes-- ten million? Ten billion pages? John? I think ten billion was the last number I heard. Does that sound about right?

    PROFESSOR JOHN GUTTAG: I think it might actually be more by now.

    PROFESSOR ERIC GRIMSON: Maybe more by now. But let's, for the sake of argument, ten billion pages. Imagine you want to search through Google to find a particular page. You want to do it in a second. And you're going to just do it the brute force way, assuming you could even reach all of those pages in that time.

    Well, if you're going to do that, you've got to be able to find what you're looking for in a page in two steps. Where a step is a comparison or an arithmetic operation. Ain't going to happen, right? You just can't do it.

    So again, part of the point here is that things grow-- or to rephrase it, interesting things grow at an incredible rate. And as a consequence, brute force methods are typically not going to work.

    OK. So that then leads to the question about what should we do about this? And probably the obvious thing you'll think about is, we'll come up with a clever algorithm. And I want to disabuse you of that notion. It's a great idea if you can do it,

    The guy who-- I think I'm going to say this right, John, right? Sanjay? Ghemawat?-- with a guy who was a graduate of our department, who is the heart and soul behind Google's really fast search, is an incredibly smart guy, and he did come up with a really clever algorithm about how you structure that search, in order to make it happen. And he probably made a lot of money along the way. So if you have a great idea, you know, talk to a good patent attorney and get it locked away.

    But in general, it's hard to come up with the really clever algorithm. What you're much better at doing is saying how do I take the problem I've got and map it into a class of algorithms about which I know and use the efficiencies of those to try and figure out how to make it work.

    So what we want to do, is, I guess another way of saying it is, efficiency is really about choice of algorithm. And we want to help you learn how to map a problem into a class of algorithms of some efficiency. That's our goal.

    OK. So to do this, we need a little more abstract way of talking about efficiency, and so, the question is, how do we think about efficiency? Typically there's two things we want to measure. Space and time. Sounds like an astrophysics course, right?

    Now, space usually we-- ach, try it again. When we talk about space, what we usually refer to is, how much computer memory does it take to complete a computation of a particular size?

    So let me write that down, it's how much memory do I need to complete a computation. And by that, I mean, not how much memory do I need to store the size of the input, it's really how much internal memory do I use up as I go through the computation? I've got some internal variables I have to store, what kinds of things do I have to keep track of?

    You're going to see the arguments about space if you take some of the courses that follow on, and again, some nice courses about that. For this course, we're not going to worry about space that much. What we're really going to focus on is time.

    OK. So we're going to focus here. And the obvious question I could start with is, and suppose I ask you, how long does the algorithm implemented by this program take to run? How might I answer that question? Any thoughts? Yeah.

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Ah, you're jumping ahead of me, great. The answer was, find a mathematical expression depending on the number of inputs. It was exactly where I want to go. Thank you.

    I was hoping for a simpler answer, which is, just run it. Which is, yeah I know, seems like a dumb thing to say, right?

    One of the things you could imagine is just try it on and input, see how long it takes. You're all cleverer than that, but I want to point out why that's not a great idea.

    First of all, that depends on which input I've picked. All right? Obviously the algorithm is likely to depend on the size of the input, so this is not a great idea.

    Second one is, it depends on which machine I'm running on. If I'm using a PDP6, it's going to take a whole lot longer than if I'm using an Air Mac. All right?

    Third one is, it may depend on which version of Python I'm running. Depends on how clever the implementer of Python was.

    Fourth one is, it may depend on which programming language I'm doing it in.

    So I could do it empirically, but I don't want to do that typically, it's just not a great way to get at it. And so in fact, what we want is exactly what the young lady said.

    I'm going to ask the following question, which is-- let me write it down-- what is the number of the basic steps needed as a function of the input size?

    That's the question we're going to try and address. If we can do this, this is good, because first of all, it removes any questions about what machine I'm running on, it's talking about fundamentally, how hard is this problem, and the second thing is, it is going to do it specifically in terms of the input. Which is one of the things that I was worried about.

    OK. So to do this, we're going to have to do a couple of things. All right, the first one is, what do we mean by input size?

    And unfortunately, this depends on the problem. It could be what's the size of the integer I pass in as an argument, if that's what I'm passing in. It could be, how long is the list, if I'm processing a list or a tuple It could be, how many bits are there in something. So it-- that is something where we have to simply be clear about specifying what we're using as input size. And we want to characterize it mathematically as some number, or some variable rather, the length of the list, the size of the integer, would be the thing we'd want to do.

    Second thing we've got to worry about is, what's a basic step? All right, if I bury a whole lot of computation inside of something, I can say, wow, this program, you know, runs in one step. Unfortunately, that one step calls the Oracle at Delphi and gets an answer back. Maybe not quite what you want.

    We're typically going to use as basic steps the built-in primitives that a machine comes with. Or another way of saying it is, we're going to use as the basic steps, those operations that run in constant time, so arithmetic operations. Comparisons. Memory access, and in fact one of the things we're going to do here, is we're going to assume a particular model, called a random access model, which basically says, we're going to assume that the length of time it takes me to get to any location in memory is constant.

    It's not true, by the way, of all programming languages. In fact, Professor Guttag already talked about that, in some languages lists take a time linear with the length to get to it. So we're to assume we can get to any piece of data, any instruction in constant time, and the second assumption we're going to make is that the basic primitive steps take constant time, same amount of time to compute. Again, not completely true, but it's a good model, so arithmetic operations, comparisons, things of that sort, we're all going to assume are basically in that in that particular model.

    OK. Having done that, then, there are three things that we're going to look at. As I said, what we want to do is, we want to count the number of basic steps it takes to compute a computation as a function of input size.

    And the question is, what do we want to count? Now, one possibility is to do best case. Over all possible inputs to this function, what's the fastest it runs? The fewest, so the minimum, if you like. It's nice, but not particularly helpful.

    The other obvious one to do would be worst case. Again, over all possible inputs to this function, what's the most number of steps it takes to do the computation?

    And the third possibility, is to do the expected case. The average. I'm going to think of it that way.

    In general, people focus on worst case. For a couple of reasons. In some ways, this would be nicer, do expected cases, it's going to tell you on average how much you expect to take, but it tends to be hard to compute, because to compute that, you have to know a distribution on input. How likely are all the inputs, are they all equally likely, or are they going to depend on other things? And that may depend on the user, so you can't kind of get at that.

    We're, as a consequence, going to focus on worst case. This is handy for a couple of reasons. One, it means there are no surprises. All right? If you run it, you have a sense of the upper bound, about how much time it's going to take to do this computation, so you're not going to get surprised by something showing up.

    The second one is, a lot of the time, the worst case is the one that happens. Professor Guttag used an example of looking in the dictionary for something. Now, imagine that dictionary actually has something that's a linear search to go through it, as opposed to the hashing he did, so it's a list, for example. If it's in there, you'll find it perhaps very quickly. If it's not there, you've got to go through everything to say it's not there. And so the worst case often is the one that shows up, especially in things like search.

    So, as a consequence, we're going to stick with the worst case analysis.

    Now, I've got two minutes left. I was going to start showing you some examples, but I think, rather than doing that, I'm going to stop here, I'm going to give you two minutes back of time, but I want to just point out to you that we are going to have fun next week, because I'm going to show you what in the world that has to do with efficiency.

    So with that, we'll see you next time.
  • 14
    MIT Lecture 8: Complexity: Log, Linear, Quadratic, Exponential Algorithms
    50:03
    Complexity; log, linear, quadratic, exponential algorithms. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR ERIC GRIMSON: Last time, we ended up, we sort of did this tag team thing, Professor Guttag did the first half, I did the second half of the lecture, and the second half of the lecture, we started talking about complexity. Efficiency. Orders of growth. And that's what we're going to spend today on, is talking about that topic. I'm going to use it to build over the next couple of lectures.

    I want to remind you that we were talking at a fairly high level about complexity. We're going to get down into the weeds in a second here. But the things we were trying to stress were that it's an important design decision, when you are coming up with a piece of code, as to what kind of efficiency your code has.

    And the second thing that we talked about is this idea that we want you to in fact learn how to relate a choice you make about a piece of code to what the efficiency is going to be. So in fact, over the next thirty or forty minutes, we're going to show you a set of examples of sort of canonical algorithms, and the different classes of complexity.

    Because one of the things that you want to do as a good designer is to basically map a new problem into a known domain. You want to take a new problem and say, what does this most look like? What is the class of algorithm that's-- that probably applies to this, and how do I pull something out of that, if you like, a briefcase of possible algorithms to solve?

    All right, having said that, let's do some examples. I'm going to show you a sequence of algorithms, they're mostly simple algorithms, that's OK. But I want you to take away from this how we reason about the complexity of these algorithms. And I'll remind you, we said we're going to mostly talk about time. We're going to be counting the number of basic steps it takes to solve the problem.

    So here's the first example I want to do. I'm going to write a function to compute integer power exponents. a to the b where b is an integer. And I'm going to do it only using multiplication and addition and some simple tests. All right? And yeah, I know it comes built in, that's OK, what we want to do is use it as an example to look at it.

    So I'm going to build something that's going to do iterative exponentiation. OK? And in fact, if you look at the code up here, and it's on your handout, the very first one, x 1, right here-- if I could ask you to look at it-- is a piece of code to do it. And I'm less interested in the code than how we're going to analyze it, but let's look at it for a second.

    All right, you can see that this little piece of code, it's got a loop in there, and what's it doing? It's basically cycling through the loop, multiplying by a each time. So first time through the loop, the answer is 1. Second time it-- sorry, as it enters the loop, at the time it enter-- exits, the answer is a. Next time through the loop it goes to a squared. Next time through the loop it goes to a cubed. And it's just gathering together the multiplications while counting down the exponent. And you can see it when we get down to the end test here, we're going to pop out of there and we're going to return the answer.

    I could run it, it'll do the right thing. What I want to think about though, is, how much time does this take? How many steps does it take for this function to run? Well, you can kind of look at it, right? The key part of that is that WHILE loop.

    And what are the steps I want to count? They're inside that loop-- I've got the wrong glasses so I'm going to have to squint-- and we've got one test which is a comparison, we've got another test which is a multiplication-- sorry, not a test, we've got another step which is a multiplication-- and another step that is a subtraction. So each time through the loop, I'm doing three steps. Three basic operations.

    How many times do I go through the loop? Somebody help me out. Hand up? Sorry. b times. You're right. Because I keep counting down each time around-- mostly I've got to unload this candy, which is driving me nuts, so-- thank you. b times. So I've got to go 3 b steps. All right, I've got to go through the loop b times, I've got three steps each time, and then when I pop out of the loop, I've got two more steps. All right, I've got the initiation of answer and the return of it. So I take 2 plus 3 b steps to go through this loop. OK. So if b is 300, it takes 902 steps. b is 3000, it takes 9002 steps. b is 30,000 you get the point, it takes 90,002 steps.

    OK. So the point here is, first of all, I can count these things, but the second thing you can see is, as the size of the problems get larger, that additive constant, that 2, really doesn't matter. All right? The difference between 90,000 steps and 90,002 steps, who cares about the 2, right? So, and typically, we're not going to worry about those additive constants.

    The second one is, this multiplicative constant here is 3, in some sense also isn't all that crucial. Does it really matter to you whether your code is going to take 300 years or 900 years to run? Problem is, how big is that number? So we're going to typically also not worry about the multiplicative constants. This factor here.

    What we really want to worry about is, as the size of the problem gets larger, how does this thing grow? How does the cost go up? And so what we're going to primarily talk about as a consequence is the rate of growth as the size of the problem grows. If it was, how much bigger does this get as I make the problem bigger? And what that really says is, that we're going to use this using something we're going to just call asymptotic notation-- I love spelling this word-- meaning, as in the limit as the size of the problem gets bigger, how do I characterize this growth? All right?

    You'll find out, if you go on to some of the other classes in course 6, there are a lot of different ways that you can measure this. The most common one, and the one we're going to use, is what's often called big Oh notation. This isn't big Oh as in, oh my God I'm shocked the markets are collapsing, This is called big Oh because we use the Greek letter, capital letter, omicron to represent it.

    And the way we're going to do this, or what this represents, let me write this carefully for you, big Oh notation is basically going to be an upper limit to the growth of a function as the input grow-- as the input gets large.

    Now we're going to see a bunch of examples, and I know those are words, let me give you an example. I would write f of x is in big Oh of n squared. And what does it say? It says that function, f of x, is bounded above, there's an upper limit on it, that this grows no faster than quadratic in n, n squared.

    OK. And first of all, you say, wait a minute, x and n? Well, one of the things we're going to see is x is the input to this particular problem, n is a measure of the size of x. And we're going to talk about how we come up with that. n measures the size of x.

    OK. In this example I'd use b. All right, as b get-- b is the thing that's changing as I go along here, but it could be things like, how many elements are there in a list if the input is a list, could be how many digits are there in a string if the input's a string, it could be the size of the integer as we go along. All right.? And what we want to do then, is we want to basically come up with, how do we characterize the growth-- God bless you-- of this problem in terms of this quadra-- sorry, terms of this exponential growth

    Now, one last piece of math. I could cheat. I said I just want an upper bound. I could get a really big upper bound, this thing grows exponentially. That doesn't help me much. Usually what I want to talk about is what's the smallest size class in which this function grows? With all of that, what that says, is that this we would write is order b. That algorithm is linear. You can see it. I've said the product was is 2 plus 3 b. As I make b really large, how does this thing grow? It grows as b. The 3 doesn't matter, it's just a constant, it's growing linearly. Another way of saying it is, if I, for example, increase the size of the input by 10, the amount of time increases by 10. And that's a sign that it's linear.

    OK. So there's one quick example. Let's look at another example. If you look at x 2, this one right here in your handout. OK. This is another way of doing exponentiation, but this one's a recursive function. All right?

    So again, let's look at it. What does it say to do? Well, it's basically saying a similar thing. It says, if I am in the base case, if b is equal to 1, the answer is just a. I could have used if b is equal to 0, the answer is 1, that would have also worked.

    Otherwise, what do I say? I say, ah, I'm in a nice recursive way, a to the b is the same as a times a to the b minus 1. And I've just reduced that problem to a simpler version of the same problem. OK, and you can see that this thing ought to unwrap, it's going to keep extending out those multiplications until gets down to the base case, going to collapse them all together.

    OK. Now I want to know what's the order of growth here? What's the complexity of this? Well, gee. It looks like it's pretty straightforward, right? I've got one test there, and then I've just got one thing to do here, which has got a subtraction and a multiplication.

    Oh, but how do I know how long it takes to do x 2? All right, we were counting basic steps. We don't know how long it takes to do x 2. So I'm going to show you a little trick for figuring that out. And in particular, I'm going to cheat slightly, I'm going to use a little bit of abusive mathematics, but I'm going to show you a trick to figure it out.

    In the case of a recursive exponentiator, I'm going to do the following trick. I'm going to let t of b be the number of steps it takes to solve the problem of size b. OK, and I can figure this out. I've got one test, I've got a subtraction, I've got a multiplication, that's three steps, plus whatever number of steps it takes to solve a problem of size b minus 1.

    All right, this is what's called a recurrence relation, there are actually cool ways to solve them. We can kind of eyeball it. In particular, how would I write an expression for t of b minus 1? Well the same way. This is 3 plus 3 plus t of b minus 2. Right? I'm using exactly the same form to reduce this. You know, you can see what's going to happen. If I reduce that, it would be 3 plus t of b minus 3, so in general, this is 3 k plus t of b minus k. OK. I'm just expanding it out.

    When am I done? How do I stop this? Any suggestions? Don't you hate it when professors ask questions? Yeah. Actually, I think I want b minus k equal to 1. Right? When this gets down to t of 1, I'm in the base case. So I'm done when b minus k equals 1, or k equals b minus 1. Right, that gets me down to the base case, I'm solving a problem with size 1, and in that case, I've got two more operations to do, so I plug this all back in, I-- t of b is I'm going to put k for b minus 1 I get 3 b minus 1 plus t of 1, so t of 1 is 2, so this is 3 b minus 1 plus 2, or 3 b minus 1.

    OK. A whole lot of work to basically say, again, order b is linear. But that's also nice, it lets you see how the recursive thing is simply unwrapping but the complexity in terms of the amount of time it takes is going to be the same. I owe you a candy. Thank you.

    OK. At this point, if we stop, you'll think all algorithms are linear. This is really boring. But they're not. OK? So let me show you another way I could do exponentiation. Taking an advantage of a trick. I want to solve a to the b.

    Here's another way I could do that. OK. If b is even, then a to the b is the same as a squared all to the b over 2. All right, just move the 2's around. It's the same thing. You're saying, OK, so what? Well gee, notice. This is a primitive operation. That's a primitive operation. But in one step, I've reduced this problem in half. I didn't just make it one smaller, I made it a half smaller. That's a nice deal.

    OK. But I'm not always going to have b as even. If b is odd, what do I do? Well, go back to what I did before. Multiply a by a to the b minus 1. You know, that's nice, right? Because if b was odd, then b minus one is even, which means on the next step, I can cut the problem in half again.

    OK? All right. x 3, as you can see right here, does exactly that. OK? You can take a quick look at it, even with the wrong glasses on, it says if a-- sorry, b is equal to 1, I'm just going to return a. Otherwise there's that funky little test. I'll do the remainder multiplied by 2, because these are integers, that gives me back an integer, I just check to see if it's equal to b, that tells me whether it's even or odd. And in the even case, I'd square, divide by half, call this again: in the odd case, I go b minus 1 and then multiply by a.

    I'll let you chase it through, it does work. What I want to look at is, what's the order of growth here?

    This is a little different, right? It's going to take a little bit more work, so let's see if we can do it. In the b even case, again I'm going to let t of b be the number of steps I want to go through. And we can kind of eyeball this thing, right? If b is even, I've got a test to see if b is equal to 1, and then I've got to do the remainder, the multiplication, and the test, I'm up to four. And then in the even case, I've got to do a square and the divide. So I've got six steps, plus whatever it takes to solve the problem size b over 2, right? Because that's the recursive call. b as odd, well I can go through the same kind of thing. I've got the same first four steps, I've got a check to see is it 1, I got a check to see if it's even, and then in the odd case, I've got to subtract 1 from b, that's a fifth step, I've got to go off and solve the recursive problem, and then I'm going to do one more multiplication, so it's 6 plus, in this case, t of b minus 1. Because it's now solving a one-smaller problem.

    On the next step though, this, we get substituted by that. Right, on the next step, I'm back in the even case, it's going to take six more steps, plus t of b minus 1. Oops, sorry about that, over 2. Because b minus 1 is now even. Don't sweat the details here, I just want you to see the reason it goes through it.

    What I now have, though, is a nice thing. It says, in either case, in general, t of b-- and this is where I'm going to abuse notation a little bit-- but I can basically bound it by t, 12 steps plus t of b over 2. And the abuse is, you know, it's not quite right, it depends upon whether it's all ready, but you can see in either case, after 12 steps, 2 runs through this and down to a problem size b over 2.

    Why's that nice? Well, that then says after another 12 steps, we're down to a problem with size t of b over 4. And if I pull it out one more level, it's 12 plus 12 plus t of b over 8, which in general is going to be, after k steps, 12 k because I'll have 12 of those to add up, plus t of b over 2 to the k.

    When am I done? When do I get down to the base case? Somebody help me out. What am I looking for? Yeah. You're jumping slightly ahead of me, but basically, I'm done when this is equal to 1, right? Because I get down to the base case, so I'm done when b u is over 2 to the k is equal to 1, and you're absolutely right, that's when k is log base 2 of b.

    You're sitting a long ways back, I have no idea if I'll make it this far or not. Thank you.

    OK. There's some constants in there, but this is order log b. Logarithmic. This matters. This matters a lot. And I'm going to show you an example in a second, just to drive this home, but notice the characteristics. In the first two cases, the problem reduced by 1 at each step. Whether it was recursive or iterative. That's a sign that it's probably linear. This case, I reduced the size of the problem in half. It's a good sign that this is logarithmic, and I'm going to come back in a second to why logs are a great thing.

    Let me show you one more class, though, about-- sorry, let me show you two more classes of algorithms. Let's look at the next one g-- and there's a bug in your handout, it should be g of n and m, I apologize for that, I changed it partway through and didn't catch it.

    OK. Order of growth here. Anybody want to volunteer a guess? Other than the TAs, who know? OK. Let's think it through. I've got two loops. All right? We already saw with one of the loops, you know, it looked like it might be linear, depending on what's inside of it, but let's think about this. I got two loops with g. What's g do? I've got an initialization of x, and then I say, for i in the range, so that's basically from up to n minus 1, what do I do? Well, inside of there, I've got another loop, for j in the range from up to m minus 1.

    What's the complexity of that inner loop? Sorry? OK. You're doing the whole thing for me. What's the complexity just of this inner loop here? Just this piece. How many times do I go through that loop? m. Right? I'm going to get back to your answer in a second, because you're heading in the right direction. The inner loop, this part here, I do m times. There's one step inside of it. Right? How many times do I go through that loop? Ah, n times, because for each value of i, I'm going to do that m thing, so that is, close to what you said, right? The order complexity here, if I actually write it, would be-- sorry, order n times m, and if m was equal to n, that would be order n squared, and this is quadratic. And that's a different behavior.

    OK. What am I doing? Building up examples of algorithms. Again, I want you to start seeing how to map the characteristics of the code-- the characteristics of the algorithm, let's not call it the code-- to the complexity. I'm going to come back to that in a second with that, but I need to do one more example, and I've got to use my high-tech really expensive props. Right. So here's the fourth or fifth, whatever we're up to, I guess fifth example.

    This is an example of a problem called Towers of Hanoi. Anybody heard about this problem? A few tentative hands.

    OK. Here's the story as I am told it. There's a temple in the middle of Hanoi. In that temple, there are three very large diamond-encrusted posts, and on those posts are sixty-four disks, all of a different size. And they're, you know, covered with jewels and all sorts of other really neat stuff. There are a set of priests in that temple, and their task is to move the entire stack of sixty-four disks from one post to a second post. When they do this, you know, the universe ends or they solve the financial crisis in Washington or something like that actually good happens, right? Boy, none of you have 401k's, you're not even wincing at that thing.

    All right. The rules, though, are, they can only move one disk at a time, and they can never cover up a smaller disk with a larger disk. OK. Otherwise you'd just move the whole darn stack, OK? So we want to solve that problem. We want to write a piece of code that helps these guys out, so I'm going to show you an example. Let's see if we can figure out how to do this.

    So, we'll start with the easy one. Moving a disk of size 1. OK, that's not so bad. Moving a stack of size 2, if I want to go there, I need to put this one temporarily over here so I can move the bottom one before I move it over. Moving a stack of size 3, again, if I want to go over there, I need to make sure I can put the spare one over here before I move the bottom one, I can't cover up any of the smaller ones with the larger one, but I can get it there. Stack of size 4, again I'm going there, so I'm going to do this initially, no I'm not, I'm going to start again. I'm going to go there initially, so I can move this over here, so I can get the base part of that over there, I want to put that one there before I put this over here, finally I get to the point where I can move the bottom one over, now I've got to be really careful to make sure that I don't cover up the bottom one in the wrong way before I get to the stage where I wish they were posts and there you go. All right? [APPLAUSE] I mean, I can make money at Harvard Square doing this stuff, right?

    All right, you ready to do five? Got the solution? Not so easy to see. All right, but this is actually a great one of those educational moments. This is a great example to think recursively. If I wanted to think about this problem recursively-- what do I mean by thinking recursively? How do I reduce this to a smaller-size problem in the same instant?

    And so, if I do that, this now becomes really easy. If I want to move this stack here, I'm going to take a stack of size n minus 1, move it to the spare spot, now I can move the base disk over, and then I'm going to move that stack of size n minus 1 to there. That's literally what I did, OK?

    So there's the code. Called towers. I'm just going to have you-- let you take a look at it. I'm giving it an argument, which is the size of the stack, and then just labels for the three posts. A from, a to, and a spare. And in fact, if we look at this-- let me just pop it over to the other side-- OK, I can move a tower, I'll say of size 2, from, to, and spare, and that was what I did. And if I want to move towers, let's say, size 5, from, to, and spare, there are the instructions for how to move it. We ain't going to do sixty-four. OK.

    All right. So it's fun, and I got a little bit of applause out of it, which is always nice for me, but I also showed you how to think about it recursively. Once you hear that description, it's easy to write the code, in fact. This is a place where the recursive version of it is much easier to think about than the iterative one.

    But what I really want to talk about is, what's the order of growth here? What's the complexity of this algorithm? And again, I'm going to do it with a little bit of abusive notation, and it's a little more complicated, but we can kind of look at. All right?

    Given the code up there, if I want to move a tower of size n, what do I have to do? I've got to test to see if I'm in the base case, and if I'm not, then I need to move a tower of size n minus 1, I need to move a tower of size 1, and I need to move a second-- sorry about that-- a second tower of size n minus 1.

    OK. t of 1 I can also reduce. In the case of a tower of size 1, basically there are two things to do, right? I've got to do the test, and then I just do the move. So the general formula is that.

    Now. You might look at that and say, well that's just a lot like what we had over here. Right? We had some additive constant plus a simpler version of the same problem reduced in size by 1. But that two matters. So let's look at it. How do I rea-- replace the expression FOR t of n minus 1? Substitute it in again. t of n minus 1 is 3 plus 2 t of n minus 2. So this is 3, plus 2 times 3, plus 4 t minus 2. OK. And if I substitute it again, I get 3 plus 2 times 3 plus 4 times 3 plus 8 t n minus 3.

    This is going by a little fast. I'm just substituting in. I'm going to skip some steps. But basically if I do this, I end up with 3 times 1 plus 2 plus 4 to 2 to the k minus 1 for all of those terms, plus 2-- I want to do this right, 2 to the k, sorry-- t of n minus k.

    OK. Don't sweat the details, I'm just expanding it out. What I want you to see is, because I've got two versions of that problem. The next time down I've got four versions. Next time down I've got eight versions. And in fact, if I substitute, I can solve for this, I'm done when this is equal to 1. If you substitute it all in, you get basically order 2 to the n.

    Exponential. That's a problem. Now, it's also the case that this is fundamentally what class this algorithm falls into, it is going to take exponential amount of time. But it grows pretty rapidly, as n goes up, and I'm going to show you an example in a second. Again, what I want you to see is, notice the characteristic of that. That this recursive call had two sub-problems of a smaller size, not one. And that makes a big difference.

    So just to show you how big a difference it makes, let's run a couple of numbers. Let's suppose n is 1000, and we're running at nanosecond speed. We have seen log, linear, quadratic, and exponential. So, again, there could be constants in here, but just to give you a sense of this. If I'm running at nanosecond speed, n, the size of the problem, whatever it is, is 1000, and I've got a log algorithm, it takes 10 nanoseconds to complete. If you blink, you miss it. If I'm running a linear algorithm, it'll take one microsecond to complete. If I'm running a quadratic algorithm, it'll take one millisecond to complete. And if I'm running an exponential algorithm, any guesses? I hope Washington doesn't take this long to fix my 401k plan. All right? 10 to the 284 years. As Emeril would say, pow! That's a some spicy whatever.

    All right. Bad jokes aside, what's the point? You see, these classes have really different performance. Now this is a little misleading. These are all really fast, so just to give you another set of examples, I'm not going to do the-- If I had a problem where the log one took ten milliseconds, then the linear one would take a second, the quadratic one would take 16 minutes. So you can see, even the quadratic ones can blow up in a hurry.

    And this goes back to the point I tried to make last time. Yes, the computers are really fast. But the problems can grow much faster than you can get a performance boost out of the computer. And you really, wherever possible, want to avoid that exponential algorithm, because that's really deadly. Yes.

    All right. The question is, is there a point where it'll quit. Yeah, when the power goes out, or-- so let me not answer it quite so facetiously. We'd be mostly talking about time.

    In fact, if I ran one of these things, it would just keep crunching away. It will probably quit at some point because of space issues, unless I'm writing an algorithm that is using no additional space. Right. Those things are going to stack up, and eventually it's going to run out of space. And that's more likely to happen, but, you know. The algorithm doesn't know that it's going to take this long to compute, it's just busy crunching away, trying to see if it can make it happen. OK. Good question, thank you.

    All right. I want to do one more extended example here., because we've got another piece to do, but I want to capture this, because it's important, so let me again try and say it the following way. I want you to recognize classes of algorithms and match what you see in the performance of the algorithm to the complexity of that algorithm. All right?

    Linear algorithms tend to be things where, at one pass-through, you reduce the problem by a constant amount, by one. If you reduce it by two, it's going to be the same thing. Where you go from problem of size n to a problem of size n minus 1.

    A log algorithm typically is one where you cut the size of the problem down by some multiplicative factor. You reduce it in half. You reduce it in third. All right?

    Quadratic algorithms tend to have this-- I was about to say additive, wrong term-- but doubly-nested, triply-nested things are likely to be quadratic or cubic algorithms, all right, because you know-- let me not confuse things-- double-loop quadratic algorithm, because you're doing one set of things and you're doing it some other number of times, and that's a typical signal that that's what you have there.

    OK. And then the exponentials, as you saw is when typically I reduce the problem of one size into two or more sub-problems of a smaller size. And you can imagine this gets complex and there's lots of interesting things to do to look to the real form, but those are the things that you should see.

    Now. Two other things, before we do this last example. One is, I'll remind you, what we're interested in is asymptotic growth. How does this thing grow as I make the problem size big?

    And I'll also remind you, and we're going to see this in the next example, we talked about looking at the worst case behavior. In these cases there's no best case worst case, it's just doing one computation. We're going to see an example of that in a second. What we really want to worry about, what's the worst case that happens.

    And the third thing I want you to keep in mind is, remember these are orders of growth. It is certainly possible, for example, that a quadratic algorithm could run faster than a linear algorithm. It depends on what the input is, it depends on, you know, what the particular cases are. So it is not the case that, on every input, a linear algorithm is always going to be better than a quadratic algorithm. It is just in general that's going to hold true, and that's what I want you to see.

    OK. I want to do one last example. I'm going to take a little bit more time on it, because it's going to both reinforce these ideas, but it's also going to show us how we have to think about what's a primitive step., and in a particular, how do data structures interact with this analysis? Here I've just been running integers, it's pretty simple, but if I have a data structure, I'm going to have to worry about that a little bit more.

    So let's look at that. And the example I want to look at is, suppose I want to search a list that I know is sorted, to see if an element's in the list. OK? So the example I'm going to do, I'm going to search a sorted list. All right. If you flip to the second side of your handout, you'll see that I have a piece of code there, that does this-- let me, ah, I didn't want to do that, let me back up slightly-- this is the algorithm called search. And let's take a look at it. OK?

    Basic idea, before I even look at the code, is pretty simple. If I've got a list that is sorted, in let's call it, just in increasing order, and I haven't said what's in the list, could be numbers, could be other things, for now, we're going to just assume they're integers. The easy thing to do would be the following: start at the front end of the list, check the first element. If it's the thing I'm looking for, I'm done. It's there. If not, move on to the next element. And keep doing that. But if, at any point, I get to a place in the list where the thing I'm looking for is smaller than the element in the list, I know everything else in the rest of the list has to be bigger than that, I don't have to bother looking anymore. It says the element's not there. I can just stop.

    OK. So that's what this piece of code does here. Right.? I'm going to set up a variable to say, what's the answer I want to return, is it there or not. Initially it's got that funny value none. I'm going to set up an index, which is going to tell me where to look, starting at the first part of the list, right? And then, when I got-- I'm also going to count how many comparisons I do, just so I can see how much work I do here, and then notice what it does. It says while the index is smaller than the size of the list, I'm not at the end of the list, and I don't have an answer yet, check. So I'm going to check to see if-- really can't read that thing, let me do it this way-- right, I'm going to increase the number of compares, and I'm going to check to say, is the thing I'm looking for at the i'th spot in the list? Right, so s of i saying, given the list, look at the i'th element, is it the same thing?

    If it is, OK. Set the answer to true. Which means, next time through the loop, that's going to pop out and return an answer. If it's not, then check to see, is it smaller than that element in the current spot of the list? And if that's true, it says again, everything else in the list has to be bigger than this, thing can't possibly be in the list, I'm taking advantage of the ordering, I can set the answer to false, change i to go to the next one, and next time through the loop, I'm going to pop out and print it out. OK?

    Right. Order of growth here. What do you think? Even with these glasses on, I can see no hands up, any suggestions? Somebody help me out. What do you think the order of growth is here? I've got a list, walk you through it an element at a time, do I look at each element of the list more than once? Don't think so, right? So, what does this suggest? Sorry?

    Constant. Ooh, constant says, no matter what the length of the list is, I'm going to take the same amount of time. And I don't think that's true, right? If I have a list ten times longer, it's going to take me more time, so-- not a bad guess, I'm still reward you, thank you.

    Somebody else. Yeah. Linear. Why? You're right, by the way, but why? Yeah. All right, so the answer was it's linear, which is absolutely right. Although for a reason we're going to come back in a second. Oh, thank you, I hope your friends help you out with that, thank you.

    Right? You can see that this ought to be linear, because what am I doing? I'm walking down the list. So one of the things I didn't say, it's sort of implicit here, is what is the thing I measuring the size of the problem in? What's the size of the list? And if I'm walking down the list, this is probably order of the length of the list s, because I'm looking at each element once.

    Now you might say, wait a minute. Thing's ordered, if I stop part way through and I throw away half the list, doesn't that help me? And the answer is yes, but it doesn't change the complexity. Because what did we say? We're measuring the worst case. The worst case here is, the things not in the list, in which case I've got to go all the way through the list to get to the end.

    OK. Now, having said that, and I've actually got a subtlety I'm going to come back to in a second, there ought to be a better way to do this. OK? And here's the better way to think about.

    I'll just draw out sort of a funny representation of a list. These are sort of the cells, if you like, in memory that are holding the elements of the list. What we've been saying is, I start here and look. If it's there, I'm done. If not, I go there. If it's there, I'm done, if not, I keep walking down, and I only stop when I get to a place where the element I'm looking for is smaller than the value in the list., in which case I know the rest of this is too big and I can stop. But I still have to go through the list.

    There's a better way to think about this, and in fact Professor Guttag has already hinted at this in the last couple of lectures. The better way to think about this is, suppose, rather than starting at the beginning, I just grabbed some spot at random, like this one. And I look at that value. If it's the value I'm looking for, boy, I ought to go to Vegas, I'm really lucky. And I'm done, right?

    If not, what could I do? Well, I could look at the value here, and compare it to the value I'm trying to find, and say the following; if the value I'm looking for is bigger than this value, where do I need to look? Just here. All right? Can't possibly be there, because I know this thing is over.

    On the other hand, if the value I'm looking for here-- sorry, the value I'm looking for is smaller than the value I see here, I just need to look here. All right?

    Having done that, I could do the same thing, so I suppose I take this branch, I can pick a spot like, say, this one, and look there. Because there, I'm done, if not, I'm either looking here or there. And I keep cutting the problem down.

    OK. Now, having said that, where should I pick to look in this list? I'm sorry? Halfway. Why? You're right, but why? Yeah. So the answer, in case you didn't hear it, was, again, if I'm a gambling person, I could start like a way down here. All right? If I'm gambling, I'm saying, gee, if I'm really lucky, it'll be only on this side, and I've got a little bit of work to do, but if I'm unlucky, I'm scrawed, the past pluperfect of screwed, OK., or a Boston fish. I'll look at the rest of that big chunk of the list, and that's a pain. So halfway is the right thing to do, because at each step, I'm guaranteed to throw away at least half the list. Right? And that's nice.

    OK. What would you guess the order of growth here is? Yeah. Why? Good. Exactly. Right? Again, if you didn't hear it, the answer was it's log. Because I'm cutting down the problem in half at each time. You're right, but there's something we have to do to add to that, and that's the last thing I want to pick up on. OK.

    Let's look at the code-- actually, let's test this out first before we do it. So I've added, as Professor Guttag did-- ah, should have said it this way, let's write the code for it first, sorry about that-- OK, I'm going to write a little thing called b search. I'm going to call it down here with search, which is simply going to call it, and then print an answer out.

    In binary search-- ah, there's that wonderful phrase, this is called a version of binary search, just like you saw bin-- or bi-section methods, when we were doing numerical things-- in binary search, I need to keep track of the starting point and the ending point of the list I'm looking at. Initially, it's the beginning and the end of it. And when I do this test, what I want to do, is say I'm going to pick the middle spot, and depending on the test, if I know it's in the upper half, I'm going to set my start at the mid point and the end stays the same, if it's in the front half I'm going to keep the front the same and I'm going to change the endpoint. And you can see that in this code here.

    Right? What does it say to do? It says, well I'm going to print out first and last, just so you can see it, and then I say, gee, if last minus first is less than 2, that is, if there's no more than two elements left in the list, then I can just check those two elements, and return the answer. Otherwise, we find the midpoint, and notice what it does. First, it's pointing to the beginning of the list, which initially might be down here at but after a while, might be part way through. And to that, I simply add a halfway point, and then I check. If it's at that point, I'm done, if not, if it's greater than the value I'm looking for, I either take one half or the other.

    OK. You can see that thing is cutting down the problem in half each time, which is good, but there's one more thing I need to deal with. So let's step through this with a little more care. And I keep saying, before we do it, let's just actually try it out. So I'm going to go over here, and I'm going to type test search-- I can type-- and if you look at your handout, it's just a sequence of tests that I'm going to do.

    OK. So initially, I'm going to set up the list to be the first million integers. Yeah, it's kind of simple, but it gives me an ordered list of these things, And let's run it. OK. So I'm first going to look for something that's not in the list, I'm going to see, is minus 1 in this list, so it's going to be at the far end, and if I do that in the basic case, bam. Done. All right? The basic, that primary search, because it looks at the first element, says it's smaller than everything else, I'm done.

    If I look in the binary case, takes a little longer. Notice the printout here. The printout is simply telling me, what are the ranges of the search. And you can see it wrapping its way down, cutting in half at each time until it gets there, but it takes a while to find. All right. Let's search to see though now if a million is in this list, or 10 million, whichever way I did this, it must be a million, right?

    In the basic case, oh, took a little while. Right, in the binary case, bam. In fact, it took the same number of steps as it did in the other case, because each time I'm cutting it down by a half. OK. That's nice.

    Now, let's do the following; if you look right here, I'm going to set this now to-- I'm going to change my range to 10 million, I'm going to first say, gee, is a million in there, using the basic search. It is. Now, I'm going to say, is 10 million in this, using the basic search. We may test your hypothesis, about how long does it take, if I time this really well, I ought to be able to end when it finds it, which should be right about now. That was pure luck. But notice how much longer it took.

    On the other hand, watch what happens with binary. Is the partway one there? Yeah. Is the last one there? Wow. I think it took one more step. Man, that's exactly what logs should do, right? I make the problem ten times bigger, it takes one more step to do it. Whereas in the linear case, I make it ten times bigger, it takes ten times longer to run. OK.

    So I keep saying I've got one thing hanging, it's the last thing I want to do, but I wanted you see how much of a difference this makes. But let's look a little more carefully at the code for binary search-- for search 1. What's the complexity of search 1? Well, you might say it's constant, right? It's only got two things to do, except what it really says is, that the complexity of search 1 is the same as the complexity of b search, because that's the call it's doing. So let's look at b search. All right? We've got the code for b search up there.

    First step, constant, right? Nothing to do. Second step, hm. That also looks constant, you think? Oh but wait a minute. I'm accessing s. I'm accessing a list. How long does it take for me to get the nth element of a list? That might not be a primitive step. And in fact, it depends on how I store a list.

    So, for example, in this case, I had lists that I knew were made out of integers. As a consequence, I have a list of ints. I might know, for example, that it takes four memory chunks to represent one int, just for example. And to find the i'th element, I'm simply going to take the starting point, that point at the beginning of memory where the list is, plus 4 times i, that would tell me how many units over to go, and that's the memory location I want to look for the i'th element of the list.

    And remember, we said we're going to assume a random access model, which says, as long as I know the location, it takes a constant amount of time to get to that point. So if the-- if I knew the lists were made of just integers, it'd be really easy to figure it out. Another way of saying it is, this takes constant amount of time to figure out where to look, it takes constant amount of time to get there, so in fact I could treat indexing into a list as being a basic operation.

    But we know lists can be composed of anything. Could be ints, could be floats, could be a combination of things, some ints, some floats, some lists, some strings, some lists of lists, whatever. And in that case, in general lists, I need to figure out what's the access time.

    And here I've got a choice. OK, one of the ways I could do would be the following. I could have a pointer to the beginning of the list where the first element here is the actual value, and this would point to the next element in the list. Or another way of saying it is, the first part of the cell could be some encoding of how many cells do I need to have to store the value, and then I've got some way of telling me where to get the next element of the list. And this would point to value, and this would point off someplace in memory.

    Here's the problem with that technique, and by the way, a number of programming languages use this, including Lisp. The problem with that technique, while it's very general, is how long does it take me to find the i'th element of the list? Oh fudge knuckle. OK. I've got to go to the first place, figure out how far over to skip, go to the next place, figure out how far over to skip, eventually I'll be out the door. I've got to count my way down, which means that the access would be linear in the length of the list to find the i'th element of the list, and that's going to increase the complexity.

    There's an alternative, which is the last point I want to make, which is instead what I could do, I should have said these things are called linked lists, we'll come back to those, another way to do it, is to have the start of the list be at some point in memory, and to have each one of the successive cells in memory point off to the actual value. Which may take up some arbitrary amount of memory. In that case, I'm back to this problem. And as a consequence, access time in the list is constant, which is what I want.

    Now, to my knowledge, most implementations of Python use this way of storing lists, whereas Lisp and Scheme do not.

    The message I'm trying to get to here, because I'm running you right up against time, is I have to be careful about what's a primitive step. With this, if I can assume that accessing the i'th element of a list is constant, then you can't see that the rest of that analysis looks just like the log analysis I did before, and each step, no matter which branch I'm taking, I'm cutting the problem down in half. And as a consequence, it is log. And the last piece of this, is as said, I have to make sure I know what my primitive elements are, in terms of operations.

    Summary: I want you to recognize different classes of algorithms. I'm not going to repeat them. We've seen log, we've seen linear, we've seen quadratic, we've seen exponential. One of the things you should begin to do, is to recognize what identifies those classes of algorithms, so you can map your problems into those ranges.

    And with that, good luck on Thursday.
  • 15
    Assignment 5: Word games
    12 pages
  • 16
    MIT 9: Binary Search, Bubble and Selection Sorts
    47:30
    Binary search, bubble and selection sorts. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR ERIC GRIMSON: Let's recap where we were. Last lecture, we talked about, or started to talk about, efficiency. Orders of growth. Complexity. And I'll remind you, we saw a set of algorithms, and part of my goal was to get you to begin to recognize characteristics of algorithms that map into a particular class.

    So what did we see? We saw linear algorithms. Typical characterization, not all the time, but typical characterization, is an algorithm that reduces the size of a problem by one, or by some constant amount each time, is typically an example of a linear algorithm. And we saw a couple of examples of linear algorithms.

    We also saw a logarithmic algorithm. and we like log algorithms, because they're really fast. A typical characteristic of a log algorithm is a pro-- or sorry, an algorithm where it reduces the size of the problem by a constant factor. Obviously-- and that's a bad way of saying it, I said constant the previous time-- in the linear case, it's subtract by certain amount. In the log case, it's divide by an amount. Cut the problem in half. Cut the problem in half again. And that's a typical characterization of a log algorithm.

    We saw some quadratic algorithms, typically those are things with multiple nested loops, or iterative or recursive calls, where you're doing, say, a linear amount of time but you're doing it a linear number of times, and so it becomes quadratic, and you'll see other polynomial kinds of algorithms.

    And finally, we saw an example of an exponential algorithm, those Towers of Hanoi. We don't like exponential algorithms, or at least you shouldn't like them, because they blow up quickly. And we saw some examples of that. And unfortunately, some problems are inherently exponential, you're sort of stuck with that, and then you just have to try be as clever as you can.

    OK. At the end of the lecture last time, I also showed you an example of binary search. And I want to redo that in a little more detail today, because I felt like I did that a little more quickly than I wanted to, so, if you really got binary search, fall asleep for about ten minutes, just don't snore, your neighbors may not appreciate it, but we're going to go over it again, because it's a problem and an idea that we're going to come back to, and I really want to make sure that I do this in a way that makes real good sense you.

    Again. Basic premise of binary search, or at least we set it up was, imagine I have a sorted list of elements. We get, in a second, to how we're going to get them sorted, and I want to know, is a particular element in that list.. And the basic idea of binary search is to start with the full range of the list, pick the midpoint, and test that point. If it's the thing I'm looking for, I'm golden. If not, because the list is sorted, I can use the difference between what I'm looking for and that midpoint to decide, should I look in the top half of the list, or the bottom half of the list? And I keep chopping it down.

    And I want to show you a little bit more detail of that, so let's create a simple little list here. All right? I don't care what's in there, but just assume that's my list. And just to remind you, on your handout, and there it is on the screen, I'm going to bring it back up, there's the little binary search algorithm. We're going to call search, which just calls binary search.

    And you can look at it, and let's in fact take a look at it to see what it does. We're going to call binary search, it's going to take the list to search and the element, but it's also going to say, here's the first part of the list, and there's the last part of the list, and what does it do inside that code? Well, it checks to see, is it bigger than two? Are there more than two elements there? If there are less than two elements there, I just check one or both of those to see if I'm looking for the right thing.

    Otherwise, what does that code say to do? It says find the midpoint, which says, take the start, which is pointing to that place right there, take last minus first, divide it by 2, and add it to start. And that basically, somewhere about here, gives me the midpoint.

    Now I look at that element. Is it the thing I'm looking for? If I'm really lucky, it is. If not, I look at the value of that point here and the thing I'm looking for. And for sake of argument, let's assume that the thing I'm looking for is smaller than the value here.

    Here's what I do. I change-- oops! Let me do that this way-- I change last to here, and keep first there, and I throw away all of that. All right? That's just the those-- let me use my pointer-- that's just these two lines here. I checked the value, and in one case, I'm changing the last to be mid minus 1, which is the case I'm in here, and I just call again. All right?

    I'm going to call exactly the same thing. Now, first is pointing here, last is pointing there, again, I check to see, are there more than two things left? There are, in this case. So what do I do? I find the midpoint by taking last minus first, divide by 2, and add to start. Just for sake of argument, we'll assume it's about there, and I do the same thing.

    Is this value what I'm looking for? Again, for sake of argument, let's assume it's not. Let's assume, for sake of argument, the thing I'm looking for is bigger than this. In that case, I'm going to throw away all of this, I'm going to hit that bottom line of that code.

    Ah. What does that do? It changes the call. So in this case, first now points there, last points there. And I cut around.

    And again, notice what I've done. I've thrown away most of the array-- most of the list, I shouldn't say array-- most of the list. All right? So it cuts it down quickly as we go along.

    OK. That's the basic idea of binary search. And let's just run a couple of examples to remind you of what happens if we do this.

    So if I call, let's [UNINTELLIGIBLE], let's set up s to be, I don't know, some big long list. OK. And I'm going to look to see, is a particular element inside of that list, and again, I'll remind you, that's just giving me the integers from zero up to 9999 something or other.

    If I look for, say, minus 1, you might go, gee, wait a minute, if I was just doing linear search, I would've known right away that minus one wasn't in this list, because it's sorted and it's smaller than the first elements. So this looks like it's doing a little bit of extra work, but you can see, if you look at that, how it cuts it down at each stage. And I'll remind you, what I'm printing out there is, first and last, with the range I'm looking over, and then just how many times the iteration called.

    So in this case, it just keeps chopping down from the back end, which kind of makes sense, all right? But in a fixed number, in fact, twenty-three calls, it gets down to the point of being able to say whether it's there.

    Let's go the other direction. And yes, I guess I'd better say s not 2, or we're going to get an error here. Again, in twenty-three checks. In this case, it's cutting up from the bottom end, which makes sense because the thing I'm looking for is always bigger than the midpoint, and then, I don't know, let's pick something in between. Somebody want-- ah, I keep doing that-- somebody like to give me a number? I know you'd like to give me other things, other expression, somebody give me a number. Anybody? No? Sorry. Thank you. Good number.

    OK, walks in very quickly. OK? And if you just look at the numbers, you can see how it cuts in from one side and then the other side as it keeps narrowing that range, until it gets down to the place where there are at most two things left, and then it just has to check those two to say whether it's there or not.

    Think about this compared to a linear search. All right? A linear search, I start at the beginning of the list and walk all the way through it. All right, if I'm lucky and it's at the low end, I'll find it pretty quickly. If it's not, if it's at the far end, I've got to go forever, and you saw that last time where this thing paused for a little while while it actually searched a list this big.

    OK. So, what do I want you to take away from this? This idea of binary search is going to be a really powerful tool. And it has this property, again, of chopping things into pieces.

    So in fact, what does that suggest about the order of growth here? What is the complexity of this? Yeah. Logarithmic. Why?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: Yeah. Thank you. I mean, I know I sort of said it to you, but you're right. It's logarithmic, right? It's got that property of, it cuts things in half.

    Here's another way to think about why is this log. Actually, let me ask a slightly different question. How do we know this always stops? I mean, I ran three trials here, and it did. But how would I reason about, does this always stop? Well let's see. Where's the end test on this thing? The end test-- and I've got the wrong glasses on-- but it's up here, where I'm looking to see, is last minus first less than or equal to 2?

    OK. So, soon as I get down to a list that has no more than two elements in it, I'm done. Notice that. It's a less than or equal to. What if I just tested to see if it was only, say, one? There was one element in there. Would that have worked?

    I think it depends on whether the list is odd or even in length. Actually, that's probably not true. With one, it'll probably always get it down there, but if I've made it just equal to two, I might have lost.

    So first of all, I've got to be careful about the end test. But the second thing is, OK, if it stops whenever this is less than two, am I convinced that this will always halt? And the answer is sure. Because what do I do? At each stage, no matter which branch, here or here, I take, I'm cutting down the length of the list that I'm searching in half. All right?

    So if I start off with a list of length n, how many times can I divide it by 2, until I get to something no more than two left? Log times, right.? Exactly as the gentleman said.

    Oh, I'm sorry. You're patiently waiting for me to reward. Or actually, maybe you're not. Thank you.

    OK. So this is, in fact, log. Now, having said that, I actually snuck something by you. And I want to spend a couple of minutes again reinforcing that. So if we look at that code, and we were little more careful about this, what did we say to do? We said look an-- sorry. Count the number of primitive operations in each step.

    OK. So if I look at this code, first of all I'm calling search, it just has one call, so looks like search is constant, except I don't know what happens inside of b search. So I've got to look at b search.

    So let's see. The first line, that print thing, is obviously constant, right? Just take it as a constant amount of operations But. let's look at the next one here, or is that second line?

    OK. If last minus first is greater than or equal to 2-- sorry, less than 2, then either look at this thing or look at that thing. And that's where I said we've got to be careful. That's accessing an element of a list. We have to make sure that, in fact, that operation is not linear.

    So let me expand on that very slightly, and again, we did this last time but I want to do one more time. I have to be careful about how I'm actually implementing a list.

    So, for example: in this case, my list is a bunch of integers. And one of the things I could take advantage of, is I'm only going to need a finite amount of space to represent an integer.

    So, for example, if I want to allow for some fairly large range of integers, I might say, I need four memory cells in a row to represent an integer. All right, if it's a zero, it's going to be a whole bunch of ones-- of zeroes, so one, it may be a whole bunch of zeroes in the first three and then a one at the end of this thing, but one of the way to think about this list in memory, is that I can decide in constant time how to find the i'th element of a list.

    So in particular, here's where the zero-th element of the list starts, there's where the first element starts, here's where the third element starts, these are just memory cells in a row, and to find the zero-th element, if start is pointing to that memory cell, it's just at start.

    To find the first element, because I know I need four memory cells to represent an integer, it's at start plus 4. To get to the second element, I know that that's-- you get the idea-- at the start plus 2 times 4, and to get to the k'th element, I know that I want to take whatever the start is which points to that place in memory, take care, multiply by 4, and that tells me exactly where to go to find that location.

    This may sound like a nuance, but it's important. Why? Because that's a constant access, right? To get any location in memory, to get to any value of the list, I simply have to say which element do I want to get, I know that these things are stored in a particular size, multiply that index by 4, add it to start, and then it's in a constant amount of time I can go to that location and get out the cell.

    OK. That works nicely if I know that I have things stored in constant size. But what if I have a list of lists? What if I have a homogeneous list, a list of integers and strings and floats and lists and lists of lists and lists of lists of lists and all that sort of cool stuff? In that case, I've got to be a lot more careful.

    So in this case, one of the standard ways to do this, is to use what's called a linked list. And I'm going to do it in the following way.

    Start again, we'll point to the beginning of the list. But now, because my elements are going to take different amounts of memory, I'm going to do the following thing. In the first spot, I'm going to store something that says, here's how far you have to jump to get to the next element.

    And then, I'm going to use the next sequence of things to represent the first element, or

    the zero-th element, if you like. In this case I might need five. And then in the next spot, I'm going to say how far you have to jump to get to the next element. All right, followed by whatever I need to represent it, which might only be a blank one.

    And in the next spot, maybe I've got a really long list, and I'm going to say how to jump to get to the next element.

    All right, this is actually kind of nice. This lets me have a way of representing things that could be arbitrary in size. And some of these things could be huge, if they're themselves lists.

    Here's the problem. How do I get to the nth-- er, the k'th element in the list, in this case? Well I have to go to the zero-th element, and say OK, gee, to get to the next element, I've got to jump this here. And to get to the next element, I've got to jump to here, and to get to the next element, I've got to jump to here, until I get there.

    And so, I get some power. I get the ability to store arbitrary things, but what just happened to my complexity? How long does it take me to find the

    k'th element? Linear. Because I've got to walk my way down it. OK? So in this case, you have linear access. Oh fudge knuckle. Right? If that was the case in that code, then my complexity is no longer log, because I need linear access for each time I've got to go to the list, and it's going to be much worse than that.

    All right. Now. Some programming languages, primarily Lisp, actually store lists these ways. You might say, why? Well it turns out there's some trade-offs to it. It has some advantages in terms of power of storing things, it has some disadvantages, primarily in terms of access time.

    Fortunately for you, Python decided, or the investors of Python decided, to store this a different way. And the different way is to say, look, if I redraw this, it's called a box and pointer diagram, what we really have for each element is two things. And I've actually just reversed the order here.

    We have a pointer to the location in memory that contains the actual value, which itself might be a bunch of pointers, and we have a pointer to the actual-- sorry, a pointer the value and we have a pointer to the next element in the list. All right?

    And one of the things we could do if we look at that is, we say, gee, we could reorganize this in a pretty straightforward way. In particular, why don't we just take all of the first cells and stick them together? Where now, my list is a list of pointers, it's not a set of values but it's actually a pointer off to some other piece of memory that contains the value.

    Why is this nice? Well this is exactly like this. All right? It's now something that I can search in constant time. And that's what's going to allow me to keep this thing as being log.

    OK. With that in mind, let's go back to where we were. And where were we? We started off talking about binary search, and I suggested that this was a log algorithm, which it is, which is really kind of nice.

    Let's pull together what this algorithm actually does. If I generalize binary search, here's what I'm going to stake that this thing does.

    It says one: pick the midpoint.

    Two: check to see if this is the answer, if this is the thing I'm looking for.

    And then, three: if not, reduce to a smaller problem, and repeat.

    OK, you're going, yeah, come on, that makes obvious sense. And it does. But I want you to keep that template in mind, because we're going to come back to that. It's an example of a very common tool that's going to be really useful to us, not just for doing search, but for doing a whole range of problems. That is, in essence, the template the describes a log style algorithm. And we're going to come back to it.

    OK. With that in mind though, didn't I cheat? I remind you, I know you're not really listening to me, but that's OK. I reminded you at the beginning of the lecture, I said, let's assume we have a sorted list, and then let's go search it. Where in the world

    did that sorted list come from? What if I just get a list of elements, what do I do? Well let's see. My fall back is, I could just do linear search, walk down the list one at a time, just comparing those things. OK. So that's sort of my base. But what if I wanted, you know, how do I want to get to that sorted list? All right?

    Now. One of the questions, before we get to doing the sorting, is even to ask, what should I do in a search case like that? All right, so in particular, does it make sense, if I'm given an unsorted list, to first sort it, and then search it? Or should I just use the basically linear case? All right?

    So, here's the question. Should we sort before we search?

    OK. So let's see, if I'm going to do this, how fast could we sort a list? Can we sort a list in sublinear time? Sublinear meaning, something like log less than linear time? What do you think? It's possible? Any thoughts? Don't you hate professors who stand here waiting for you to answer, even when they have candy?

    Does it make sense to think we could do this in less than linear time? You know, it takes a little bit of thinking. What would it mean-- [UNINTELLIGIBLE PHRASE] do I see a hand, way at the back, yes please? Thank you. Man, you're going to really make me work here, I have no idea if I can get it that far, ah, your friend will help you out. Thank you.

    The gentleman has it exactly right. How could I possibly do it in sublinear time, I've got to look at least every element once. And that's the kind of instinct I'd like you to get into thinking about. So the answer here is no.

    OK. Can we sort it in linear time? Hmmm. That one's not so obvious. So let's think about this for a second. To sort a list in linear time, would say, I have to look at each element in the list at most a constant number of times. It doesn't have to be just once, right? It could be two or three times.

    Hmm. Well, wait a minute. If I want to sort a list, I'll take one element, I've got to look at probably a lot of the other elements in the list in order to decide where it goes. And that suggests it's going to depend on how long the list is. All right, so that's a weak argument, but in fact, it's a way of suggesting, probably not.

    All right. So how fast could I sort a list? How fast can we sort it? And we're going to come back to this, probably next time if I time this right, but the answer is, we can do it in n log n time. We're going to come back to that. All right? And I'm going to say-- sort of set that stage here, so that-- It turns out that that's probably about the best we can do, or again ends at the length of the list.

    OK, so that's still comes back to my question. If I want to search a list, should I sort it first and then search it? Hmmm. OK, so let's do the comparison. I'm just going to take an unsorted list and search it, I could do it in linear time, right? One at a time. Walk down the elements until I find it. That would be order n. On the other hand, if I want to sort it first, OK, if I want to do sort and search, I want to sort it, it's going to take n log n time to sort it, and having done that, then I can search it in log n time.

    Ah. So which one's better? Yeah. Ah-ha. Thank you. Hold on to that thought for second, I'm going to come back to it. That does not assume I'm running a search it wants, which one's better?

    The unsorted, and you have exactly the point I want to get to-- how come all the guys, sorry, all the people answering questions are way, way up in the back? Wow. that's a Tim Wakefield pitch right there, all right. Thank you.

    He has it exactly right. OK? Is this smaller than that? No. Now that's a slight lie. Sorry, a slight misstatement, OK? I could run for office, couldn't I, if I can do that kind of talk. It's a slight misstatement in the sense that these should really be orders of growth. There are some constants in there, it depends on the size, but in general, n log n has to be bigger than n.

    So, as the gentleman back there said, if I'm searching it once, just use the linear search. On the other hand, am I likely to only search a list once? Probably not. There are going to be multiple elements I'm going to be looking for, so that suggests that in fact, I want to amortize the cost.

    And what does that say? It says, let's assume I want to do k searches of a list. OK. In the linear case, meaning in the unsorted case, what's the complexity of this? k times n, right? Order n to do the search, and I've got to do it k times, so this would be k times n.

    In the [GARBLED PHRASE] sort and search case, what's my cost? I've got to sort it, and we said, and we'll come back to that next time, that I can do the sort in n log n, and then what's the search in this case? Let's log n to do one search, I want to do k of them, that's k log n, ah-ha!

    Now I'm in better shape, right? Especially for really large n or for a lot of k, because now in general, this is going to be smaller than that. So this is a place where the amortized cost actually helps me out. And as the gentleman at the back said, the question he asked is right, it depends on what I'm trying to do. So when I do the analysis, I want to think about what am I doing here, am I capturing all the pieces of it? Here, the two variables that matter are what's the length of the list, and how many times I'm going to search it? So in this case, this one wins, whereas in this case, that one wins.

    OK. Having said that, let's look at doing some sorts. And I'm going to start with a couple of dumb sorting mechanisms. Actually, that's the wrong way saying it, they're simply brain-damaged, they're not dumb, OK? They are computationally challenged, meaning, at the time they were invented, they were perfectly good sorting algorithms, there are better ones, we're going to see a much better one next time around, but this is a good way to just start thinking about how to do the algorithm, or how to do the sort. Blah, try again. How to do this sort.

    So the first one I want to talk about it's what's called selection sort. And it's on your handout, and I'm going to bring the code up here, you can see it, it's called cell sort, just for selection sort. And let's take a look at what this does. OK. And in fact I think the easy way to look at what this does-- boy. My jokes are that bad. Wow-- All right.

    I think the easiest way to look at what this does, is let's take a really simple example-- I want to make sure I put the right things out-- I've got a simple little list of values there. And if I look at this code, I'm going to run over a loop, you can see that there, i is going to go from zero up to the length minus 1, and I'm going to keep track of a couple of variables. Min index, I think I called it min val.

    OK. Let's simulate the code. Let's see what it's doing here. All right, so we start off. Initially i-- ah, let me do it this way, i is going to point there, and I want to make sure I do it right, OK-- and min index is going to point to the value of i, which is there, and min value is initially going to have the value 1. So we're simply catting a hold of what's the first value we've got there.

    And then what do we do? We start with j pointing here, and we can see what this loop's going to do, right? j is just going to move up.

    So it's going to look at the rest of the list, walking along, and what does it do? It says, right. If j is-- well it says until j is at the less than the length of l-- it says, if min value is bigger than the thing I'm looking at, I'm going to do something, all right? So let's walk this. Min value is 1,. Is 1 bigger than 8? No. I move j up. Is 1 bigger than 3? No. 1 bigger than 6? No. 1 bigger than 4? No.

    I get to the end of the loop, and I actually do a little bit of wasted motion there. And the little bit of wasted motion is, I take the value at i, store it away temporarily, take the value where min index is pointing to, put it back in there, and then swap it around.

    OK. Having done that, let's move i up to here. i is now pointing at that thing. Go through the second round of the loop.

    OK. What does that say? I'm going to change min index to also point there n value is 8, j starts off here, and I say, OK, is the thing I'm looking at here smaller than that? Yes.

    Ah-ha. What does that say to do? It says, gee, make min index point to there, min value be 3. Change j. Is 6 bigger than 3? Yes. Is 4 bigger than 3? Yes. Get to the end.

    And when I get to the end, what do I do? Well, you see, I say, take temp, and store away what's here, all right? Which is that value, and then take what min index is pointing to, and stick it in there, and finally, replace that value.

    OK. Aren't you glad I'm not a computer? Slow as hell.

    What's this thing doing? It's walking along the list, looking for the smallest thing in the back end of the list, keeping track of where it came from, and swapping it with that spot in the list. All right?

    So in the first case, I didn't have to do any swaps because 1 was the smallest thing. In the second case, I found in the next smallest element and moved here, taking what was there and moving it on, in this case I would swap the 4 and the 8, and in next case I wouldn't have to do anything.

    Let's check it out. I've written a little bit of a test script here, so if we test cell sort, and I've written this so that it's going to print out what the list is at the end of each round, OK.

    Ah-ha. Notice what-- where am I, here-- notice what happened in this case. At the end of the first round, I've got the smallest element at the front. At the end of the second round, I've got the smallest two elements at the front, in fact I got all of them sorted out. And it actually runs through the loop multiple times, making sure that it's in the right form.

    Let's take another example. OK. Smallest element at the front. Smallest two elements at the front. Smallest three elements at the front. Smallest four elements at the front, you get the idea. Smallest five elements at the front.

    So this is a nice little search-- sorry, a nice little sort algorithm . And in fact, it's relying on something that we're going to come back to, called the loop invariant. Actually, let me put it on this board so you can see it.

    The loop invariant what does the loop invariant mean? It says, here is a property that is true of this structure every time through the loop. In the loop invariant here is the following: the list is split, into a prefix or a first part, and a suffix, the prefix is sorted, the suffix is not, and basically, the loop starts off with the prefix being nothing and it keeps increasing the size of the prefix by 1 until it gets through the entire list, at which point there's nothing in the suffix and entire prefix is sorted.

    OK? So you can see that, it's just walking through it, and in fact if I look at a couple of another-- another couple of examples, it's been a long day, again, you can see that property. You'll also notice that this thing goes through the entire list, even if the list is sorted before it gets partway through.

    And that you might look at, for example, that first example, and say, man by this stage it was already sorted, yet it had to go through and check that the third element was in the right place, and then the fourth and then the fifth and then the six.

    OK. What order of growth? What's complexity of this? I've got to get rid of this candy. Anybody help me out? What's the complexity of this? Sorry, somebody at the back.

    n squared. Yeah, where n is what? Yeah, and I can't even see who's saying that. Thank you. Sorry, I've got the wrong glasses on, but you're absolutely right, and in case the rest of you didn't hear it, n squared.

    How do I figure that out? Well I'm looping down the list, right? I'm walking down the list. So it's certainly at least linear in the length of the list. For each starting point, what do I do? I look at the rest of the list to decide what's the element to swap into the next place.

    Now, you might say, well, wait a minute. As I keep moving down, that part gets smaller, it's not always the initial length of the list, and you're right. But if you do the sums, or if you want to think of it this way, if you think about this more generally, it's always on average at least the length of the list. So I've got to do n things n times. So it's quadratic, in terms of that sort.

    OK. That's one way to do this sort. Let's do another one.

    The second one we're going to do is called bubble sort. All right? And bubble sort is also on your handout. And you want to take the first of these, let me-- sorry, for a second let me uncomment that, and let me comment this out--

    All right, you can see the code for bubble sort there. Let's just look at it for a second, then we'll try some examples, and then we'll figure out what it's actually doing. So bubble sort, which is right up here. What's it going to do? It's going to let j run over the length of the list, all right, so it's going to start at some point to move down, and then it's going to let i run over range, that's just one smaller, and what's it doing there?

    It's looking at successive pairs, right? It's looking at the i'th and the i plus first element, and it's saying, gee, if the i'th element is bigger than the i'th plus first element, what's the next set of three things doing? Just swapping them, right? I temporarily hold on to what's in the i'th element so I can move the i plus first one in, and then replace that with the i'th element.

    OK. What's this thing doing then, in terms of sorting? At the end of the first pass, what could I say about the result of this thing? What's the last element in the list look like? I hate professors who do this.

    Well, let's try it. Let's try a little test. OK? Test bubble sort-- especially if I could type-- let's run it on the first list. OK, let's try it on another one. Oops sorry. Ah, I didn't want to do it this time, I forgot to do the following, bear with me. I gave away my punchline. Let's try it again. Test bubble sort.

    OK, there's the first run, I'm going to take a different list. Can you see a pattern there? Yeah.

    STUDENT: The last cell in the list is always going to [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: Yeah. Why? You're right, but why?

    STUDENT: [UNINTELLIGIBLE PHRASE]

    PROFESSOR ERIC GRIMSON: Exactly right. Thank you. The observation is, thank you, on the first pass through, the last element is the biggest thing in the list. On the next pass through, the next largest element is at the second point in the list.

    OK? Because what am I doing? It's called bubble sort because it's literally bubbling along, right? I'm walking along the list once, taking two things, and saying, make sure the biggest one is next. So wherever the largest element started out in the list, by the time I get through it, it's at the end. And then I go back and I start again, and I do the same thing.

    OK. The next largest element has to end up in the second last spot. Et cetera. All right, so it's called bubble sort because it does this bubbling up until it gets there.

    Now. What's the order of growth here? What's the complexity? I haven't talked to the side of the room in a while, actually I have. This gentleman has helped me out. Somebody else help me out. What's the complexity here? I must have the wrong glasses on to see a hand. No help.

    Log? Linear? Exponential? Quadratic? Yeah. Log. It's a good think, but why do you think it's log? Ah-ha. It's not a bad instinct, the length is getting shorter each time, but what's one of the characteristics of a log algorithm? It drops in half each time. So this isn't--

    OK. And you're also close. It's going to be linear, but how many times do I go through this? All right, I've got to do one pass to bubble the last element to the end. I've got to do another pass to bubble the second last element to the end. I've got to do another pass. Huh. Sounds like a linear number of times I've got to do-- oh fudge knuckle. A linear number of things, quadratic. Right?

    OK. So this is again an example, this was quadratic, and this one was quadratic. And I have this, to write it out, this is order the length of the list squared, OK? Just to make it clear what we're actually measuring there. All

    right. Could we do better? Sure. And in fact, next time we're going to show you that n log n algorithm, but even with bubble sort, we can do better. In a particular, if I look at those traces, I can certainly see cases where, man, I already had the list sorted much earlier on, and yet I kept going back to see if there was anything else to bubble up.

    How would I keep track of that? Could I take advantage of that? Sure. Why don't I just keep track on each pass through the algorithm whether I have done any swaps? All right? Because if I don't do any swaps on a pass through the algorithm, then it says everything's in the right order.

    And so, in fact, the version that I commented out-- which is also in your handout and I'm now going to uncomment, let's get that one out, get rid of this one-- notice the only change. I'm going to keep track of a little variable called swap, it's initially true, and as long as it's true, I'm going to keep going, but inside of the loop I'm going to set it to false, and only if I do a swap will I set it to true.

    This says, if I go through an entire pass through the list and nothing gets changed, I'm done. And in fact if I do that, and try test bubble sort, well, in the first case, looks the same. Ah. On the second case, I spot it right away. On the third case, it takes me the same amount of time. And the fourth case, when I set it up, I'm done.

    OK. So what's the lesson here? I can be a little more careful about keeping track of what goes on inside of that loop. If I don't have any more work to do, let me just stop.

    All right. Nonetheless, even with this change, what's the order growth for bubble sort? Still quadratic, right? I'm looking for the worst case behavior, it's still quadratic, it's quadratic in the length of the list, so I'm sort of stuck with that.

    Now. Let me ask you one last question, and then we'll wrap this up. Which of these algorithms is better? Insertion sort or bubble sort?

    STUDENT: Bubble.

    PROFESSOR ERIC GRIMSON: Bubble. Bubble bubble toil and trouble. Who said bubble? Why?

    STUDENT: Well, the first one was too inefficient [UNINTELLIGIBLE] store and compare each one, so [UNINTELLIGIBLE]

    PROFESSOR ERIC GRIMSON: It's not a bad instinct. Right. So it-- so, your argument is, bubble is better because it's is essentially not doing all these extra comparisons. Another way of saying it is, I can do this stop when I don't need to. All right?

    OK. Anybody have an opposing opinion? Wow, this sounds like a presidential debate. Sorry, I should reward you. Thank you for that statement. Anybody have an opposing opinion? Everybody's answering these things and sitting way up at the back. Nice catch. Yeah.

    STUDENT: [INAUDIBLE]

    PROFESSOR ERIC GRIMSON: I don't think so, right? I think selection sort, I still have to go through multiple times, it was still quadratic, OK, but I think you're heading towards a direction I want to get at, so let me prime this a little bit. How many swaps do I do in general in bubble sort, compared to selection source?

    God bless. Oh, sorry, that wasn't a sneeze, it was a two? How many swaps do I do in bubble sort? A lot. Right. Potentially a lot because I'm constantly doing that, that says I'm running that inner loop a whole bunch of times.

    How many swaps do I do in selection sort? Once each time. Right? I only do one swap potentially, it-- though not one potentially, each time at the end of the loop I do a swap.

    So this actually suggests again, the orders of growth are the same, but probably selection sort is a more efficient algorithm, because I'm not doing that constant amount of work every time around. And in fact, if you go look up, you won't see bubble sort used very much. Most-- I shouldn't say most, many computer scientists don't think it should be taught, because it's just so inefficient. I disagree, because it's a clever idea, but it's still something that we have to keep track of.

    All right. We haven't gotten to our n log n algorithm, we're going to do that next time, but I want to set the stage here by pulling out one last piece.

    OK. Could we do better in terms of sorting? Again, remember what our goal was. If we could do sort, then we saw, if we amortized the cost, that searching is a lot more efficient if we're searching a sorted list.

    How could we do better? Let me set the stage. I already said, back here, when I used this board, that this idea was really important. And that's because that is a version of a divide and conquer algorithm.

    OK. Binary search is perhaps the simplest of the divide and conquer algorithms, and what does that mean? It says, in order to solve a problem, cut it down to a smaller problem and try and solve that one.

    So to just preface what we're going to do next time, what would happen if I wanted to do sort, and rather than in sorting the entire list at once, I broke it into pieces, and sorted the pieces, and then just figured out a very efficient way to bring those two pieces and merge them back together again? Where those pieces, I would do the same thing with, I would divide them up into smaller chunks, and sort those. Is that going to give me a more efficient algorithm?

    And if you come back on Thursday, we'll answer that question.
  • 17
    MIT Lecture 10: Divide and Conquer Methods, Merge Sort, Exceptions
    46:19
    Divide and conquer methods, merge sort, exceptions. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: Last time we were talking about binary search and I sort of left a promise to you which I need to pick up. I want to remind you, we were talking about search, which is a very fundamental thing that we do in a whole lot of applications. We want to go find things in some data set. And I'll remind you that we sort of a separated out two cases. We said if we had an ordered list, we could use binary search. And we said that was log rhythmic, took log n time where n is the size of the list.

    If it was an unordered list, we were basically stuck with linear search. Got to walk through the whole list to see if the thing is there. So that was of order in. And then one of the things that I suggested was that if we could figure out some way to order it, and in particular, if we could order it in n log n time, and we still haven't done that, but if we could do that, then we said the complexity changed a little bit. But it changed in a way that I want to remind you.

    And the change was, that in this case, if I'm doing a single search, I've got a choice. I could still do the linear case, which is order n or I could say, look, take the list, let's sort it and then search it. But in that case, we said well to sort it was going to take n log n time, assuming I can do that.

    Once I have it sorted I can search it in log n time, but that's still isn't as good as just doing n. And this led to this idea of amortization, which is I need to not only factor in the cost, but how am I going to use it? And typically, I'm not going to just search once in a list, I'm going to search multiple times. So if I have to do k searches, then in the linear case, I got to do order n things k times. It's order k n.

    Whereas in the ordered case, I need to get them sorted, which is still n log n, but then the search is only log n. I need to do k of those. And we suggested well this is better than that. This is certainly better than that. m plus k all times log n is in general going to be much better than k times n. It depends on n and k but obviously as n gets big, that one is going to be better.

    And that's just a way of reminding you that we want to think carefully, but what are the things we're trying to measure when we talk about complexity here? It's both the size of the thing and how often are we going to use it? And there are some trade offs, but I still haven't said how I'm going to get an n log n sorting algorithm, and that's what I want to do today. One of the two things I want to do today.

    To set the stage for this, let's go back just for a second to binary search. At the end of the lecture I said binary search was an example of a divide and conquer algorithm. Sort of an Attila the Hun kind of approach to doing things if you like. So let me say -- boy, I could have made a really bad political joke there, which I will forego, right.

    Let's say what this actually means, divide and conquer. Divide and conquer says basically do the following: split the problem into several sub-problems of the same type. I'll come back in a second to help binary searches matches in that, but that's what we're going to do. For each of those sub-problems we're going to solve them independently, and then we're going to combine those solutions.

    And it's called divide and conquer for the obvious reason. I'm going to divide it up into sub-problems with the hope that those sub-problems get easier. It's going to be easier to conquer if you like, and then I'm going to merge them back. Now, in the binary search case, in some sense, this is a little bit trivial. What was the divide? The divide was breaking a big search up into half a search. We actually threw half of the list away and we kept dividing it down, until ultimately we got something of size one to search. That's really easy.

    The combination was also sort of trivial in this case because the solution to the sub-problem was, in fact, the solution to the larger problem. But there's the idea of divide and conquer. I'm going to use exactly that same ideas to tackle sort. Again, I've got an unordered list of n elements. I want to sort it into a obviously a sorted list. And that particular algorithm is actually a really nice algorithm called merge sort. And it's actually a fairly old algorithm. It was invented in 1945 by John von Neumann one of the pioneers of computer science. And here's the idea behind merge sort, actually I'm going to back into it in a funny way.

    Let's assume that I could somehow get to the stage where I've got two sorted lists. How much work do I have to do to actually merge them together? So let me give you an example. Suppose I want to merge two lists, and they're sorted.

    Just to give you an example, here's one list, 3121724 Here's another list, 12430. I haven't said how I'm going to get those sorted lists, but imagine I had two sorted lists like that. How hard is it to merge them? Well it's pretty easy, right? I start at the beginning of each list, and I say is one less than three? Sure. So that says one should be the first element in my merge list.

    Now, compare the first element in each of these lists. Two is less than three, so two ought to be the next element of the list. And you get the idea. What am I going to do next? I'm going to compare three against four. Three is the smallest one, and I'm going to compare four games twelve, which is going to give me four. And then what do? I have to do twelve against thirty, twelve is smaller, take that out. Seventeen against thirty, twenty-four against thirty And by this stage I've got nothing left in this element, so I just add the rest of that list in.

    Wow I can sort two lists, so I can merge two lists. I said it poorly. What's the point? How many operations did it take me to do this? Seven comparisons, right? I've got eight elements. It took me seven comparisons, because I can take advantage of the fact I know I only ever have to look at the first element of each sub-list. Those are the only things I need to compare, and when I run out of one list, I just add the rest of the list in.

    What's the order of complexity of merging? I heard it somewhere very quietly.

    STUDENT: n.

    PROFESSOR: Sorry, and thank you. Linear, absolutely right? And what's n by the way here? What's it measuring?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR: In both lists, right. So this is linear, order n and n is this sum of the element, or sorry, the number of elements in each list. I said I was going to back my way into this. That gives me a way to merge things. So here's what merge sort would do.

    Merge sort takes this idea of divide and conquer, and it does the following: it says let's divide the list in half. There's the divide and conquer. And let's keep dividing each of those lists in half until we get down to something that's really easy to sort. What's the simplest thing to sort? A list of size one, right? So continue until we have singleton lists.

    Once I got a list of size one they're sorted, and then combine them. Combine them by doing emerge the sub-lists. And again, you see that flavor. I'm going to just keep dividing it up until I get something really easy, and then I'm going to combine. And this is different than binary search now, the combine is going to have to do some work.

    So, I'm giving you a piece of code that does this, and I'm going to come back to it in the second, but it's up there. But what I'd like to do is to try you sort sort of a little simulation of how this would work. And I was going to originally make the TAs come up here and do it, but I don't have enough t a's to do a full merge sort. So I'm hoping, so I also have these really high-tech props. I spent tons and tons of department money on them as you can see.

    I hope you can see this because I'm going to try and simulate what a merge sort does. I've got eight things I want to sort here, and those initially start out here at top level. The first step is divide them in half. All right? I'm not sure how to mark it here, remember I need to come back there. I'm not yet done. What do I do? Divide them in half again.

    You know, if I had like shells and peas here I could make some more money. What do I do? I divide them in half one more time. Let me cluster them because really what I have, sorry, separate them out. I've gone from one problem size eight down to eight problems of size one.

    At this stage I'm at my singleton case. So this is easy. What do I do? I merge. And the merge is, put them in order. What do I do next? Obvious thing, I merge these. And that as we saw was a nice linear operation. It's fun to do it upside down, and then one more merge which is I take the smallest elements of each one until I get to where I want.

    Wow aren't you impressed. No, don't please don't clap, not for that one. Now let me do it a second time to show you that -- I'm saying this poorly. Let me say it again. That's the general idea. What should you see out of that? I just kept sub-dividing down until I got really easy problems, and then I combine them back.

    I actually misled you slightly there or maybe a lot, because I did it in parallel. In fact, let me just shuffle these up a little bit. Really what's going to happen here, because this is a sequential computer, is that we're going to start off up here, at top level, we're going to divide into half, then we're going to do the complete subdivision and merge here before we ever come back and do this one. We're going to do a division here and then a division there. At that stage we can merge these, and then take this down, do the division merge and bring them back up.

    Let me show you an example by running that. I've got a little list I've made here called test. Let's run merge sort on it, and then we'll look at the code. OK, what I would like you to see is I've been printing out, as I went along, actually let's back up slightly and look at the code. There's merge sort. Takes in a list. What does it say to do? It says check to see if I'm in that base case. It's the list of length less than two. Is it one basically? In which case, just return a copy the list. That's the simple case.

    Otherwise, notice what it says to do. It's says find the mid-point and split the list in half. Copy of the back end, sorry, copy of the left side, copy of the right side. Run merge sort on those. By induction, if it does the right thing, I'm going to get back two lists, and I'm going to then merge them together. Notice what I'm going to do. I'm going to print here the list if we go into it, and print of the when we're done and then just return that.

    Merge up here. There's a little more code there. I'll let you just grok it but you can see it's basically doing what I did over there. Setting up two indices for the two sub-list, it's just walking down, finding the smallest element, putting it into a new list. When it gets to the end of one of the lists, it skips to the next part, and only one of these two pieces will get called because only one of them is going to have things leftovers. It's going to add the other pieces in.

    OK, if you look at that then, let's look at what happened when we ran this. We started off with a call with that list. Ah ha, split it in half. It's going down the left side of this. That got split in half, and that got split in half until I got to a list of one.

    Here's the first list of size one. There's the second list of size one. So I merged them. It's now in the right order, and that's coming from right there. Having done that, it goes back up and picks the second sub-list, which came from there. It's a down to base case, merges it. When these two merges are done, we're basically at a stage in that branch where we can now merge those two together, which gives us that, and it goes through the rest of it.

    A really nice algorithm. As I said, an example of divide and conquer. Notice here that it's different than the binary search case. We're certainly dividing down, but the combination now actually takes some work. I'll have to actually figure out how to put them back together. And that's a general thing you want to keep in mind when you're thinking about designing a divide and conquer kind of algorithm.

    You really want to get the power of dividing things up, but if you end up doing a ton of work at the combination stage, you may not have gained anything. So you really want to think about that trade off. All right, having said that, what's the complexity here?

    Boy, there's a dumb question, because I've been telling you for the last two lectures the complexity is n log n, but let's see if it really is. What's the complexity here? If we think about it, we start off with the problem of size n. What do we do? We split it into two problems of size n over 2. Those get split each into two problems of size n over 4, and we keep doing that until we get down to a level in this tree where we have only singletons left over.

    Once we're there, we have to do the merge. Notice what happens here. We said each of the merge operations was of order n. But n is different. Right? Down here, I've just got two things to merge, and then I've got things of size two to merge and then things of size four to merge. But notice a trade off. I have n operations if you like down there of size one. Up here I have n over two operations of size two. Up here I've got n over four operations of size four.

    So I always have to do a merge of n elements. How much time does that take? Well, we said it, right? Where did I put it? Right there, order n. So I have order n operations at each level in the tree. And then how many levels deep am I? Well, that's the divide, right? So how many levels do I have? Log n, because at each stage I'm cutting the problem in half. So I start off with n then it's n over two n over four n over eight.

    So I have n operations log n times, there we go, n log n. Took us a long time to get there, but it's a nice algorithm to have. Let me generalize this slightly. When we get a problem, a standard tool to try and attack it with is to say, is there some way to break this problem down into simpler, I shouldn't say simpler, smaller versions of the same problem. If I can do that, it's a good candidate for divide and conquer. And then the things I have to ask is how much of a division do I want to do? The obvious one is to divide it in half, but there may be cases where there are different divisions you want to have take place.

    The second question I want to ask is what's the base case? When do I get down to a problem that's small enough that it's basically trivial to solve? Here it was lists of size one. I could have stopped at lists of size two right. That's an easy comparison. Do one comparison and return one of two possible orders on it, but I need to decide that.

    And the third thing I need to decide is how do I combine? You know, point out to you in the binary search case, combination was trivial. The answer to the final search was just the answer all the way up. Here, a little more work, and that's why I'll come back to that idea.

    If I'm basically just squeezing jello, that is, I'm trying to make the problem simpler, but the combination turns out to be really complex, I've not gained anything.

    So things that are good candidates for divide and conquer are problems where it's easy to figure out how to divide down, and the combination is of little complexity. It would be nice if it was less than linear, but linear is nice because then I'm going to get that n log in kind of behavior. And if you ask the TAs in recitation tomorrow, they'll tell you that you see a lot of n log n algorithms in computer science. It's a very common class of algorithms, and it's very useful one to have.

    Now, one of the questions we could still ask is, right, we've got binary search, which has got this nice log behavior. If we can sort things, you know, we get this n log n behavior, and we got a n log n behavior overall. But can we actually do better in terms of searching.

    I'm going to show you one last technique. And in fact, we're going to put quotes around the word better, but it does better than even this kind of binary search, and that's a method called hashing. You've actually seen hashing, you just don't know it. Hashing is the the technique that's used in Python to represent dictionaries. Hashing is used when you actually come in to Logan Airport and Immigration or Homeland Security checks your picture against a database. Hashing is used every time you enter a password into a system.

    So what in the world is hashing? Well, let me start with a simple little example. Suppose I want to represent a collection of integers. This is an easy little example. And I promise you that the integers are never going to be anything other than between the range of zero to nine. OK, so it might be the collection of one and five. It might be two, three, four, eight. I mean some collection of integers, but I guarantee you it's between zero and nine.

    Here's the trick I can play. I can build -- I can't count -- I could build a list with spots for all of those elements, zero, one, two, three, four, five, six, seven, eight, nine. And then when I want to create my set, I could simply put a one everywhere that that integer falls.

    So if I wanted to represent, for example, this is the set two, six and eight, I put a one in those slots. This seems a little weird, but bear with me for second, in fact, I've given you a little piece a code to do it, which is the next piece of code on the hand out. So let's take a look at it for second.

    This little set of code here from create insert and number. What's create do? It says, given a low and a high range, in this case it would be zero to nine. I'm going to build a list. Right, you can see that little loop going through there. What am I doing? I'm creating a list with just that special symbol none in it. So I'm building the list. I'm returning that as my set. And then to create the object, I'll simply do a set of inserts. If I want the values two, six and eight in there, I would do an insert of two into that set, an insert of six into that set, and an insert of eight into the set. And what does it do? It marks a one in each of those spots.

    Now, what did I want to do? I wanted to check membership. I want to do search. Well that's simple. Given that representation and some value, I just say gee is it there? What's the order complexity here? I know I drive you nuts asking questions? What's the order complexity here? Quadratic, linear, log, constant? Any takers? I know I have the wrong glasses on the see hands up too, but...

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR: Who said it?

    STUDENT: Constant.

    PROFESSOR: Constant, why?

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR: Yes, thank you. All right, it is constant. You keep sitting back there where I can't get to you. Thank you very much. It has a constant. Remember we said we design lists so that the access, no matter where it was on the list was of constant time. That is another way of saying that looking up this thing here is constant. So this is constant time, order one.

    Come on, you know, representing sets of integers, this is pretty dumb. Suppose I want to have a set of characters. How could I do that? Well the idea of a hash, in fact, what's called a hash function is to have some way of mapping any kind of data into integers.

    So let's look at the second example, all right, -- I keep doing that -- this piece of code from here to here gives me a way of now creating a hash table of size 256. Ord as a built in python representation. There is lots of them around that takes any character and gives you back an integer.

    In fact, just to show that to you, if I go down here and I type ord, sorry, I did that wrong. Let me try again. We'll get to exceptions in a second. I give it some character. It gives me back an integer representing. It looks weird. Why is three come back to some other thing? That's the internal representation that python uses for this. If I give it some other character, yeah, it would help if I could type, give it some other character. It gives me back a representation.

    So now here's the idea. I build a list 256 elements long, and I fill it up with those special characters none. That's what create is going to do right here. And then hash character takes in any string or character, single character, gives me back a number. Notice what I do. If I want to create a set or a sequence representing these things, I simply insert into that list. It goes through and puts ones in the right place.

    And then, if I want to find out if something's there, I do the same thing. But notice now, hash is converting the input into an integer. So, what's the idea? If I know what my hash function does, it maps, in this case characters into a range zero to 256, which is zero to 255, I create a list that long, and I simply mark things. And my look up is still constant.

    Characters are simple. Suppose you want to represent sets of strings, well you basically just generalize the hash function. I think one of the classic ones for strings is called the Rabin-Karp algorithm. And it's simply the same idea that you have a mapping from your import into a set of integers.

    Wow, OK, maybe not so wow, but this is now constant. This is constant time access. So I can do searching in constant time which is great. Where's the penalty? What did I trade off here? Well I'm going to suggest that what I did was I really traded space for time. It makes me sound like an astro physicist somehow right?

    What do I mean by that? I have constant time access which is great, but I paid a price, which is I had to use up some space. In the case of integers it was easy. In the case of characters, so I have to give up a list of 256, no big deal.

    Imagine now you want to do faces. You've got a picture of somebody's face, it's a million pixels. Each pixel has a range of values from zero to 256. I want to hash a face with some function into an integer. I may not want to do the full range of this, but I may decide I have to use a lot of gigabytes of space in order to do a trade off.

    The reason I'm showing you this is it that this is a gain, a common trade off in computer science. That in many cases, I can gain efficiency if I'm willing to give up space. Having said that though, there may still be a problem, or there ought to be a problem that may be bugging you slightly, which is how do I guarantee that my hash function takes any input into exactly one spot in the storage space?

    The answer is I can't. OK, in the simple case of integers I can, but in the case of something more complex like faces or fingerprints or passwords for that matter, it's hard to design a hash function that has completely even distribution, meaning that it takes any input into exactly one output spot. So what you typically do and a hash case is you design your code to deal with that.

    You try to design -- actually I'm going to come back to that in a second. It's like you're trying to use a hash function that spread things out pretty evenly. But the places you store into in those lists may have to themselves have a small list in there, and when you go to check something, you may have to do a linear search through the elements in that list.

    The good news is the elements in any one spot in a hash table are likely to be a small number, three, four, five. So the search is really easy. You're not searching a million things. You're searching three or four things, but nonetheless, you have to do that trade off.

    The last thing I want to say about hashes are that they're actually really hard to create. There's been a lot of work done on these over the years, but in fact, it's pretty hard to invent a good hash function. So my advice to you is, if you want to use something was a hash, go to a library. Look up a good hash function.

    For strings, there's a classic set of them that work pretty well. For integers, there are some real simple ones. If there's something more complex, find a good hash function, but designing a really good hash function takes a lot of effort because you want it to have that even distribution. You'd like it to have as few duplicates if you like in each spot in the hash table for each one of the things that you use.

    Let me pull back for a second then. What have we done over the last three or four lectures? We've started introducing you to classes of algorithms. Things that I'd like you to be able to see are how to do some simple complexity analysis.

    Perhaps more importantly, how to recognize a kind of algorithm based on its properties and know what class it belongs to. This is a hint. If you like, leaning towards the next quiz, that you oughta be able to say that looks like a logarithmic algorithm because it's got a particular property. That looks like an n log n algorithm because it has a particular property.

    And the third thing we've done is we've given you now a set of sort of standard algorithms if you like. Root force, just walk through every possible case. It works well if the problem sizes are small. We've had, there are a number of variants of guess and check or hypothesize and test, where you try to guess the solution and then check it and use that to refine your search.

    Successive approximation, Newton-Raphson was one nice example, but there's a whole class of things that get closer and closer, reducing your errors as you go along.

    Divide and conquer and actually I guess in between there bi-section, which is really just a very difficult of successive approximation, but divide and conquer is a class of algorithm. These are tools that you want in your tool box. These are the kinds of algorithms that you should be able to recognize. And what I'd like you to begin to do is to look at a problem and say, gee, which kind of algorithm is most likely to be successful on this problem, and map it into that case.

    OK, starting next -- don't worry I'm not going to quit 36 minutes after -- I got one more topic for today. But jumping ahead, I'm going to skip in a second now to talk about one last linguistic thing from Python, but I want to preface Professor Guttag is going to pick up next week, and what we're going to start doing then is taking these classes of algorithms and start looking at much more complex algorithms. Things you're more likely to use in problems. Things like knapsack problems as we move ahead.

    But the tools you've seen so far are really the things that were going to see as we build those algorithms. OK, I want to spend the last portion of this lecture doing one last piece of linguistics stuff. One last little thing from Python, and that's to talk about exceptions.

    OK, you've actually seen exceptions a lot, you just didn't know that's what they were, because exceptions show up everywhere in Python. Let me give you a couple of examples. I'm going to clear some space here. Before I type in that expression, I get an error, right?

    So it's not defined. But in fact, what this did was it threw an exception. An exception is called a name error exception. It says you gave me something I didn't know how to deal. I'm going to throw it, or raise it, to use the right term to somebody in case they can handle it, but it's a particular kind of exception.

    I might do something like, remind you I have test. If I do this, try and get the 10th element of a list that's only eight long. I get what looks like an error, but it's actually throwing an exception. The exception is right there. It's an index error, that is it's trying to do something going beyond the range of what this thing could deal with.

    OK, you say, come on, I've seen these all the time. Every time I type something into my program, it does one of these things, right? When we're just interacting with idol, with the interactive editor or sorry, interactive environment if you like, that's what you expect. What's happening is that we're typing in something, an expression it doesn't know how to deal. It's raising the exception, but is this simply bubbling up at the top level saying you've got a problem.

    Suppose instead you're in the middle of some deep piece of code and you get one of these cases. It's kind of annoying if it throws it all the way back up to top level for you to fix. If it's truly a bug, that's the right thing to do. You want to catch it. But in many cases exceptions or things that, in fact, you as a program designer could have handled.

    So I'm going to distinguish in fact between un-handled exceptions, which are the things that we saw there, and handled exceptions. I'm going to show you in a second how to handle them, but let's look at an example. What do I mean by a handled exception?

    Well let's look at the next piece of code. OK, it's right here. It's called read float. We'll look at it in a second. Let me sort of set the stage up for this -- suppose I want to input -- I'm sorry I want you as a user to input a floating point number.

    We talked about things you could do to try make sure that happens. You could run through a little loop to say keep trying until you get one. But one of the ways I could deal with it is what's shown here. And what's this little loop say to do? This little loop says I'm going to write a function or procedures that takes in two messages.

    I'm going to run through a loop, and I'm going to request some input, which I'm going to read in with raw input. I'm going to store that into val. And as you might expect, I'm going to then try and see if I can convert that into a float. Oh wait a minute, that's a little different than what we did last time, right? Last time we checked the type and said if it is a float you're okay. If not, carry on.

    In this case what would happen? Well float is going to try and do the cohersion. It's going to try and turn it into a floating point number. If it does, I'm great, right. And I like just to return val. If it doesn't, floats going to throw or raise, to use the right term, an exception. It's going to say something like a type error.

    In fact, let's try it over here. I if I go over here, and I say float of three, it's going to do the conversion. But if I say turn this into a float, ah it throws a value error exception. It says it's a wrong kind of value that I've got. So I'm going to write a little piece of code that says if it gives me a float, I'm set, But if not, I'd like to have the code handle the exception. And that's what this funky tri-accept thing does. This is a tri-accept block and here's the flow of control that takes place in there.

    When I hit a tri-block. It's going to literally do that. It's going to try and execute the instructions. If it can successfully execute the instructions, it's going to skip past the except block and just carry on with the rest of the code. If, however, it raises an exception, that exception, at least in this case where it's a pure accept with no tags on it, is going to get, be like thrown directly to the except block, and it's going to try and execute that. So notice what's going to happen here, then. If I give it something that can be turned into a float, I come in here, I read the input, if it can be turned into a float, I'm going to just return the value and I'm set.

    If not, it's basically going to throw it to this point, in which case I'm going to print out an error message and oh yeah, I'm still in that while loop, so it's going to go around. So in fact, if I go here and, let me un-comment this and run the code. It says enter a float. And if I give it something that can be -- sorry, I've got, yes, never mind the grades crap. Where did I have that? Let me comment that out. Somehow it's appropriate in the middle of my lecture for it to say whoops at me but that wasn't what I intended. And we will try this again.

    OK, says it says enter a float. I give it something that can be converted into a float, it says fine. I'm going to go back and run it again though. If I run it again, it says enter a float. Ah ha, it goes into that accept portion, prints out a message, and goes back around the while loop to say try again. And it's going to keep doing this until I give it something that does serve as a float.

    Right, so an exception then has this format that I can control as a programmer. Why would I want to use this? Well some things I can actually expect may happen and I want to handle them. The float example is a simple one. I'm going to generalize in a second.

    Here's a better example. I'm writing a piece of code that wants to input a file. I can certainly imagine something that says give me the file name, I'm going to do something with it. I can't guarantee that the file may exist under that name, but I know that's something that might occur. So a nice way to handle it is to write it as an exception that says, here's what I want to do if I get the file. But just in case the file name is not there, here's what I want to do in that case to handle it. Let me specify what the exception should do.

    In the example I just wrote here, this is pretty trivial, right. OK, I'm trying to input floats. I could generalize this pretty nicely. Imagine the same kind of idea where I want to simply say I want to take input of anything and try and see how to make sure I get the right kind of thing. I want to make it polymorphic.

    Well that's pretty easy to do. That is basically the next example, right here. In fact, let me comment this one out. I can do exactly the same kind of thing. Now what I'm going to try and do is read in a set of values, but I'm going to give a type of value as well as the messages. The format is the same. I'm going to ask for some input, and then I am going to use that procedure to check, is this the right type of value. And I'm trying to use that to do the coercion if you like. Same thing if it works, I'm going to skip that, if it not, it's going to throw the exception.

    Why is this much nice? Well, that's a handy piece of code. Because imagine I've got that now, and I can now store that away in some file name, input dot p y, and import into every one of my procedure functions, pardon me, my files of procedures, because it's a standard way of now giving me the input.

    OK, so far though, I've just shown you what happens inside a peace a code. It raises an exception. It goes to that accept clause. We don't have to use it just inside of one place. We can actually use it more generally.

    And that gets me to the last example I wanted to show you. Let me uncomment this. Let's take a look at this code. This looks like a handy piece of code to have given what we just recently did to you. All right, get grades. It's a little function that's going to say give me a file name, and I'm going to go off and open that up and bind it to a local variable. And if it's successful, then I'd just like to go off and do some things like turn it into a list so I can compute average score or distributions or something else. I don't really care what's going on here.

    Notice though what I've done. Open, it doesn't succeed is going to raise a particular kind of exception called I O error. And so I've done a little bit different things here which is I put the accept part of the block with I O error. What does that say? It says if in the code up here I get an exception of that sort, I'm going to go to this place to handle it.

    On the other hand, if I'm inside this procedure and some other exception is raised, it's not tagged by that one, it's going to raise it up the chain. If that procedure was called by some other procedure it's going to say is there an exception block in there that can handle that. If not, I am going to keep going up the chain until eventually I get to the top level. And you can see that down here. I'm going to run this in a second. This is just a piece of code where I'm going to say, gee, if I can get the grades, do something, if not carry on.

    And if I go ahead and run this -- now it's going to say woops, at me. What happened? I'm down here and try, I'm trying do get grades, which is a call to that function, which is not bound in my computer. That says it's in here. It's in this tri-block. It raised an exception, but it wasn't and I O error. So it passes it back, past this exception, up to this level, which gets to that exception.

    Let me say this a little bit better then. I can write exceptions inside a piece of code. Try this, if it doesn't work I can have an exception that catches any error at that level. Or I can say catch only these kinds of errors at that level, otherwise pass them up the chain. And that exception will keep getting passed up the chain of calls until it either gets to the top level, in which case it looks like what you see all the time. It looks like an error, but it tells you what the error came from, or it gets an exception , it can deal with it.

    OK, so the last thing to say about this is what's the difference between an exception and an assert? We introduced asserts earlier on. You've actually seen them in some pieces of code, so what's the difference between the two of them? Well here's my way of describing it. The goal of an assert, or an assert statement, is basically to say, look, you can make sure that my function is going to give this kind of result if you give me inputs of a particular type. Sorry, wrong way of saying it. If you give me inputs that satisfy some particular constraints.

    That was the kind of thing you saw. Asserts said here are some conditions to test. If they're true, I'm going to let the rest of the code run. If not, I'm going to throw an error. So the assertion is basically saying we got some pre-conditions, those are the clauses inside the assert that have to be true, and there's a post condition. and in essence, what the assert is saying is, or rather the programmer is saying using the assert is, if you give me input that satisfies the preconditions, I'm guaranteeing to you that my code is going to give you something that meets the post condition. It's going to do the right thing.

    And as a consequence, as you saw with the asserts, if the preconditions aren't true, it throws an error. It goes back up the top level saying stop operation immediately and goes back up the top level. Asserts in fact are nice in the sense that they let you check conditions at debugging time or testing time. So you can actually use them to see where your code is going. An exception, when you use an exception, basically what you're saying is, look, you can do anything you want with my function, and you can be sure that I'm going to tell you if something is going wrong. And in many cases I'm going to handle it myself.

    So as much as possible, the exceptions are going to try to handle unexpected things, actually wrong term, you expected them, but not what the user did. It's going to try to handle conditions other than the normal ones itself. So you can use the thing in anyway. If it can't, it's going to try and throw it to somebody else to handle, and only if there is no handler for that unexpected condition, will it come up to top level.

    So, summarizing better, assert is something you put in to say to the user, make sure you're giving me input of this type, but I'm going to guarantee you the rest of the code works correctly. Exceptions and exception handlers are saying, here are the odd cases that I might see and here's what I'd like to do in those cases in order to try and be able to deal with them.

    Last thing to say is why would you want to have exceptions? Well, let's go back to that case of inputting a simple little floating point. If I'm expecting mostly numbers in, I can certainly try and do the coercion. I could have done that just doing the coercion.

    The problem is, I want to know if, in fact, I've got something that's not of the form I expect. I'm much better having an exception get handled at the time of input than to let that prop -- that value rather propagate through a whole bunch of code until eventually it hits an error 17 calls later, and you have no clue where it came from.

    So the exceptions are useful when you want to have the ability to say, I expect in general this kind of behavior, but I do know there are some other things that might happen and here's what I'd like to do in each one of those cases. But I do want to make sure that I don't let a value that I'm not expecting pass through.

    That goes back to that idea of sort of discipline coding. It's easy to have assumptions about what you think are going to come into the program when you writ it. If you really know what they are use them as search, but if you think there's going to be some flexibility, you want to prevent the user getting trapped in a bad spot, and exceptions as a consequence are a good thing to use.
  • 18
    Assignment 6: Word games 2
    6 pages
  • 19
    MIT Lecture 11: Testing and Debugging
    48:59
    Testing and debugging. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: One of the things that you should probably have noticed is as we're moving in the terms, the problem sets are getting less well defined. line And I've seen a lot of email traffic of the nature of what should we do with this, what should we do with that? For example, suppose the computer player runs out of time. Or the person runs out of time playing the game. Should it stop right away? Should it just give them zero score? That's left out of the problem set. In part because one of the things were trying to accomplish is to have you folks start noticing ambiguities in problem statements. Because that's life in computing. And so this is not like a math problem set or a physics problem set. Or, like a high school physics lab, where we all know what the answer should be, and you could fake your lab results anyway. These are things where you're going to have to kind of figure it out. And for most of these things, all we ask is that you do something reasonable, and that you describe what it is you're doing. So I don't much care, for example, whether you give the human players zero points for playing after the time runs out. Or you say you're done when the time runs out. Any of that -- thank you, Sheila -- is ok with me. Whatever. What I don't want your program to do is crash when that happens. Or run forever. Figure out something reasonable and do it. And again, we'll see this as an increasing trend as we work our way through the term.

    The exception will be the next problem set, which will come out Friday. Because that's not a programming problem. It's a problem, as you'll see, designed to give you some practice at dealing with some of the, dare I say, more theoretical concepts we've covered in class. Like algorithmic complexity. That are not readily dealt with in a prograing problem. It also deals with issues of some of the subtleties of things like aliasing. So there's no programming. And in fact, we're not even going to ask you to hand it in. It's a problem set where we've worked pretty hard to write some problems that we think will provide you with a good learning experience. And you should just do it to learn the material. We'll help you, if you need -- to see the TAs because you can't do them, by all means make sure you get some help. So I'm not suggesting that it's an optional problem set that you shouldn't do. Because you will come to regret it if you don't do it. But we're not going to grade it. And since we're not going to grade it, it seems kind of unfair to ask you to hand it in. So it's a short problem set, but make sure you know how to do those problems.

    OK. Today, for the rest of the lecture, we're going to take a break from the topic of algorithms, and computation, and things of the sort. And do something pretty pragmatic. And we're going to talk briefly about testing. And at considerable length about debugging. I have tried to give this lecture at the beginning of the term, at the end of the term. Now I'm trying it kind of a third of the way, or middle of the term. I never know the right time to give it. These are sort of pragmatic hints that are useful. I suspect all of you have found that debugging can be a frustrating activity. My hope is that at this point, you've experienced enough frustration that the kind of pragmatic hints I'm going to talk about will not be, "yeah sure, of course." But they'll actually make sense to you. We'll see.

    OK. In a perfect world, the weather would always be like it's been this week. The M in MIT would stand for Maui, instead of Massachusetts. Quantum physics would be easier to understand. All the supreme court justices would share our social values. And most importantly, our programs would work the first time we typed them. By now you may have noticed that we do not live in an ideal world. At least one of those things I mentioned is not true. I'm only going to address the last one. Why our programs don't work. And I will leave the supreme court up to the rest of you. There is an election coming up.

    Alright, First a few definitions. Things I want to make sure we all understand what they mean. Validation is a process. And I want to emphasize the word process. Designed to uncover problems and increase confidence that our program does what we think it's intended to do. I want to emphasize that it will increase our confidence, but we can never really be sure we've got it nailed. And so it's a process that goes on and on. And I also want to emphasize that a big piece of it is to uncover problems. So we need to have a method not designed to give us unwarranted confidence. But in fact warranted confidence in our programs. It's typically a combination of two things. Testing and reasoning. Testing, we run our program on some set of inputs. And check the answers, and say yeah, that's what we expected. But it also involves reasoning. About why that's an appropriate set of inputs to test it on it. Have we tested it on enough inputs? Maybe just reading the code and studying it and convincing ourselves that works. So we do both of those as part of the validation process. And we'll talk about all of this as we go along.

    Debugging is a different process. And that's basically the process of ascertaining why the program is not working. Why it's failing to work properly. So validation says whoops, it's not working. And now we try and figure out why not. And then of course, once we figure out why not, we try and fix it. but today I'm going to emphasize not how do you fix it, but how do you find out what's wrong. Usually when you know why it's not working, it's obvious what you have to do to make it work.

    There are two aspects of it. Thus far, the problem sets have mostly focused on function. Does it exhibit the functional behavior? Does it give you the answer that you expected it to give? Often, in practical problems, you'll spend just as much time doing performance debugging. Why is it slow? Why is it not getting the answer as fast as I want it to? And in fact, in a lot of industry -- for example, if you're working on building a computer game, you'll discover that in fact the people working the game will spend more time on performance debugging than on getting it to do the right thing. Trying to make it do it fast enough. Or get to run on the right processor.

    Some other terms we've talked about is defensive programming. And we've been weaving that pretty consistently throughout the term. And that's basically writing your programs in such a way that it will facilitate both validation and debugging. And we've talked about a lot of ways we do that. One of the most important things we do is we use assert statements so that we catch problems early. We write specifications of our functions. We modularize things. And we'll come back to this. As every time we introduce a new programming concept, we'll relate it back, as we have been doing consistently, to defensive programming.

    So one of the things I want you to notice here is that testing and debugging are not the same thing. When we test, we compare an input output pair to a specification. When we debug, we study the events that led to an error. I'll return to testing later in the term. But I do want to make a couple of quick remarks with very broad strokes.

    There are basically two classes of testing. There's unit testing, where we validate each piece of the program independently. Thus far, for us it's been testing individual functions. Later in the term, we'll talk about unit testing of classes. The other kind of testing is integration testing. Where we put our whole program together, and we say does the whole thing work? People tend to want to rush in and do this right away. That's usually a big mistake. Because usually it doesn't work. And so one of the things that I think is very important is to always begin by testing each unit. So before I try and run my program, I test each part of it independently. And that's because it's easier to test small things than big things. And it's easier to debug small things than big things. Eventually, it's a big program, I run it. It never works the first time if it's a big program. And I end up going back and doing unit testing anyway, to try and figure out why it doesn't work. So over the years, I've just convinced myself I might as well start where I'm going to end up.

    What's so hard about testing? Why is testing always a challenge? Well, you could just try it and see if it works, right? That's what testing is all about. So we could look at something small. Just write a program to find the max of x and y. Where x and y are floats. However many quotes I need. Well, just see if it works. Let's test it in all possible combinations of x and y and see if we get the right answer. Well, as Carl Sagan would have said, there are billions and billions of tests we would have to do. Or maybe it's billions and billions and billions. Pretty impractical. And it's hard to imagine a simpler program than this. So we very quickly realize that exhaustive testing is just never feasible for an interesting program.

    So as we look at testing, what we have to find is what's called a test suite. A test suite is small enough so that we can test it in a reasonable amount of time. But also large enough to give us some confidence. Later in the term, we'll spend part of a lecture talking about, how do we find such a test suite? A test suite that will make us feel good about things. For now, I just want you to be aware that you're always doing this balancing act.

    So let's assume we've run our test suite. And, sad to say, at least one of our tests produced an output that we were unhappy with. It took it too long to generate the output. Or more likely, it was just the wrong output. That gets us to debugging.

    So a word about debugging. Where did the name come from? Well here's a fun story, at least. This was one of the very first recorded bugs in the history of computation. Recorded September 9th, 1947, in case you're interested. This was the lab book of Grace Murray Hopper. Later Admiral Grace Murray Hopper. The first female admiral in the U.S. navy. Who was also one of the word's first programmers. So she was trying to write this program, and it didn't work. It was a complicated program. It was computing the arctan. So you can imagine, right? You had a whole team of people trying to figure out how to do arctans. Times were different in those days. And they tried to run it, and it ran a long time. Then it basically stopped. Then they started the cosine tape. That didn't work. Well they couldn't figure out what was wrong. And they spent a long time trying to debug the program. They didn't apparently call it debugging. And then they found the problem. In relay number 70, a moth had been trapped. And the relay had closed on the poor creature, crushing it to death. The defense department didn't care about the loss of a moth. But they did care about the fact that the relay was now stuck. It didn't work. They removed the moth, and the program worked. And you'll see at the bottom, it says the first actual case of a bug being found. And they were very proud of themselves. Now it's a wonderful story, and it is true. After all, Grace wouldn't have lied. But it's not the first use of the term "bug." And as you'll see by your handout, I've attempted tend to trace it. And the first one I could find was in 1896. In a handbook on electricity.

    Alright. Now debugging is a learned skill. Nobody does it well instinctively. And a large part of being a good programmer, or learning to be a good programmer, is learning how to debug. And it's one of these things where it's harder. It's slow, slow, and you suddenly have an epiphany. And you now get the hang of it. And I'm hoping that today's lecture will help you learn faster. The nice thing, is once you learn to debug programs, you will discover it's a transferable skill. And you can use it to debug other complex systems. So for example, a laboratory experience. Why isn't this experiment working? There's a lecture I've given several times at hospitals, to doctors, on doing diagnosis of complex multi illnesses. And I go through it, and almost the same kind of stuff I'm going to talk to you about, about debugging. Explaining that it's really a process of engineering.

    So I want to start by disabusing you of a couple of myths about bugs. So myth one is that bugs crawl into programs. Well it may have been true in the old days, when bugs flew or crawled into relays. It's not true now. If there is a bug in the program, it's there for only one reason. You put it there. i.e. you made a mistake. So we like to call them bugs, because it doesn't make us feel stupid. But in fact, a better word would be mistake.

    Another myth is that the bugs breed. They do not. If there are multiple bugs in the program, it's because you made multiple mistakes. Not because you made one or two and they mated and produced many more bugs. It doesn't work that way. That's a good thing. Typically, even though they don't breed, there are many bugs. And keep in mind that the goal of debugging is not to eliminate one bug. The goal is to move towards a bug free program. I emphasize this because it often leads to a different debugging strategy. People can get hung up on sort of hunting these things down, and stamping them out, one at a time. And it's a little bit like playing Whack-a-Mole. Right? They keep jumping up at you. So the goal is to figure out a way to stamp them all out.

    Now, should you be proud when you find a bug? I've had graduate students come to me and say I found a bug in my program. And they're really proud of themselves. And depending on the mood I'm in, I either congratulate them, or I say ah, you screwed up, huh? Then you had to fix it. If you find a bug, it probably means there are more of them. So you ought to be a little bit careful. The story I've heard told is you're at somebody's house for dinner, and you're sitting at the dining room table, then you hear a [BANG]. And then your hostess walks in with the turkey in a tray, and says, "I killed the last cockroach." Well it wouldn't increase my appetite, at least. So be worried about it.

    For at least four decades, people have been building tools called debuggers. Things to help you find bugs. And there are some built into Idol. My personal view is most of them are not worth the trouble. The two best debugging tools are the same now that they have almost always been. And they are the print statement, and reading. There is no substitute for reading your code. Getting good at this is probably the single most important skill for debugging. And people are often resistant to that. They'd rather single step it through using Idol or something, than just read it and try and figure things out. The most important thing to remember when you're doing all of this is to be systematic. That's what distinguishes good debuggers from bad debuggers. Good debuggers have evolved a way of systematically hunting for the bugs. And what they're doing as they hunt, is they're reducing the search space. And they do that to localize the source of the problem. We've already spent a fair amount of time this semester talking about searches. Algorithms for searching. Debugging is simply a search process. When you are searching a list to see whether it has an element, you don't randomly probe the list, hoping to find whether or not it's there. You find some way of systematically going through the list. Yet, I often see people, when they're debugging, proceeding at what, to me, looks almost like a random fashion of looking for the bug. That is a problem that may not terminate. So you need to be careful.

    So let's talk about how we go about being systematic, as we do this. So debugging starts when we find out that there exists a problem. So the first thing to do is to study the program text, and ask how could it have produced this result? So there's something subtle about the way I've worded this. I didn't ask, why didn't it produce the result I wanted it to produce? Which is sort of the question we'd immediately like to ask. Instead, I asked why did it produce the result it did. So I'm not asking myself what's wrong? Or how could I make it right? I'm asking how could have done this? I didn't expect it to do this. If you understand why it did what it did, you're half way there.

    The next big question you ask, is it part of a family? This gets back to the question of trying to get the program to be bug free. So for example, oh, it did this because it was aliasing, where I hadn't expected it. Or some side effect of some mutation with lists. And then I say, oh you know I've used lists all over this program. I'll bet this isn't the only place where I've made this mistake. So you say well, rather than rushing off and fixing this one bug, let me pull back and ask, is this a systematic mistake that I've made throughout the program? And if so, let's fix them all at once, rather than one at a time.

    And that gets me to the final question. How to fix it. When I think about debugging, I think about it in terms of what you learned in high school as the scientific method. Actually, I should ask the question. Maybe I'm dating myself. Do they still teach the scientific method in high school? Yes, alright good. All is not lost with the American educational system. So what does the scientific method tell us to do? Well it says you first start by studying the available data. In this case, the available data are the test results. And by the way, I mean all the test results. Not just the one where it didn't work, but also the ones where it did. Because maybe the program worked on some inputs and not on others. And maybe by understanding why it worked on a and not on b, you'll get a lot of insight that you won't if you just focus on the bug. You'll also feel a little bit better knowing your program works on at least something. The other big piece of available data we have is, of course, the program text. As you the study the program text, keep in mind that you don't understand it. Because if you really did, you wouldn't have the bug. So read it with sort of a skeptical eye. You then form a hypothesis consistent with all the data. Not just some of the data, but all of the data. And then you design and run a repeatable experiment.

    Now what is the thing we learned in high school about how to design these experiments? What must this experiment have the potential to do, to be a valid scientific experiment? Somebody? What's the key thing? It must have the potential to refute the hypothesis. It's not a valid experiment if it has no chance of showing that my hypothesis is flawed. Otherwise why bother running it? So it has to have that. Typically it's nice if it can have useful intermediate results. Not just one answer at the end. So we can sort of check the progress of the code. And we must know what the result is supposed to be. Typically when you run an experiment, you say, and I think the answer will be x. If it's not x, you've refuted the hypothesis. This is the place where people typically slip up in debugging. They don't think in advance what they expect the result to be. And therefore, they are not systematic about interpreting the results. So when someone comes to me, and they're about to do a test, I ask them, what do you expect your program to do? And if they can't answer that question, I say well, before you even run it, have an answer to that.

    Why might repeatability be an issue? Well as we'll see later in the term, we're going to use a lot of randomness in a lot of our programs. Where we essentially do the equivalent of flipping coins or rolling dice. And so the program may do different things on different runs. We'll see a lot of that, because it's used a lot in modern computing. And so you have to figure out how to take that randomness out of the experiment. And yet get a valid test. Sometimes it can be timing. If you're running multiple processes. That's why your operating systems and your personal computers often crash for no apparent reason. Just because two things happen to, once in a while, occur at the same time. And often there's human input. And people have to type things out of it. So you want to get rid of that. And we'll talk more about this later. Particularly when we get to using randomness. About how to debug programs where random choices are being made.

    Now let's think about designing the experiment itself. The goal here, there are two goals. Or more than two. One is to find the simplest input that will provoke the bug. So it's often the case that a program will run a long time, and then suddenly a bug will show up. But you don't want to have to run it a long time, every time you have a hypothesis. So you try and find a smaller input that will produce the problem. So if your word game doesn't work when the words are 12 letters long, instead of continuing to debug 12 letter hands, see if you can make it fail on a three letter hand. If you can figure out why fails on three letters instead of 12, you'll be more than half way to solving the problem. What I typically do is I start with the input that provoked the problem, and I keep making it smaller and smaller. And see if I can't get it to show up.

    The other thing you want to do is find the part of the program that is most likely at fault. In both of these cases, I strongly recommend binary search. We've talked about this binary search a lot already. Again, the trick is, if you can get rid of half of the data at each shot, or half of the code at each shot., you'll quickly converge on where the problem is.

    So I now want to work through an example where we can see this happening. So this is the example on the handout. I've got a little program called Silly. And it's called Silly because it's really a rather ugly program. It's certainly not the right way to write a program to do this. But it will let us illustrate a few points. So the trick, what we're going to go through here, is this whole scientific process. And see what's going on. So let's try running Silly. So this is to test whether a list is a palindrome. So we'll put one as the first element, maybe a is the second element. And one is the third element. And just return, it's done. It is a palindrome. That make sense. The list one a one reads the same from the front or from the back. So that's good. Making some progress. Let's try it again. And now let's do one, a, two. Whoops. It tells me it is a palindrome. Well, it isn't really. I have a bug. Alright. Now what do I do? Well I'm going to use binary search to see if I can't find this bug. As I go through, I'm going to try and eliminate half of the code at each step. And the way I'm going to do that is by printing intermediate values, as I go part way through the code. I'm going to try and predict what the value is going to be. And then see if, indeed, I get what I predicted.

    Now, as I do this, I'm going to use binary search. I'm going to start somewhere near the middle of the code. Again, a lot of times, people don't do that. And they'll test an intermediate value near the end or near the beginning. Kind of in the hope of getting there in one shot. And that's like kind of hoping that the element you're searching for is the first in the list and the last in the list. Maybe. But part of the process of being systematic is not assuming that I'm going to get a lucky guess. But not even thinking really hard at this point. But just pruning the search space. Getting rid of half at each step.

    Alright. So let's start with the bisection. So we're going to choose a point about in the middle of my program. That's close to the middle. It might even be the middle. And we're going to see, well all right. The only thing I've done in this part of the program, now I'm going to go and read the code, is I've gotten the user to input a bunch of data. And built up the list corresponding to the three items that the user entered. So the only intermediate value I have here is really res. So I'm going to, just so when I'm finished I know what it is that I think I've printed. But in fact maybe I'll do even more than that here. Let me say what I think it should be. And then we'll see if it is. So I think I put in one a two, right? Or one a two? So it should be something like one, a, two. So I predicted what answer I'm expecting to get. And I've put it in my debugging code. And now I'll run it and see what we get. We'll save it. Well all right, a syntax error. This happens. And there's a syntax error. I see. Because I've got a quote in a quote. Alright I'm just going to do that.

    What I expected. So what have I learned? I've learned that with high probability, the error is not in the first part of the program. So I can now ignore that. So now I have these six lines. So we'll try and go in the middle of that. See if we can find it here. And notice, by the way, that I commented out the previous debugging line, rather than got rid of it. Since I'm not sure I won't need to go back to it. So what should I look at here? Well there are a couple of interesting intermediate values here, right? There's tmp. And there's res. Never type kneeling. Right? I find something to tmp. And I need to make sure maybe I haven't messed up res. Now it would be easy to assume, don't bother looking at [UNINTELLIGIBLE]. Because the code doesn't change res. Well remember, that I started this with a bug. That means it was something I didn't understand. So I'm going to be cautious and systematic. And say let's just print them both. And see whether they're okay.

    Now, let's do this. So it says tmp is two a one, and res is two a one. Well let's think it. Is this what we wanted, here? What's the basic idea behind this program? How is it attempting to work? Well what it's attempting to do, and now is when I have to stand back and form a hypothesis and think about what's going on, is it gets in the list, it reverses the list, and then sees whether the list and the reverse were identical. If so it was a palindrome, otherwise it wasn't. So I've now done this, and what do you think? Is this good or bad? Is this what I should be getting? No. What's wrong? Somebody? yeah.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR: Yeah. Somehow I wanted to -- Got to work on those hands. I didn't want to change res. So, I now know that the bug has got to be between these two print statements. I'm narrowing it down. It's getting a little silly, but you know I'm going to really be persistent and just follow the rules here of binary search, rather than jumping to conclusions. Well clearly what I probably want to do here is what? Print these same two things. See what I get. Whoops. I have to, of course, do that. Otherwise it just tells me that Silly happens to be a function. Alright. How do I feel about this result? I feel pretty good here. Right? The idea was to make a copy of res and temp. And sure enough, they're both the same. What I expected them to be. So I know the bug is not above. Now I'm really honing in. I now know it's got to be between these two statements. So let's put it there. Aha. It's gone wrong. So now I've narrowed the bug down to one place. I know exactly which statement it's in. So something has happened there that wasn't what I expected. Who wants to tell me what that bug is? Yeah?

    STUDENT: [UNINTELLIGIBLE].

    PROFESSOR: Right. Bad throw, good catch. So this is a classic error. I've not made a copy of the list. I've got an alias of the list. This was the thing that tripped up many of you on the quiz. And really what I should have done is this. Now we'll try it. Ha. It's not a palindrome. So small silly little exercise, but I'm hoping that you've sort of seen how by being patient. Patience is an important part of the debugging process. I have not rushed. I've calmly and slowly narrowed the search. Found where the statement is, and then fixed it. And now I'm going to go hunt through the rest of my code to look for places where I used assignment, when I should have use cloning as part of the assignment. The bug, the family here, is failure to clone when I should have cloned. Thursday we'll talk a little bit more about what to do once we've found the bug, and then back to algorithms.
  • 20
    MIT Lecture 12: Debugging, Knapsack Problem, Introduction to Dynamic Programming
    49:47
    More about debugging, knapsack problem, introduction to dynamic programming. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: I want to take a few minutes at the start of today's lecture to wrap up a few more things about debugging. And then I'll move on to the main event. Just a few small points. First of all, keep in mind that the bug is probably not where you think it is. Because if a bug were there, you'd have already found it.

    There are some simple things that you should always look at when you're looking for bugs. Reversed order of arguments. You might have a function that takes to floats and you just passed them in the wrong order. You may have noticed that when Professor Grimson and I used examples in class, we were pretty careful how we name our parameters. And I often end up using the same names for the actuals and the formals. And this just helps me not get confused about what order to do things in.

    Spelling. These are just dumb little things to look for. Did you spell all the identifiers the way you think you did. The problem is, when you read code you see what you expect to see. And if you've typed in l when you should've typed a 1, or a 1 when you should've typed an l, you're likely to miss it. If you've made a mistake with upper and lower case. Things like that. So just be very careful looking at that.

    Initialization. A very common bug is to initialize a variable. Go through a loop, and then forget to reinitialize it when it needs to be reinitialized again. So it's initialized outside the loop when it should be initialized inside the loop. Or conversely, inside the loop when it should be outside the loop. So look carefully at when variables are being initialized.

    Object versus value equality. Particularly when you use double equals, are you asking whether you've got the same object or the same value? They mean very different things, as we have seen. Keep track of that.

    Related to that is aliasing. And you'll see that, by the way, on the problem set that we're posting tomorrow. Where we're going to ask you to just look at some code, that there's some issues there with the fact that you alias things, maybe on purpose, maybe on accident. And by that I mean two different ways to get to the same value, the same object. Frequently that introduces bugs into the program.

    A particular instance of that is deep versus shallow copy. When you decide to make a copy of something like a list, you have to think hard about are you only copying the top level of the list, but if it's, say, a list that contains a list, are you also getting a new copy of everything it contains? It doesn't matter if it's a list of integers or a list of strings or a list of tuples, but it matters a lot if it's a list of lists. Or a list of anything that could be mutable. So when you copy, what are you getting? When you copy a dictionary, for example. Think through all of that.

    And again, a related problem that people run into is side-effects. You call a function and it returns a value, but maybe it, on purpose or on accident, modifies one of the actual parameters. So you could each make a list of things. Every experienced programmer over time develops a personal model of the mistakes they make. I know the kinds of things I get wrong. And so when I'm looking for a bug, I always say, oh, have you done this dumb thing you always do. So be a little bit introspective. Keep track of the mistakes you make. And if your program doesn't work, that should be your first guess.

    Couple of other hints. Keep a record of what you've tried. People will look at things and they'll come in and that'll talk to a TA, and the TA will say, did you try this? And the person will say, I don't know. That leads you to end up doing the same thing over and over again. This gets back to my main theme of Tuesday, which is be systematic. Know what you've already tried, don't waste your time trying it again.

    Think about reconsidering your assumptions. A huge one is, are you actually running the code you're looking at in your editor. This is a mistake I make all the time in Python. I sit there in idle, and I edit some code. I then click on the shell, rather than hitting F5, try and run it and say, it's not doing what I thought I should do. And I'll sit there staring at the output, staring at the code, not realizing that that output was not produced by that code. It's a mistake I've learned that I make. It's an easy one to make in Python. Lots of assumptions like that. You thought you knew what the built-in function sort did, method sort did. Well, does it do what you think it does? Does append do what you think it does? Go back and say, alright, I've obviously, some assumption is not right.

    When you're debugging somebody else's code, debug the code, not the comments. I've often made the mistake of believing the comments somebody wrote in your code about what some function does. Sometimes it's good to believe it. But sometimes you have to read it.

    Important thing. When it gets really tough, get help. Swallow your pride. Any one of my graduate students will tell you that from time to time I walk into their office and say, do you have a minute? And if they foolishly say yes, I drag them back to my office and say I'm stuck on this bug, what am I doing wrong? And it's amazing what -- well, a, they're smarter than I am which helps. But even if they weren't smarter than I am, just a fresh set of eyes. I've read through the same thing twenty times and I've missed something obvious. Someone who's never seen it before looks at and says, did you really mean to do this? Are you sure you didn't want to do for i in range of list, rather than for i in list? That sort of thing. Makes a big difference to just get that set of eyes.

    And in particular, try and explain to somebody else what you think the program is doing. So I'll sit there with the student and I'll say, I'm going to try and explain to you why what I think this program is doing. And probably 80% of the time, halfway through my explanation, I say, oh, never mind. I'm embarrassed. And I send them away. Because just the act of explaining it to him or her has helped me understand it. So when you're really stuck, just get somebody. And try and explain to them why you think your program is doing what it's doing.

    Another good thing to do when you're really stuck is walk away. Take a break. Come back and look at it with fresh eyes of your own. Effectively, what you're doing here is perhaps trading latency for efficiency. It may, if you go back, come back and two hours later, maybe you'll, it'll take you at least two hours to have found the bugs, because you've been away for two hours. And maybe if you'd stayed there and worked really hard you'd have found it an hour and fifty eight minutes. But is it really worth the two minutes? This is another reason, by the way, to start it long before it's due. That you actually have the leisure to walk away.

    All right. What do you do when you've found the bug and you need to fix it? Remember the old saw, haste makes waste. I don't know this but I'll bet Benjamin Franklin said this. Don't rush into anything. Think about the fix, don't make the first change that comes to mind and see if it works. Ask yourself, will it fix all of the symptoms you've seen? Or if not, are the symptoms independent, will at least fix some of them? What are the ramifications of the proposed change. Will it break other things? That's a big issue. You can fix one thing, you break something else. So think through what this change might break.

    Does it allow you to tidy up other things? This is important, I think, that code should not always grow. We all have a tendency to fix code by adding code. And the program just gets bigger and bigger and bigger. The more code you have, the harder it is to get it right. So, sometimes, what you need to, so you just pull back. And say, well, let me just tidy things up. That, by the way, is also a good debugging tool. Sometimes when I'm really stuck, I say, alright let me stop looking for the bug. Let me just clean up the code a little bit. Make my program prettier. And in the process of tidying it up and making it prettier, I'll often by accident find the bug. So it's a good trick to remember.

    Finally, and this is very important, make sure that you can revert. There's nothing more frustrating then spending four hours debugging, and realizing at the end of four hours your program is worse than it was when you started. And you can't get back to where it was when you started.

    So if you look at one of my directories, you'll find that I've been pretty anal about saving old versions. I can always pretty much get back to some place I've been. So if I've made a set of changes and I realize I've broken something that used to work, I can find a version of the code in which it used to work, and figure out what was going on there. Disk space is cheap. Don't hesitate to save your old versions. It's a good thing.

    Alright, that's the end of my, maybe sermon is the right word on debugging. Polemic, I don't know. I hope it will be helpful. And I hope you'll remember some of these things as you try and get your programs to work.

    I now want to get back to algorithms. And where we've sort of been for a while, and where we will stay for a while. More fundamentally, I want to get back to what I think of as the main theme of 6.00, which is taking a problem and finding some way to formulate the problem so that we can use computing to help us get an answer.

    And in particular, we're going to spend the next few lectures looking at a class of problems known as optimization problems. In general, every optimization problem is the same. It has two parts. Some function that you're either attempting to maximize or minimize the value of. And these are duals. And some set of constraints that must be honored. Possibly an empty set of constraints.

    So what are some of the classic optimization problems? Well one of the most well-known is the so-called shortest path problem. Probably almost every one of you has used a shortest path algorithm. For example, if you go to Mapquest, or Google Maps and ask how do I get from here to there. You give it the function, probably in this case to minimize, and it gives you a choice of functions. Minimize time, minimize distance.

    And maybe you give it a set of constraints. I don't want to drive on highways. And it tries to find the shortest way, subject to those constraints, to get from Point A to Point B. And there are many, many other instances of this kind of thing. And tomorrow it's recitation, we'll spend quite a bit of time on shortest path problems.

    Another classic optimization problem is the traveling salesman. Actually, I should probably, be modern, call it the traveling salesperson, the traveling salesperson problem. So the problem here, roughly, is given, a number of cities, and say the cost of traveling from city to city by airplane, what's the least cost round trip that you can find?

    So you start at one place, you have to go to a number of other places. End up where you started. It's not quite the same as the shortest path, and figure out the way to do that that involves spending the least money. Or the least time, or something else.

    What are some of the other classic optimization problems? There's bin packing. Filling up some container with objects of varying size and, perhaps, shape. So you've got the trunk of your car and you've got a bunch of luggage. More luggage than can actually fit in. And you're trying to figure out what order to put things in. And which ones you can put in. How to fill up that bin. Very important in shipping. People use bin packing algorithms to figure out, for example, how to load up container ships. Things of that nature. Moving vans, all sorts of things of that nature.

    In biology and in natural language processing and many other things, we see a lot of sequence alignment problems. For example, aligning DNA sequences, or RNA sequences. And, one we'll spend a fair amount of time on today and Tuesday is the knapsack problem. In the old days people used to call backpacks knapsacks. So we old folks sometimes even still make that mistake. And the problem there is, you've got a bunch of things. More than will fit into the knapsack, and you're trying to figure out which things to take and which things to leave. As you plan your hiking trip. How much water should you take. How many blankets? How much food? And you're trying to optimize the value of the objects you can take subject to the constraint that the backpack is of finite size.

    Now, why am I telling you all of this at this lightning speed? It's because I want you to think about it, going forward, about the issue of problem reduction. We'll come back to this. What this basically means is, you're given some problem to solve, that you've never seen before. And the first thing you do is ask is it an instance of some problem that other people have already solved?

    So when the folks at Mapquest sat down to do their program, I guarantee you somebody opened an algorithms book and said, what have other people done to solve shortest path problems? I'll rely on fifty years of clever people rather than trying to invent my own. And so frequently what we try and do is, we take a new problem and map it onto an old problem so that we can use an old solution.

    In order to be able to do that, it's nice to have in our heads an inventory of previously solved problems. To which we can reduce the current problem. So as we go through this semester, we'll look at, briefly or not so briefly, different previously solved problems in the hope that at some time in your future, when you have a problem to deal with, you'll say, I know that's really like shortest path, or really like graph coloring. Let me just take my problem and turn it into that problem, and use an existing solution.

    So we'll start looking in detail at one problem, and that's the knapsack problem. Let's see. Where do I want to start? Oh yes, OK.

    So far, we've been looking at problems that have pretty fast solutions. Most optimization problems do not have fast solutions. That is to say, when you're dealing with a large set of things, it takes a while to get the right answer. Consequently, you have to be clever about it.

    Typically up till now, we've looked at things that can be done in sublinear time. Or, at worst, polynomial time. We'll now look at a problem that does not fall into that. And we'll start with what's called the continuous knapsack problem.

    So here's the classic formulation. Assume that you are a burglar. And you have a backpack that holds, say, eight pounds' worth of stuff. And you've now broken into a house and you're trying to decide what to take. Well, let's assume in the continuous world, what you is you walk into the house and you see something like four pounds of gold dust. And you see three pounds of silver dust, and maybe ten pounds of raisins. And I don't actually know the periodic table entry for raisins. So I'll have to write it out.

    Well, how would you solve this problem? First, let's say, what is the problem? How can we formulate it? Well, let's assume that what we want to do is, we have something we want to optimize. So we're looking for a function to maximize, in this case. What might that function be? Well, let's say it's some number of, some amount of, the cost of the value of gold. Times however many pounds of gold. Plus the cost of silver times however many - no, gold, is a g, isn't it. Pounds of silver, plus the cost of raisins times the number of pounds of raisins.

    So that's the function I want to optimize. I want to maximize that function. And the constraint is what? It's that the pounds of gold plus the pounds of silver plus the pounds of raisins is no greater than eight. So I've got a function to maximize and a constraint that must be obeyed.

    Now, the strategy here is pretty clear. As often is for the continuous problem. What's the strategy? I pour in the gold till I run out of gold. I pour in the silver until I run out of silver. And then I take as many raisins as will fit in and I leave. Right? I hope almost every one of you could figure out that was the right strategy. If not, you're not well suited to a life of crime.

    What I just described is an instance of a greedy algorithm. In a greedy algorithm, at every step you do what maximizes your value at that step. So there's no planning ahead. You just do what's ever best. It's like when someone gets their food and they start by eating dessert. Just to make sure they get to the best part before they're full.

    In this case, a greedy algorithm actually gives us the best possible solution. That's not always so. Now, you've actually all implemented a greedy algorithm. Or are in the process thereof. Where have we implemented a greedy algorithm, or have been asked to do a greedy algorithm? Well, there are not that many things you guys have been working on this semester. Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly right. So what you were doing there, it was a really good throw. But it was a really good answer you gave. So I'll forgive you the bad hands. You were asked to choose the word that gave you the maximum value. And then do it again with whatever letters you had left. Was that guaranteed to win? To give you the best possible scores? No. Suppose, for example, you had the letters this, doglets. Well, the highest scoring word might have been something like Doges, these guys used to rule Venice, but if you did that you'd been left with the letters l and t, which are kind of hard to use. So you've optimized the first step. But now you're stuck with something that's not very useful. Whereas in fact, maybe you would have been better to go with dog, dogs, and let.

    So what we see here is an example of something very important and quite general. Which was that locally optimal decisions do not always lead to a global optimums. So you can't just repeatedly do the apparently local thing and expect to necessarily get to it.

    Now, as it happens with the continuous knapsack problem as we've formulated it, greedy is good. But let's look for a slight variant of it, where greedy is not so good. And that's what's called the zero-one knapsack problem.

    This is basically a discrete version of the knapsack problem. The formulation is that we have n items and at every step we have to either take the whole item or none of the item. In the continuous problem, the gold dust was assumed to be infinitely small. And so you could take as much of it as you wanted. Here it's as if you had gold bricks. You get to take the whole brick or no brick at all. Each item has a weight and a value, and we're trying to optimize it as before. So let's look at an example of a zero-one knapsack problem.

    Again we'll go back to our burglar. So the burglar breaks into the house and finds the following items available. And you'll see in your handout a list of items and their value and how much, what they weight. Finds a watch, a nice Bose radio, a beautiful Tiffany vase, and a large velvet Elvis. And now this burglar finds, in fact, two of each of those. Person is a real velvet Elvis fan and needed two copies of this one.

    Alright, and now he's trying to decide what to take. Well if the knapsack were large enough the thief would take it all and run, but let's assume that it can only hold eight pounds, as before. And therefore the thief has choices to make. Well, there are three types of thieves I want to consider: the greedy thief, the slow thief, and you.

    We'll start with the greedy thief. Well, the greedy thief follows the greedy algorithm. What do you get if you follow the greedy algorithm? What's the first thing the thief does? Takes the most valuable item, which is a watch. And then what does he do after that? Takes another watch. And then? Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: And then?

    STUDENT: [INAUDIBLE]

    PROFESSOR: No. Not unless he wants to break the vase into little pieces and stuff it in the corners. The backpack is now full, right? There's no more room. So the greedy thief take that and leaves. But it's not an optimal solution. What should the thief have done? What's the best thing you can do? Instead of taking that one vase, the thief could take two radios. And get more value. So the greedy thief, in some sense, gets the wrong answer. But maybe isn't so dumb.

    While greedy algorithms are not guaranteed to get you the right answer all the time, they're often very good to use. And what they're good about is, they're easy to implement. And they're fast to run. You can imagine coding the solution up and it's pretty easy. And when it runs, it's pretty fast. Just takes the most valuable, the next most valuable, the next most valuable, I'm done. And the thief leaves, and is gone. So that's a good thing.

    On the other hand, it's often the case in the world that that's not good enough. And we're not looking for an OK solution, but we're looking for the best solution. Optimal means best. And that's what the slow thief does. So the slow thief thinks the following. Well, what I'll do is I'll put stuff in the backpack until it's full. I'll compute its value. Then I'll empty the backpack out, put another combination of stuff compute its value, try all possible ways of filling up the backpack, and then when I'm done, I'll know which was the best. And that's the one I'll do.

    So he's packing and unpacking, packing and unpacking, trying all possible combinations of objects that will obey the constraint. And then choosing the winner. Well, this is like an algorithm we've seen before. It's not greedy. What is this? What category of algorithm is that? Somebody? Louder?

    STUDENT: Brute force.

    PROFESSOR: Brute force, exhaustive enumeration, exactly. We're exhausting all possibilities. And then choosing the winner. Well, that's what the slow thief tried. Unfortunately it took so long that before he finished the owner returned home, called the police and the thief ended up in jail. It happens. Fortunately, while sitting in jail awaiting trial, the slow thief decided to figure what was wrong. And, amazingly enough, he had studied mathematics. And had a blackboard in the cell. So he was able to work it out.

    So he first said, well, let me try and figure out what I was really doing and why it took so long. So first, let's think about what was the function the slow thief was attempting to maximize. The summation, from i equals 1 to n, where n is the number of items, so I might label watch 1-0, watch 2-2, I don't care that they're both watches. They're two separate items. And then what I want to maximize is the sum of the price of item i times whether or not I took x i. So think of x as a vector of 0's and 1's. Hence the name of the problem.

    If I'm going to keep that item, item i, if I'm going to take it, I give it a 1. If I'm not going to take it I give it a 0. And so I just take the value of that item times whether or not it's or 1. So my goal here is to choose x such that this is maximized. Choose x such that that function is maximized, subject to a constraint. And the constraint, is it the sum from 1 to n of the weight of the item, times x i, is less than or equal to c, the maximum weight I'm allowed to put in my backpack. In this case it was eight.

    So now I have a nice, tidy mathematical formulation of the problem. And that's often the first step in problem reduction. Is to go from a problem that has a bunch of words, and try and write it down as a nice, tight mathematical formulation. So now I know the formulation is to find x, the vector x, such that this constraint is obeyed and this function is maximized. That make sense to everybody? Any questions about that? Great.

    So as the thief had thought, we can clearly solve this problem by generating all possible values of x and seeing which one solves this problem. But now the thief started scratching his head and said, well, how many possible values of x are there? Well, how can we think about that? Well, a nice way to think about that is to say, I've got a vector. So there's the vector with eight 0's in it. Three, four, five, six, seven, eight. Indicating I didn't take any of the items.

    There's the vector, and again you'll see this in your handout, says I only took the first item. There's the one that says I only took the second item. There's the one that says I took the first and the second item. And at the last, I have all 1's. This series look like anything familiar to you? These are binary numbers, right? Eight digit binary numbers. Zero, one, two, three. What's the biggest number I can represent with eight of these? Somebody?

    Well, suppose we had decimal numbers. And I said I'm giving you three decimal digits. What's the biggest number you can represent with three decimal digits? Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Right. Which is roughly what? 10 to the?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Right. Yes.

    STUDENT: [INAUDIBLE]

    PROFESSOR: Right. Exactly right. So, in this case it's 2 to the 8th. Because I have eight digits. But exactly right. More generally, it's 2 to the n. Where n is the number of possible items I have to choose from. Well, now that we've figured that out, what we're seeing is that the brute force algorithm is exponential. In the number of items we have to choose from. Not in the number that we take, but the number we have to think about taking.

    Exponential growth is a scary thing. So now we can look at these two graphs, look at the top one. So there, we've compared n squared, quadratic growth, which by the way Professor Grimson told you was bad, to, really bad, which is exponential growth. And in fact, if you look at the top figure it looks as exponential or, quadratic isn't even growing at all. You see how really fast exponential growth is?

    You get to fifteen items and we're up at seventy thousand already and counting. The bottom graph has exactly the same data. But what I've done is, I've use the logarithmic y-axis. Later in the term, we'll spend quite a lot of time talking about how do we visualize data. How do we make sense of data. I've done that because you can see here that the quadratic one is actually growing. It's just growing a lot more slowly. So the moral here is simple one. Exponential algorithms are typically not useful. n does not have to get very big for exponential to fail.

    Now, imagine that you're trying to pack a ship and you've got ten thousand items to choose from. 2 to the 10,000 is a really big number. So what we see immediately, and the slow thief decided just before being incarcerated for years and years, was that it wasn't possible to do it that way. He threw up his hands and said, it's an unsolvable problem, I should have been greedy, there's no good way to do this.

    That gets us to the smart thief. Why is this thief smart? Because she took 600. And she learned that in fact there is a good way to solve this problem. And that's what we're going to talk about next. And that's something called dynamic programming.

    A lot of people think this is a really hard and fancy concept, and they teach in advanced algorithms classes. And they do, but in fact as you'll see it's really pretty simple. A word of warning. Don't try and figure out why it's called dynamic programming. It makes no sense at all. It was invented by a mathematician called Bellman. And he was at the time being paid by the Defense Department to work on something else. And he didn't want them to know what he was doing. So he made up a name that he was sure they would have no clue what it meant. Unfortunately, we now have lived with it forever, so don't think of it as actually being anything dynamic particularly. It's just a name.

    It's very useful, and why we spend time on it for solving a broad range of problems that on their surface are exponentially difficult. And, in fact, getting very fast solutions to them. The key thing in dynamic programming, and we'll return to both of these, is you're looking for a situation where there are overlapping sub-problems and what's called optimal substructure. Don't expect to know what these mean yet. Hopefully by the end of the day, Tuesday, they will both make great sense to you. Let's first look at the overlapping sub-problems example.

    You have on your handout, a recursive implementation of Fibonacci. Which we've seen before. What I've done is I've augmented it with this global called num calls just so I can keep track of how many times it gets called. And let's see what happens if we call fib of 5.

    Well, quite a few steps. What's the thing that you notice about this output? There's something here that should tip us off that maybe we're not doing this the most efficient way possible. I called fib with 5 and then it calls it with 4. And then that call calls fib with 3. So I'm doing 4, 3, 2, 2, 1, 0. And I'm doing 1, 2. Well, what we see is I'm calling fib a lot of times with the same argument. And it makes sense. Because I start with fib of 5, and then I have to do fib of 4 and fib of 3. Well, fib of 4 is going to also have to do a fib of 3 and a fib of 2 and a fib of 1, and a fib of 0. And then the fib of 3 is going to do a fib of 2 and a fib of 1 and a fib of 0. And so I'm doing the same thing over and over again. That's because I have what are called overlapping sub-problems.

    I have used divide and conquer, as we seen before, to recursively break it into smaller problems. But the smaller problem of fib of 4 and the smaller problem of fib of 3 overlap with each other. And that leads to a lot of redundant computation. And I've done fib of 5, which is a small number. If we look at some other things, for example, let's get rid of this. Let's try see fib of 10. Well, there's a reason I chose 10 rather than, say, 20.

    Here fib got called 177 times. Roughly speaking, the analysis of fib is actually quite complex, of a recursive fib. And I won't go through it, but what you can see is it's more or less, it is in fact, exponential. But it's not 2 to the something. It's a more complicated thing. But it grows quite quickly. And the reason it does is because of this overlapping.

    On Tuesday we'll talk about a different way to implement Fibonacci, where the growth will be much less dramatic. Thank you.
  • 21
    Assignment 7: Review problems
    3 pages
  • 22
    MIT Lecture 13: Dynamic Programming: Overlapping Subproblems, Optimal Substructure
    48:56
    Dynamic programming: overlapping subproblems, optimal substructure. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: OK. I want to start where we left off. You remember last time we were looking at Fibonacci. And so we've got it up here, a nice little recursive implementation of it. And the thing I wanted to point out is, we've got this global variable number of calls. Which is there not because Fibonacci needs it but just for pedagogical reasons, so that we can keep track of how much work this thing is doing. And I've turned on a print statement which was off last time. So we can see what it's doing is it runs. So let's try it here with Fib of 6.

    So, as we would hope, Fib of 6 happens to be 8. That right? That right, everybody? Should Fib of 6 be 8? I don't think so. So first thing we should do is scratch our heads and see what's going on here. Alright, let's look at it. What's happening here? This is your morning wake-up call. What is happening? Yes.

    STUDENT: [INAUDIBLE]

    PROFESSOR: See if I can get it all the way to the back. No I can't. That's embarrassing. Alright. So if n is less than or equal to 1, return n. Well that's not right, right? What should I be doing there? Because Fib of is? What? 1. So let's fix it. How about that, right? Or maybe, what would be even simpler than that? Maybe I should just do that.

    Now let's try it. We feel better about this? We like that answer? Yes, no? What's the answer, guys? What should Fibonacci of 6 be? I think that's the right answer, right? OK.

    So what do I want you to notice about this? I've computed a value which is 13. And it's taken me 25 calls. 25 recursive calls to get there. Why is it taking so many? Well, what we can see here is that I'm computing the same value over and over again. Because if we look at the recursive structure of the program, what we'll see that's going on here is, I call Fib of 5 and 4, but then Fib of 5 is also going to call Fib of 4. So I'm going to be computing it on those branches. And then it gets worse and worse as I go down.

    So if I think about computing Fib of I'm going to be computing that a lot of times. Now, fortunately, Fib of 0 is short. But the other ones are not so short. And so what I see is that as I run this, I'm doing a lot of redundant computation. Computing values whose answer I should already know. That's, you'll remember last time, I talked about the notion of overlapping sub-problems. And that's what we have here. As with many recursive algorithms, I solve a bigger problem by solving a smaller instance of the original problem. But here there's overlap. The instant unlike binary search, where each instance was separate, here the instances overlap. They share something in common. In fact, they share quite a lot in common. That's not unusual.

    That will lead me to use a technique I mentioned again last time, called memoization. Effectively, what that says is, we record a value the first time it's computed, then look it up the subsequent times we need it.

    So it makes sense. If I know I'm going to need something over and over again, I squirrel it away somewhere and then get it back when I need it. So let's look at an example of that.

    So I'm going to have something called fast Fib. But first I'm going to have, let's look at what fast Fib does and then we'll come back to the next question. It takes the number whose Fibonacci I want plus a memo. And the memo will be a dictionary that maps me from a number to Fib of that number.

    So what I'm going to do, well, let's get rid of this print statement for now. I'm going to say, if n is not in memo. Remember the way dictionary works, this is the key. Is the key of a value. Then I'll call fast Fib recursively, with n minus 1 in memo, and n minus 2 in memo. Otherwise I'll return the memo.

    Well, let's look at it for a second. This is the basic idea. But do we actually believe this is going to work? And, again, I want you to look at this and think about what's going to happen here. Before we do that, or as you do that, let's look at Fib 1. The key thing to notice about Fib 1 is that it has the same specification as Fib. Because when somebody calls Fibonacci, they shouldn't worry about memos. And how I'd implemented it. That has to be under the covers. So I don't want them to have to call something with two arguments. The integer and the memo. So I'll create Fib 1, which has the same arguments as Fib. The first thing it does is it initializes the memo. And initializes it by saying, if I get I -- whoops. Aha. Let's be careful here. If I get 0 I return 1. I get 1, I return 1. So I put two things in the memo already. And then I'll call fast Fib and it returns the result it has.

    So you see the basic idea. I take something with the same parameters as the original. Add this memo. Give it some initial values. And then call. So now what do we think? Is this going to work? Or is there an issue here? What do you think? Think it through. If it's not in the memo, I'll compute its value and put it in the memo. And then I'll return it. OK? If it was already there, I just look it up. That make sense to everybody?

    Let's see what happens if we run it. Well, actually, let's turn the print statement on, since we're doing it with a small value here. So what we've seen is I've run it twice here. When I ran it up here, with the old Fib, and we printed the result, and I ran it with Fib 1 down here. The good news is we got 13 both times.

    The even better news is that instead of 25 calls, it was only 11 calls. So it's a big improvement. Let's see what happens, just to get an idea of how big the improvement is. I'm going to take out the two print statements. And let's try it with a bigger number.

    It's going to take a little bit. Well, look at this difference. It's 2,692,537 versus 59. That's a pretty darn big difference. And I won't ask you to check whether it got the right answer. At least, not in your heads.

    So you can see, and this is an important thing we look at, is that as we look at growth, it didn't look like it mattered a lot with 6. Because it was one small number to one slightly smaller number. But this thing grows exponentially. It's a little bit complicated exactly how. But you can see as I go up to 30 I get a pretty big number. And 59 is a pretty small number. So we see that the memoization here is buying me a tremendous advantage. And this is what lies at the heart of this very general technique called dynamic programming.

    And in fact, it lies at the heart of a lot of useful computational techniques where we save results. So if you think about the way something like, say, Mapquest works, and last week in recitation you looked at the fact that shortest path is exponential. Well, what it does is it saves a lot of paths. It kind of knows people are going to ask how do you get from Boston to New York City. And it may have saved that. And if you're going from Boston to someplace else where New York just happens to be on the way, it doesn't have to recompute that part of it. So it's saved a lot of things and squirreled them away. And that's essentially what we're doing here. Here we're doing it as part of one algorithm. There, they're just storing a database of previously solved problems. And relying on something called table lookup, Of which memoization is a special case. But table lookup is very common. When you do something complicated you save the answers and then you go get it later.

    I should add that in some sense this is a phony straw-man Fibonacci. Nobody in their right mind actually implements a recursive Fibonacci the way I did it originally. Because the right way to do it is iteratively. And the right way to do it is not starting at the top, it's starting at the bottom. And so you can piece it together that way. But don't worry about it, it's not, I'm just using it because it's a simpler example than the one I really want to get to, which is knapsack. OK, people get this? And see the basic idea and why it's wonderful? Alright.

    Now, when we talked about optimization problems in dynamic programming, I said there were two things to look for. One was overlapping sub-problems. And the other one was optimal substructure. The notion here is that you can get a globally optimal solution from locally optimal solutions to sub-problems.

    This is not true of all problems. But as we'll see, it's true of a lot of problems. And when you have an optimal substructure and the local solutions overlap, that's when you can bring dynamic programming to bear. So when you're trying to think about is this a problem that I can solve with dynamic programming, these are the two questions you ask.

    Let's now go back and instantiate these ideas for the knapsack problem we looked at last time. In particular, for the 0-1 knapsack problem. So, we have a collection of objects. We'll call it a. And for each object in 0, we have a value. In a, we have a value. And now we want to find the subset of a that has the maximum value, subject to the weight constraint. I'm just repeating the problem.

    Now, what we saw last time is there's a brute force solution. As you have discovered in recent problem set, it is possible to construct all subsets of a set. And so you could construct all subsets, check that the weight is less than the weight of the knapsack, and then choose the subset with the maximum value. Or a subset with the maximum value, there may be more than one, and you're done.

    On the other hand, we've seen that if the size of a is n, that's to say, we have n elements to choose from, then the number of possible subsets is 2 to the n. Remember, we saw that last time looking at the binary numbers. 2 to the n is a big number. And maybe we don't have to consider them all, because we can say, oh this one is going to be way too big. It's going to weigh too much, we don't need to look at it. But it'll still be order 2 to the n. If n is something like 50, not a big number, 2 to the 50 is a huge number.

    So let's ask, is there an optimal substructure to this problem. That would let us tackle it with dynamic programming. And we're going to do this initially by looking at a straightforward implementation based upon what's called the decision tree.

    This is a very important concept, and we'll see a lot of algorithms essentially implement decision trees. Let's look at an example. Let's assume that the weights, and I'll try a really small example to start with, are 5, 3 and 2, and the values, corresponding values, are 9, 7 and 8. And the maximum, we'll say, is 5.

    So what we do is, we start by considering for each item whether to take it or not. For reasons that will become apparent when we implement it in code, I'm going to start at the back. The last element in the list. And what I'm going to use is the index of that element to keep track of where I am. So I'm not going to worry whether this item is a vase or a watch or painting. I'm just going to say it's the n'th element. Where n'th is somewhere between and 2 in this case. And then we'll construct our tree as follows: each node, well, let me put an example here. The first node will be the to-pull 2, 5 and 0. Standing for, let me make sure I get this in the right order, well, the index which is 2, the last element in this case, so that's the index. This is the weight still available. If you see that in the shadow. And this is the value currently obtained. So I haven't included anything. Means I have all 5 pounds left. But I don't have anything of value.

    Now, the decision tree, if I branch left, it's a binary tree. This is going to be don't take. So I'm not going to take the item with an index of 2. So that means this node will have an index of 1.

    Next item to be considered, I still have 5 pounds available. And I have value. To be systematic, I'm going to build this tree depth-first left-first. At each node, I'm going to go left until I can't go any further. So we'll take another don't-take branch here. And what is this node going to look like? Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: 0, 5, 0. And then we'll go one more. And I'll just put a minus indicating I'm done. I can't look below that. I still have five pounds left, and I still have zero value.

    The next thing I'm going to do is backtrack. That is to say, I'm going to go back to a node I've already visited. Go up the tree 1. And now, of course, the only place to go is right. And now I get to include something. Yeah. And what does this node look like? Well, I'll give you a hint, it starts with a minus. What next?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Pardon. 0. And?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Pardon? 5. Alright. So far, this looks like the winner. But, we'd better keep going. We backtrack to here again. There's nothing useful to do. We backtrack to here. And we ask, what do we get with this node? 0, 3. Somebody?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Louder. 2. And then?

    STUDENT: [INAUDIBLE]

    PROFESSOR: There's a value here, what's this value? I've included item number 1, which has a value of 7. Right? So now this looks like the winner. Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: This one?

    STUDENT: Yeah. [INAUDIBLE]

    PROFESSOR: Remember, I'm working from the back. So it shouldn't be 9. Should be what? Item 0, oh, you're right, items is 9. Thank you. Right you are. Still looks like the winner. I can't hear you.

    STUDENT: [INAUDIBLE]

    PROFESSOR: Let's be careful about this. I'm glad people are watching. So we're here. And now we've got 5 pounds available. That's good. And we're considering item number 0. Which happens to weigh 5 pounds. So that's a good thing. So, it's the last item to consider. If we include it, we'll have nothing left. Because we had 5 and we're using all 5. So that will be 0. And its value is 9. So we put one item in the backpack and we've got a value of 9. Anyone have a problem with that? So far, so good?

    Now we've backed up to here. We're considering item 1 and we're trying to ask whether we can put it in. Item 1 has a weight of 3. So it would fit. And if we use it, we have 2 pounds left. And item 1 has a value of 7. So if we did this, we'd have a value of 7. But we're not done yet, right? We still have some things to consider. Well, we could consider not putting in item 0. That makes perfect sense. And we're back to where we're there, minus 2 and 7.

    And now let's ask the question how, about putting in item 0. Well, we can't. Because it would weigh 5 pounds, I only have 2 left. So there is no right branch to this one. So I'm making whatever decisions I can make along the way. Let's back up to here. And now we're going to ask about taking item 2. If we take item 2, then, well, the index after that will of course be 1. And the available weight will be 3. And the value will be 8, right? Alright, now we say, can I take item 1. Yeah. I can. It only weighs 3 and I happen to have 3 left. So that's good. Don't take it, right. Sorry. This is the don't take branch. So I go to 0, 3, 8. And then I can do another don't take branch. And this gets me to what, minus 3, 8. I'll now back up. And I'll say, alright, suppose I do take item 0, well I can't take item 0, right? Weighs too much. So, don't have that branch. Back up to here. Alright, can I take item 1? Yes, I can. And that gives me what?

    STUDENT: [INAUDIBLE]

    PROFESSOR: We have a winner. So it's kind of tedious, but it's important to see that it works. It's systematic. I have a way of exploring the possible solutions. And at the end I choose the winner. What's the complexity of this decision tree solution? Well, in the worst case, we're enumerating every possibility of in and out. Now I've shortened it a little bit by saying, ah, we've run out of weight, we're OK. But effectively it is, as we saw before, exponential. 2 to the n, every value in the bit vector we looked at last time is either 0 or 1. So it's a binary number of n bits, 2 to the n.

    Let's look at a straightforward implementation of this. I'll get rid of Fibonacci here, we don't want to bother looking at that again. Hold on a second until I comment this out, yes.

    STUDENT: [INAUDIBLE]

    PROFESSOR: Yeah. There's a branch we could finish here, but since we're out of weight we sort of know we're going to be done. So we could complete it. But it's not very interesting. But yes, we probably should have done that.

    So let's look at an implementation here. Whoops. You had these in the handout, by the way. So here's max val. It takes four arguments. The weight, w, and v, these are the two vectors we've seen here. Of the weights and the values. It takes i, which is in some sense the length of those vectors, minus 1, because of the way Python works. So that gives me my index. And the amount of weight available, a w, for available weight.

    So again, I put in this num calls, which you can ignore. First line says, if i is 0, that means I'm looking at the very last element. Then if the weight of i is less than the available weight, I can return the value of i. Otherwise it's 0. I've got one element to look at. I either put it in if I can. If I can't, I don't. Alright, so if I'm at the end of the chain, that's my value. In either event, if I'm looking at the last element, I return.

    The next line says alright, suppose I don't, I'm not at the last element. Suppose I don't include it. Then the maximum value I can get is the maximum value of what I had. But with index i minus 1. So that's the don't-take branch. As we've seen systematically in the don't-takes, the only thing that gets changed is the index.

    The next line says if the weight of i is greater than a w, well then I know I can't put it in. So I might as well return the without i value I just computed. Otherwise, let's see what I get with i. And so the value of with i will be the value of i plus whatever I can get using the remaining items and decrementing the weight by the weight of i. So that's exactly what we see as we go down the right branches. I look at the rest of the list, but I've changed the value. And I've changed the available weight.

    And then when I get to the very end, I'm going to return the bigger of with i and without i. I've computed the value if I include i. I computed the value if I don't include i. And then I'm going to just return the bigger of the two. Little bit complicated, but it's basically just implementing this decision tree.

    Let's see what happens if we run it. Well, what I always do in anything like this is, the first thing I do is, I run it on something where I can actually compute the answer in my head. So I get a sense of whether or not I'm doing the right thing. So here's a little example.

    And I'm going to pause for a minute, before I run it, and ask each of you to compute in your head what you think the answer should be. In line with what I said about debugging. Always guess before you run your program what you think it's going to do. And I'm going to wait until somebody raises their hand and gives me an answer. Yes.

    STUDENT: 29?

    PROFESSOR: So we have a hypothesis that the answer should be 29. Ooh. That's bad. So as people in the back are answering these questions because they want to test my arm. The camera probably didn't catch that, but it was a perfect throw. Anyone else think it's 29? Anyone think it's not 29? What do you think it is? Pardon

    STUDENT: 62?

    PROFESSOR: 62. That would be impressive. 62. Alright, well, we have a guess of 62. Well, let's run it and see. 29 it is. And it made a total of 13 calls. It didn't do this little optimization I did over here. But it gives us our answer. So that's pretty good. Let's try a slightly larger example.

    So here I'm going to use the example we had in class last time. This was the burglar example where they had two copies of everything. Here, it gets a maximum value of 48 and 85 calls. So we see that I've doubled the size of the vector but I've much more than doubled the number of calls. This is one of these properties of this kind of exponential growth.

    Well, let's be really brave here. And let's try a really big vector. So this particular vector, you probably can't even see the whole thing on your screen. Well, it's got 40 items in it. Let's see what happens here. Alright, who wants to tell me what the answer is while this computes away? Nobody in their right mind. I can tell you the answer, but that's because I've cheated, run the program before.

    Alright, this is going to take a while. And why is it going to take a while? Actually it's not 40, I think these are, alright. So the answer is 75 and the number of calls is 1.7 million. Pardon? 17 million. Computers are fast, fortunately. Well, that's a lot of calls.

    Let's try and figure out what's going on. But let's not try and figure out what's going on with this big, big example because we'll get really tired. Oh. Actually, before we do that, just for fun, what I want to do is write down for future reference, as we look at a fast one, that the answer is 75 and Eric, how many calls? 240,000. Alright. We'll come back to those numbers.

    Let's look at it with a smaller example. We'll look at our example of 8. That we looked at before. And we'll turn on this print statement. Ooh, what was that? Notice that I'm only printing i and a w. Why is that? Because w and v are constant. I don't want you to print them over and over again. No, I'd better call it. That would be a good thing to do, right? So let's see. We'll call it with this one.

    So it's printing a lot. It'll print, I think, 85 calls. And the thing you should notice here is that it's doing a lot of the same things over and over again. So, for example, we'll see 2, 1 here. And 2, 1 here. And 2, 1 here. Just like Fibonacci, it's doing the same work over and over again. So what's the solution?

    Well, you won't be surprised to hear it's the same solution. So let's look at that code. So I'm going to do exactly the same trick we did before. I don't want b to the Fibonacci. I'm going to introduce a max val 0, which has exactly the same arguments as max val. Here I'll initiate the memo to be 0, or to be empty, rather, the dictionary. And then I'll call fast max val passing at this extra argument of the memo.

    So the first thing I'm going to do is, I'm going to try and return the value in the memo. This is a good thing to do, right? If it's not already there, what will happen? It will raise an exception. And I'll go to the except clause. So I'm using the Python try except to check whether or not the thing is in the memo or not. I try and return the value. If it's there, I return it. If not I'll get a key error. And what do I do if I get a key error? Well, now I go through code very much like what I did for max val in the first place. I check whether it's 0, et cetera, et cetera. It's exactly, really, the same, except before I return the value I would have returned I squirrel it away in my memo for future use. So it's the same structure as before. Except, before I return the value, I save it. So the next time I need it, I can look it up.

    Let's see if it works. Well, let's do a small example first. It's calling the old max val. With all those print statements. Sorry about that. But we'll let it run. Well, it's a little bit better. It got 85. Same answer, but 50 calls instead of 85. But let's try the big one. Because we're really brave. So here's where we have 30 elements, and a maximum weight of 40. I'm not going to call the other max val here, because we know what happens when I do that. I've created my own little memo over there.

    And let's let it rip. Wow. Well, I got the same answer. That's a good thing. And instead of 17 million calls, I have 1800 calls. That's a huge improvement. And that's sort of the magic of dynamic programming.

    On Thursday we'll do a more careful analysis and try and understand how I could have accomplished this seemingly magical task of solving an exponential problem so really quickly.
  • 23
    Assignment 8: Dynamic programming
    6 pages
  • 24
    MIT Lecture 14: Introduction to Object-oriented Programming
    50:34
    Analysis of knapsack problem, introduction to object-oriented programming. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: At the end of the lecture on Tuesday, a number of people asked me questions, asked Professor Grimson questions, which made it clear that I had been less than clear on at least a few things, so I want to come back and revisit a couple of the things we talked about at the end of the lecture.

    You'll remember that I had drawn this decision tree, in part because it's an important concept I want you to understand, the concept of decision trees, and also to illustrate, I hope, visually, some things related to dynamic programming. So we had in that decision tree, is we had the weight vector, and I just given a very simple one [5,3,2], and we had a very simple value vector, [9,7,8]. And then the way we drew the tree, was we started at the top, and said all right, we're going to first look at item number 2, which was the third item in our list of items, of course, and say that we had five pounds left of weight that our knapsack that could hold, and currently had a value of 0. And then we made a decision, to not put that last item in the backpack, and said if we made that decision, the next item we had to consider was item 1, we still had five pounds available, and we still had a weight available.

    Now I, said the next item to consider is item 1, but really what I meant is, 1 and all of the items proceeding it in the list. This is my shorthand for saying the list up to and including items sub 1, kind of a normal way to think about it. And then we finish building the tree, left first step first, looking at all the no branches, 0,5,0 and then we were done, that was one branch. We then backed up, and said let's look at a yes, we'll include item number 1. Well, what happens here, if we've included that, it uses up all the available weight, and gave us the value of 9.

    STUDENT: [UNINTELLIGIBLE]

    PROFESSOR: Pardon?

    STUDENT: -- want to be off the bottom branch.

    PROFESSOR: Yup, Off by 1. Yeah, I wanted to come off this branch, because I've backtrack just 1, thank you. And then I backtrack up to this branch, and from here we got 0,2,7. And I'm not going to draw the rest of the tree for you here, because I drew it last time, and you don't need to see the whole tree. The point I wanted to make is that for every node, except the leaves, the leaves are the bottom of a tree in this case, computer scientists are weird, right, they draw trees where the root is at the top, and the leaves are at the bottom. And I don't know why, but since time immemorial that is the way computer scientists have drawn trees. That's why we're not biologists, I guess. We don't understand these things.

    But what I want you to notice is that for each node, except the leaves, the solution for that node can be computed from the solutions from it's children. So in order to look at the solution of this node, I choose one of the solutions of it's children, a or b, is the best solution if I'm here, and of course this is the better of the 2 solutions. If I look at this node, I get to choose its solution as the better of the solution for this node, and this node. All the way up to the top, where when I have to choose the best solution to the whole problem, it's either the best solution to the left node, or the best solution to the right node. This happens to be a binary decision tree. There's nothing magic about there being only two nodes, for the knapsack problem, that's just the way it works out, but there are other problems where there might be multiple decisions to make, more than a or yes or no, but it's always the case here that I have what we last time talked about as what? Optimal sub structure. As I defined it last time, it means that I can solve a problem by finding the optimal solution to smaller sub problems. Classic divide and conquer that we've seen over and over again in the term. Take a hard problem, say well, I can solve it by solving 2 smaller problems and combine their solution, and this case, the combining is choosing the best, it's a or b.

    So then, I went directly from that way of thinking about the problem, to this straightforward, at the top of the slide here, also at the top of your handout, both yesterday and today, a straightforward implementation of max val, that basically just did this. And as you might have guessed, when you're doing this sort of thing, recursion is a very natural way to implement it. We then ran this, demonstrated that it got the right answer on problems that were small enough that we knew with the right answer was, ran it on a big problem, got what we hoped was the right answer, but we had no good way to check it in our heads, but noticed it took a long time to run. And then we asked ourselves, why did it take so long to run? And when we turned on the print statement, what we saw is because it was doing the same thing over and over again. Because we had a lot of the sub-problems were the same.

    It was as if, when we went through this search tree, we never remembered what we got at the bottom, and we just re-computed things over and over. So that led us to look at memoization, the sort of key idea behind dynamic programming, which says let's remember the work we've done and not do it all over again. We used a dictionary to implement the memo, and that got us to the fast max val, which got called from max val 0, because I wanted to make sure I didn't change the specification of max val by introducing this memo that users shouldn't know even exists, because it's part of the implementation, not part of the problem statement. We did that, and all I did was take the original code and keep track of what I've done, and say have I computed this value before, if so, don't compute it again. And that's the key idea that you'll see over and over again as you solve problems with dynamic programming, is you say, have I already solved this problem, if so, let me look up the answer. If I haven't solved the problem, let me solve it, and store the answer away for later reference.

    Very simple idea, and typically the beauty of dynamic programming as you've seen here, is not only is the idea simple, even the implementation is simple. There are a lot of complicated algorithmic ideas, dynamic programming is not one of them. Which is one of the reasons we teach it here. The other reason we teach it here, in addition to it being simple, is that it's incredibly useful. It's probably among the most useful ideas there is for solving complicated problems.

    All right, now let's look at it. So here's the fast version, we looked at it last time, I'm not going to bore you by going through the details of it again, but we'll run it. This was the big example we looked at last time, where we had 30 items we could put in to choose from, so when we do it exponentially, it looks like it's 2 to the 30, which is a big number, but when we ran this, it found the answer, and it took only 1805 calls. Now I got really excited about this because, to me it's really amazing, that we've taken a problem that is apparently exponential, and solved it like that. And in fact, I could double the size of the items to choose from, and it would still run like. Eh - I'm not very good at snapping my fingers -- it would still run quickly. All right, so here's the question: have I found a way to solve an inherently exponential problem in linear time. Because what we'll see here, and we saw a little of this last time, as I double the size of the items, I only really roughly double the running time. Quite amazing. So have I done that? Well, I wish I had, because then I would be really famous, and my department head would give me a big raise, and all sorts of wonderful things would follow. But, I'm not famous, and I didn't solve that problem. What's going on?

    Well this particular algorithm takes roughly, and I'll come back to the roughly question, order (n,s) time, where n is the number of items in the list and s, roughly speaking, is the size of the knapsack. We should also observe, that it takes order and s space. Because it's not free to store all these values. So at one level what I'm doing is trading time for space. It can run faster because I'm using some space to save things. So in this case, we had 30 items and the wait was 40, and, you know, this gives us 1200 which is kind of where we were. And I'm really emphasizing kind of here, because really what I'm using the available size for, is as a proxy for the number of items that can fit in the knapsack. Because the actual running time of this, and the actual space of this algorithm, is governed, interestingly enough, not by the size of the problem alone, but by the size of the solution. And I'm going to come back to that.

    So how long it takes to run is related to how many items I end up being able to fit into the knapsack. If you think about it, this make sense. An entry is made in the memo whenever an item, and an available size pair is considered. As soon as the available size goes to 0, I know I can't enter any more items into the memo, right? So the number of items I have to remember is related to how many items I can fit in the knapsack. And of course, the amount of running time is exactly the number of things I have to remember, almost exactly, right? So you can see if you think about it abstractly, why the amount of work I have to do here will be proportional to the number of items I can fit in, that is to say, the size of the solution.

    This is not the way we'd like to talk about complexity. When we talk about the order, or big O, as we keep writing it, of a problem, we always prefer to talk about it in terms of the size of the problem. And that makes sense because in general we don't know the size of the solution until we've solved it. So we'd much rather define big O in terms of the inputs. What we have here is what's called a pseudo-polynomial algorithm. You remember a polynomial algorithm is an algorithm that's polynomial in the size of the inputs. Here we have an algorithm that's polynomial in the size of the solution, hence the qualifier pseudo. More formally, and again this is not crucial to get all the details on this, if we think about a numerical algorithm, a pseudo-polynomial algorithm has running time that's polynomial in the numeric value of the input. I'm using a numeric example because it's easier to talk about it that way. So you might to look at, say, an implementation of factorial, and say its running time is proportional to the numerical value of the number who's factorial. If I'm computing factorial of 8, I'll do 8 operations, Right Factorial of 10, I'll do 10 operations.

    Now the key issue to think about here, is that as we look at this kind of thing, what we'll see is that, if we look at a numeric value, we know that that's exponential number in the number of digits. So that's the key thing to think about, Right That you can take a problem, and typically, when we're actually formally looking at computational complexity, big O, what we'll define the in terms of, is the size of the coding of the input. The number of bits required to represent the input in the computer. And so when we say something is exponential, we're talking about in terms of the number of bits required to represent it.

    Now why am I going through all this, maybe I should use the word pseudo-theory? Only because I want you to understand that when we start talking about complexity, it can be really quite subtle. And you have to be very careful to think about what you mean, or you can be very surprised at how long something takes to run, or how much space it uses. And you have to understand the difference between, are you defining the performance in terms of the size of the problem, or the size of the solution. When you talk about the size of the problem, what do you mean by that, is it the length of an array, is it the size of the elements of the array, and it can matter. So when we ask you to tell us something about the efficiency, on for example a quiz, we want you to be very careful not to just write something like, order n squared, but to tell us what n is. For example, the number of elements in the list. But if you have a list of lists, maybe it's not just the number elements in the list, maybe it depends upon what the elements are. So just sort of a warning to try and be very careful as you think about these things, all right.

    So I haven't done magic, I've given you a really fast way to solve a knapsack problem, but it's still exponential deep down in its heart, in something. All right, in recitation you'll get a chance to look at yet another kind of problem that can be solved by dynamic programming, there are many of them. Before we leave the knapsack problem though, I want to take a couple of minutes to look at a slight variation of the problem.

    So let's look at this one. Suppose I told you that not only was there a limit on the total weight of the items in the knapsack, but also on the volume. OK, if I gave you a box of balloons, the fact that they didn't weight anything wouldn't mean you couldn't put, you could put lots of them in the knapsack, right? Sometimes it's the volume not the weight that matters, sometimes it's both. So how would we go about solving this problem if I told you not only was there a maximum weight, but there was a maximum volume. Well, we want to go back and attack it exactly the way we attacked it the first time, which was write some mathematical formulas. So you'll remember that when we looked at it, we said that the problem was to maximize the sum from i equals 1 to n, of p sub i, x sub i, maybe it should be to n minus 1, but we won't worry about that. And we had to do it subject to the constraint that the sum from 1 to n of the weight sub i times x sub i, remember x is if it was in, 1 if it wasn't, was less than or equal to the cost, as I wrote it this time, which was the maximum allowable weight. What do we do if we want to add volume, is an issue? Does this change? Does the goal change? You're answering. Answer out -- no one else can see you shake your head.

    STUDENT: No.

    PROFESSOR: No. The goal does not change, it's still the same goal. What changes?

    STUDENT: The constraints.

    PROFESSOR: Yeah, and you don't get another bar. The constraint has to change. I've added a constraint. And, what's the constraint I've added? Somebody else -- yeah?

    STUDENT: You can't exceed the volume that the knapsack can hold.

    PROFESSOR: Right, but can you state in this kind of formal way?

    STUDENT: [INAUDIBLE]

    PROFESSOR: -- sum from i equals 1 to n --

    STUDENT: [INAUDIBLE]

    PROFESSOR: Let's say v sub i, x sub i, is less than or equal to, we'll write k for the total allowable volume. Exactly. So the thing to notice here, is it's actually quite a simple little change we've made. I've simply added this one extra constraint, nice thing about thinking about it this way is it's easy to think about it, and what do you think I'll have to do if I want to go change the code? I'm not going to do it for you, but what would I think about doing when I change the code?

    Well, let's look at the simple version first, because it's easier to look at. At the top. Well basically, all I'd have to do is go through and find every place I checked the constraint, and change it. To incorporate the new constraint. And when I went to the dynamic programming problem, what would I have to do, what would change? The memo would have to change, as well as the checks, right? Because now, I not only would have to think about how much weight did I have available, but I have to think about how much volume did I have available. So whereas before, I had a mapping from the item and the weight available, now I would have to have it from a tuple of the weight and the volume. Very small changes. That's one of the things I want you to sort of understand as we look at algorithms, that they're very general, and once you've figured out how to solve one problem, you can often solve another problem by a very straightforward reduction of this kind of thing. All right, any questions about that. Yeah?

    STUDENT: I had a question about what you were talking about just before.

    PROFESSOR: The pseudo-polynomial?

    STUDENT: Yes.

    PROFESSOR: Ok.

    STUDENT: So, how do you come to a conclusion as to which you should use then, if you can determine the size based on solution, or based on input, so how do you decide?

    PROFESSOR: Great question. So the question is, how do you choose an algorithm, why would I choose to use a pseudo-polynomial algorithm when I don't know how big the solution is likely to be, I think that's one way to think about it. Well, so if we think about the knapsack problem, we can look at it, and we can ask ourselves, well first of all we know that the brute force exponential solution is going to be a loser if the number of items is large. Fundamentally in this case, what I could look at is the ratio of the number of items to the size of the knapsack, say well, I've got lots items to choose from, I probably won't put them all in. But even if I did, it would still only be 30 of them, right? It's hard. Typically what we'll discover is the pseudo-polynomial algorithms are usually better, and in this case, never worse. So this will never be worse than the brute force one. if I get really unlucky, I end up checking the same number of things, but I'd have to be really, it'd have to be a very strange structure to the problem. So it's almost always the case that, if you can find a solution that uses dynamic programming, it will be better than the brute force, and certainly not, well, maybe use more space, but not use more time. But there is no magic, here, and so the question you asked is a very good question. And it's sometimes the case in real life that you don't know which is the better algorithm on the data you're actually going to be crunching. And you pays your money and you takes your chances, right? And if the data is not what you think it's going to be, you may be wrong in your choice, so you typically do have to spend some time thinking about it, what's the data going to actually look like. Very good question. Anything else?

    All right, a couple of closing points before we leave this, things I would like you to remember. In dynamic programming, one of the things that's going on is we're trading time for space. Dynamic programming is not the only time we do that. We've solved a lot of problems that way, in fact, by trading time for space. Table look-up, for example, right, that if you're going to have trig tables, you may want to compute them all at once and then just look it up. So that's one thing. Two, don't be intimidated by exponential problems. There's a tendency for people to say, oh this problem's exponential, I can't solve it. Well, I solve 2 or 3 exponential problems before breakfast every day. You know things like, how to find my way to the bathroom is inherently exponential, but I manage to solve it anyway. Don't be intimidated. Even though it is apparently exponential, a lot of times you can actually solve it much, much faster.

    Other issues. Three: dynamic programming is broadly useful. Whenever you're looking at a problem that seems to have a natural recursive solution, think about whether you can attack it with dynamic programming. If you've got this optimal substructure, and overlapping sub-problems, you can use dynamic programming. So it's good for knapsacks, it's good for shortest paths, it's good for change-making, it's good for a whole variety of problems. And so keep it in your toolbox, and when you have a hard problem to solve, one of the first questions you should ask yourself is, can I use dynamic programming? It's great for string-matching problems of a whole variety. It's hugely useful. And finally, I want you to keep in mind the whole concept of problem reduction. I started with this silly description of a burglar, and said : Well this is really the knapsack problem, and now I can go Google the knapsack problem and find code to solve it. Any time you can reduce something to a previously solved problem, that's good. And this is a hugely important lesson to learn. People tend not to realize that the first question you should always ask yourself, is this really just something that's well-known in disguise? Is it a shortest path problem? Is it a nearest neighbor problem? Is it what string is this most similar to problem? There are scores of well-understood problems, but only really scores, it's not thousands of them, and over time you'll build up a vocabulary of these problems, and when you see something in your domain, be it physics or biology or anything else, linguistics, the question you should ask is can I transform this into an existing problem? Ok, double line. If there are no questions I'm going to make a dramatic change in topic.

    We're going to temporarily get off of this more esoteric stuff, and go back to Python. And for the next, off and on for the next couple of weeks, we'll be talking about Python and program organization. And what I want to be talking about is modules of one sort, and of course that's because what we're interested in is modularity. How do we take a complex program, again, divide and conquer, I feel like a 1-trick pony, I keep repeating the same thing over and over again. Divide and conquer to make our programs modular so we can write them a little piece at a time and understand them a little piece at a time. Now I think of a module as a collection of related functions. We've already seen these, and we're going to refer to the functions using dot notation. We've been doing this all term, right, probably somewhere around lecture 2, we said import math, and then somewhere in our program we wrote something like math dot sqrt of 11, or some other number. And the good news was we didn't have to worry about how math did square root or anything like that, we just got it and we used it. Now we have the dot notation to avoid name conflicts. Imagine, for example, that in my program I wrote something like import set, because somebody had written a module that implements mathematical sets, and somewhere else I'd written something like import table, because someone had something that implemented look-up tables of some sort, something like dictionaries, for example. And then, I wanted to ask something like membership. Is something in the set, or is something in the table? Well, what I would have written is something like table dot member. And then the element and maybe the table. And the dot notation was used to disambiguate, because I want the member operation from table, not the member one from set. This was important because the people who implemented table and set might never have met each other, and so they can hardly have been expected not to have used the same name somewhere by accident. Hence the use of the dot notation.

    I now want to talk about a particular kind of module, and those are the modules that include classes or that are classes. This is a very important concept as we'll see, it's why MIT calls things like 6.00 subjects, so that they don't get confused with classes in Python, something we really need to remember here. Now they can be used in different ways, and they have been historically used in different ways. In this subject we're going to emphasize using classes in the context of what's called object-oriented programming. And if you go look up at Python books on the web, or Java books on the web, about 80% of them will include the word object-oriented in their title. Object-oriented Python programming for computer games, or who knows what else. And we're going to use this object-oriented programming, typically to create something called data abstractions. And over the next couple of days, you'll see what we mean by this in detail. A synonym for this is an abstract data type. You'll see both terms used on the web, and the literature etc., and think of them as synonyms. Now these ideas of classes, object-oriented programming, data abstraction, are about 40 years old, they're not new ideas. But they've only been really widely accepted in practice for 10 to 15 years. It was in the mid-70's, people began to write articles advocating this style of programming, and actually building programming languages, notably Smalltalk and Clue at MIT in fact, that provided linguistic support for the ideas of data abstraction and object-oriented programming. But it really wasn't until, I would say, the arrival of Java that object-oriented programming caught the popular attention. And then Java, C++ , Python of, course. And today nobody advocates a programming language that does not support it in some sort of way. So what is this all about? What is an object in object-oriented programming?

    An object is a collection of data and functions. In particular functions that operate on the data, perhaps on other data as well. The key idea here is to bind together the data and the functions that operate on that data as a single thing. Now typically that's probably not the way you've been thinking about things. When you think about an int or a float or a dictionary or a list, you knew that there were functions that operated on them. But when you pass a parameter say, a list, you didn't think that you were not only passing the list, you were also passing the functions that operate on the list. In fact you are. It often doesn't matter, but it sometimes really does. The advantage of that, is that when you pass an object to another part of the program, that part of the program also gets the ability to perform operations on the object. Now when the only types we're dealing with are the built-in types, the ones that came with the programming language, that doesn't really matter. Because, well, the programming language means everybody has access to those operations. But the key idea here is that we're going to be generating user-defined types, we'll invent new types, and as we do that we can't assume that if as we pass objects of that type around, that the programming language is giving us the appropriate operations on that type.

    This combining of data and functions on that data is a very essence of object-oriented programming. That's really what defines it. And the word that's often used for that is encapsulation. Think of it as we got a capsule, like a pill or something, and in that capsule we've got data and a bunch of functions, which as we'll see are called methods. Don't worry, it doesn't matter that they're called methods, it's a historical artifact.

    All right, so what's an example of this? Well, we could create a circle object, that would store a representation of the circle and also provide methods to operate on it, for example, draw the circle on the screen, return the area of the circle, inscribe it in a square, who knows what you want to do with it. As we talk about this, as people talk about this, in the context of our object-oriented programming, they typically will talk about it in terms of message pass, a message passing metaphor. I want to mention it's just a metaphor, just a way of thinking about it, it's not anything very deep here. So, the way people will talk about this, is one object can pass a message to another object, and the receiving object responds by executing one of its methods on the object. So let's think about lists. So if l is a list, I can call something like s dot sort, l dot sort. You've seen this. This says, pass the object l the message sort, and that message says find the method sort, and apply it to the object l, in this case mutating the object so that the elements are now in sorted order. If c is a circle, I might write something like c dot area. And this would say, pass to the object denoted by the variable c, the message area, which says execute a method called area, and in this case the method might return a float, rather than have a side-effect.

    Now again, don't get carried away, I almost didn't talk about this whole message-passing paradigm, but it's so pervasive in the world I felt you needed to hear about it. But it's nothing very deep, and if you want to not think about messages, and just think oh, c has a method area, a circle has a method area, and c as a circle will apply it and do what it says, you won't get in any trouble at all.

    Now the next concept to think about here, is the notion of an instance. So we've already thought about, we create instances of types, so when we looked at lists, and we looked at the notion of aliasing, we used the word instance, and said this is 1 object, this is another object, each of those objects is an instance of type list. So now that gets us to a class. A class is a collection of objects with characteristics in common. So you can think of class list. What is the characteristic that all objects of class list have in common, all instances of class list? it's the set of methods that can be applied to lists. Methods like sort, append, other things. So you should think of all of the built-in types we've talked about as actually just built-in classes, like dictionaries, lists, etc. The beauty of being able to define your own class is you can now extend the language. So if, for example, you're in the business, God forbid, of writing financial software today, you might decide, I'd really like to have a class called tanking stock, or bad mortgage, or something like that or mortgage, right? Which would have a bunch of operations, like, I won't go into what they might be. But you'd like to write your program not in terms of floats and ints and lists, but in terms of mortgages, and CDOs, and all of the objects that you read about in the paper, the types you read about. And so you get to build your own special purpose programming language that helped you solve your problems in biology or finance or whatever, and we'll pick up here again on Tuesday.
  • 25
    MIT Lecture 15: Abstract Data Types, Classes and Methods
    50:25
    Abstract data types, classes and methods. Instructors: Prof. Eric Grimson, Prof. John Guttag.

    LECTURE TRANSCRIPT

    PROFESSOR: Last time, Professor Guttag introduced the idea of objects and classes and this wonderful phrase called object-oriented programming. And it's a topic I want to pick up on today, we're going to do for the next few lectures, and it's a topic I want to spend some time on because this idea of capturing data and methods, the term we're going to use for it, but data and functions that belong to that data, things that can be used to manipulate them, is a really powerful one. What we're really getting at is the idea of saying I want to have a way of grouping together information into units that make sense. So I can go back to one of those topics we had at the beginning, which is the idea of abstraction, that I can create one of those units as a simple entity, bury away the details and write really modular code. And so we're going to talk about that a lot as we go along. What we're really doing, or I shouldn't say what we're really doing, a basic piece of what we're doing, when we talk about classes or objects, is we're doing something that Professor Guttag mentioned, we're defining an abstract data type.

    Now what in the world does that mean? Well basically what we're doing is we're giving ourselves the ability to create data types the same way that we have some built-ins, so we have things like int, float, string, these are built-in data types. And if you think about it, associated with each one of those data types is a set of functions it's intended to apply to. Sometimes the functions -- sometimes a function can be used on multiple data types, plus, for example, we saw could add strings, or could add ints, but each one of those data types has associated with it a set of functions that are geared to handling them. We want to do the same thing, but with our data types. We want to create data types and functions, or we're going to call them methods, that are specifically aimed at manipulating those kinds of objects. And our goal is then to basically see how we can build systems that take advantage of that modularity.

    Right, so the fundamental idea then, is, I want to glue together information, I want to take pieces of data that naturally belong together, glue them together, and attach some methods to it. And that's, you know, saying a lot of words, let's do an example because it's probably easiest to see this by looking at a specific example. So here's the example I'm going to start with. Suppose I want to do little piece of code that's going to do planar geometry, points in the plane. All right, so I want to have some way of gluing those things together. Well you know what a point is, it's got an x- and a y- coordinate, it's natural to think about those two things as belonging as a single entity. So an easy way to do this would be to say, let's just represent them as a list. Just as a 2-list, or a list of 2 elements. It's easy to think of a point as just a list of an x- and a y- coordinate. OK, for example, I might say point p1 is that list, x is 1, y is 2. in fact, if I draw a little simple -- it's basically pointing to that point in the plane, right, x is 1, y is 2.

    OK, fine, there's another way to represent points on the plane now, and that's in polar form, so this, if you like, is Cartesian. Another way to represent a point in a plane is I've got a radius and I've got an angle from the x-axis, right, and that's a standard thing you might do. So I might define, for example, in polar form p 2, and let me see, which example did I do here, we'll make this the point of radius 2 and at angle pi by 2, I'm going to make it easy because pi by 2 is up along this axis, and that's basically that point. Ok, just fine, it's no big deal. But here now becomes the problem. I've glued things together but just using a list. Suppose I hand you one of these lists. How do you know which kind it is? How do you know whether it's in Cartesian form or in polar form? You have nothing that identifies that there, you have no way of saying what this grouping actually means. Right, and just to get a sense of this, let's look at a simple little example, so on your hand-out, you'll see I've got a little piece of code that says assuming I've got one of these points, I want to do things with it, for example I might want to add them together.

    So this first little piece of code right here says, ok you give me 2 points, I'll create another 1 of these lists and I'll simply take the x, sorry I shouldn't say x, I'm going to assume it's the x, the x-values are the two points, add them together, just right there, the y-values, add them together and return that list. And if I actually run this, which I'm going to do -- excuse me, do it again -- OK, you can see that I've added together and I've printed out the value of r, and I'll just show you that in fact that's what I've got. This looks fine, right, I'm doing the right thing. Another way of saying it is, I've actually said, what did I use there, (1,2) and (3,1), It's basically saying there is the first point, there's the second point, add them together and I get that point. OK, that sounds fine.

    Now, suppose in fact these weren't x and y glued together, these were radius and angle glued together. In that case point p 1 doesn't correspond to this point, it actually corresponds to the point of radius 2 and angle 1, which is about here. I think I wrote this down carefully so I would make sure I did it right. Sorry, said that wrong, radius 1 and angle 2, 2 radians is a little bit more than pi half. And the second point is of radius 3 and angle 1, which is up about there. So what point, sorry, bad pun, what point am I trying to make here? Different understandings of what that piece means gives you different values, and that's a bit of a problem. The second problem is, suppose actually I had p 1 and p 2 were in polar form, and I ran add points on them. This little piece of code here that I did. Does that even make any sense? Course not, right? You know when you add 2 polar forms, you add the radii together, you don't add the angles together, you need to do it in Cartesian form.

    So what I'm leading up to here is that we've got a problem. And the problem is, that we want to build this abstract data type, but we'd like to basically know what kind of object is it, and what functions actually belong to it, how do we use them? And so I'm going to go back to this idea of a class, and let's build the first of these, and that is shown right here on this piece of your handout. I'm going to define a class, and in particular, what I'm going to do, is walk through what that says. So I'm going to now build an object, it's going to represent a point. So what does that thing say up there? It's got this funky looking form, right, it says, I've got something that I'm going to call a class, got that key word class right here. And I'm going to give it a name, and right now I'm just building a simple piece of it -- but first of all, what does a class do? Think of this as, this is a template for creating instances of an object. At the moment, it's a really dumb template. I'm going to add to it in a second, but I want to build up to this. Right now it's got that second key word there called pass, which is just Python's way of saying there's an empty body in here. Right,, we're going to add to it in a second, but the idea is class is going to be a template for creating instances.

    How do I use it? Well, I call class just like a function, and you can see that below. Having created this thing called Cartesian point, I'm going to create two instances of it. c p 1 and c p 2. Notice the form of it, it's just the name of the class followed by open paren, close paren, treating it like a function. What that does, is that it creates, c p 1 and c p 2 are both instances of this type, specific versions of this type. For now the way to think about this is, when I call that class definition, it goes off and allocates a specific spot in memory that corresponds to that instance. Right now it's empty, actually it's not quite empty, it has a pointer back to the class. And I can give a name to that, so c p 1 and c p 2 are both going to point to that. Once I've got that, I can now start giving some variable names, sorry not, rephrase that, I can give some attributes, I can give some characteristics to these classes. So each instance has some internal, or will have some internal attributes. Notice how I did that up there. Having created c p 1 and c p 2, I had this weird looking form here. Not so weird, you've actually seen it before. In which I said c p 1 dot x equals 1.0.

    What's this doing? c p 1 points to an instance, it points to a particular version of this class. And I have now given an internal variable name x and a value associated with that. So I've just given it an x variable. All right, c p 1 dot y, I've said assign that to the value 2, 2,0. So now c p 1 has inside of it an x and y value. Did the same thing with c p 2, give it a different x and y value. Again, remind you, c p 2 is a different instance of this data type. All right, when I call the class definition it goes off and finds another spot in memory, says that the spot I'm going to give you a pointer back to that, give it the name c p 2, and then by running these 2 little assignments statements here, I've given it an x and a y value for c p 2.

    So you see why I say it's a template, right? Right now it's a simple template, but it's a template for creating what a class looks like, and I now have an x- and y- value associated with each instance of this. OK, and if I wanted to look at it, we can come back over here, and we can see what does c p 1 look like, interesting. It says some funky stuff, and says it's a kind of Cartesian point. And that's going to be valuable to me when I want to get back to using these things, right? You see that little thing says dot Cartesian point in there. If I want to get out right now the versions of these things, I can ask what's the value of c p 1 x, and it returns it back out. I could say c p 2 dot x, that was a bad one to use because they use the same valuable in both places, didn't I? So let's do c p 1 dot y, c p 2 dot y.

    OK, so I've just created local versions of variables with each one of these objects. I can get at them just like I would before, I can assign them in as I might have done before. OK, now that I've got that, we could think about what would I want to do with these points? Well one thing I might want to do is say, is this the same point or not? So the next little piece of code I've written here, just move down to it slightly. I've got a little piece of code called same point. And you can look at it. What does it say to do? It says, if you give me two of these data objects, I'm going to call them p 1 and p 2. I'm going to say, gee, is the x value the same in both of them, and if it is, and the y value's the same, then this is the same point, I'm going to return true. Notice the form. This is saying, that's a class, or sorry, an instance of a class, and I'm going to get the x value associated with it. I going to come back in a second to how it actually does that, but it basically says, get me x value for p 1, get me the x value for p 2, compare them, just as you would normally. I've got another little thing here that I'm going to use a little later on that just prints out values of things.

    OK, let's see what happens if I do this. Let me show you simple little example. I'm going to go over here, and let me define a couple of these things. I'm going to say p 1, try it again, p 1 is a Cartesian, it would help if I could type, Cartesian point, and I'm going to say p 1 of x is 3, p 1 of y is 4, and I'm going to make p 2 another Cartesian point. And I'll give it an x value of 3 and a y value of 4. OK, now I want to say, are these the same? Thing I've got a little procedure that could do that, but you know the simplest thing I could do is to say well, gee, wait a minute, why don't I just check to see if these are the same thing? So I can say is p 1 the same as p 2, using Scheme's built-in is comparator. Say -- sorry?

    PROFESSOR 2: Part of Python?

    PROFESSOR: Part of Scheme, whoa, there's a Freudian slip, thank you, John. I'm showing my age, and my history here, is p 1 and p 2 the same thing? Hey, there's a bad English sentence even worse, now I'm really thrown off. I'm using Python's is comparator to say is it the same thing? It says no. But if I say, are p 1 and p 2 the same point, it says yes. And this is a point I want to stress here. So what's going on in this case is, I want to distinguish between shallow equality and deep equality. The first thing is testing shallow equality. What it is doing, that's another bad English sentence, but what it is doing? Is is essentially saying, given 2 things, do they point to exactly the same referent? Or another way of thinking about it, is remember I said when I call that class definition it creates an instance, that's a pointer to some spot in memory that's got some local information around it. Is is saying, do these things point to exactly the same spot in memory, the same instance. Deep equality, we get to define, that's what I did by writing same point. OK, as I said, I want equality in the case of points to be, are the x- and y- coordinates the same? And I'm actually going to change it. just to show you this point. If I do the following, and I say, I'm going to assign p 1 to be p 2. What's that doing? It's taking the name p 1 and it's changing its value to point to exactly what p 2 points to. And then I say, are they the same thing? Answer's yes, because now they are pointing to exactly the same spot in memory. The same instance.

    OK, the reason I'm saying this is, we have one class definition, is a cookie cutter, it's a template that's going to let us build versions of these things. Every time I use it, I'm creating a new instance, that's a different thing inside of memory. And I want to have that because I want to have lots of versions of points. OK, now, let's go back to where I was. I said one of the things I want to do is, I want to have different versions of points. So I've got now things that are Cartesian points. I could do the same thing, I could build polar point. I wanted to show it to you here. I've got a class called polar point, which is right there, and same kind of thing, I can create instances of it, and then assign to them things like a radius and an angle, make instances of those. OK, John?

    PROFESSOR 2: I just want to maybe mention that in some of the reading, you'll see terms like object equality and value equality, instead of shallow equality and deep equality.

    PROFESSOR: Right, so, this object, this is, right, value quality. Right. And you will see both terms used. Some people like to use shallow and deep, object and value, but they're talking about the same thing, which is it the same object or is it the same, in this case, set of values, depending on what you want to define as you use it. OK, so as I said, now I can go off and I could create a different class. I've got Cartesian points, I could create a polar points. And I'm going to run it in a sec, but you can see, the same kind of idea. I define a class call polar point, I create a couple of them, and I give them a radius and an angle. And then I could do things like again, say, okay having done, that let me just run it here, run that, so I've now got polar point 1, and polar point 2. I can say is polar point 1 the same as polar point 2, and the answer should be no. And then I could say well, gee, are they the same point? Oops. What happened? Well it bombed out. Because, what was I expecting to do? I was expecting to compare x- and y- values, not radius and angle. And so this doesn't know how to do it, it doesn't have a method to deal with it, so it complains.

    So what's my problem here, and this is what I want to now lead up to. I could imagine writing another function for same point, and I have to give it a name like same point polar, and same point Cartesian. A different function to compare polar versions of these points. But that's starting to get to be a nuisance. What I'd really like to do is to have 1 representation for a point that supports different ways of getting information out, but has gathered within it, a method or a function for dealing with things like how do I know if it's the same point or not. So I want to take this idea classes now, and I want to generalize it. Right, and that is going to lead us then to this funky looking thing. Right there, and I'd like you to look at that in your handout. OK, I'm going to go back and rebuild the class. Ok, and again, I'm going to remind you, the class is this template. But now I'm going to change it, so what is that new version of class say. I'm going to call it c point just to make it a little shorter. You can see inside of it, it's got a set of definitions for things like functions. And that first one is this kind of interesting thing, it's two underbars, init, and two underbars. Underscores, I guess is the right way to say it, not underbars. Right that's a specific name, and what it basically says is, when I call the class instance. That's a bad mistake. When I call the class definition, that is I call c point, I'm going to call it with a specific set of arguments. And what is it going to happen is that init is going to then apply. It's going to apply to those arguments.

    So let me in fact show you an example. I've got a definition of Cartesian point, I've got a definition of polar point. Let me just run these to get them in there. Now let's do the following. Let's let p be Cartesian point, and we'll give it a couple of values. OK? So what happened? Notice in the class definition here, is there, this is the first thing that's got called, and I just called with the value for x and the value for y, and it went off and did something for me. Does that look right? This is where you all hate it, I get no eye contact anywhere. Anything look odd about that? I said. When I call this class definition, it calls init, and I give it an x and a y value. How many arguments does init take? Three. How many arguments did I give it? Two. What in the world's going on? Well, this is a piece of object-oriented coding that we get to talk about a little bit. There's this weird extra variable in there called self. So what is self? And I have to admit, I did the standard thing you do every time you run across something you don't know about, you go to Wikipedia. So I went and looked up self in Wikipedia, and I have to read it out.

    Wikipedia informs us that the self is the idea of a unified being, which is the source of an idiosyncratic consciousness. Moreover, this self is the agent responsible for the thoughts and actions of an individual to which they are ascribed. It is a substance which therefore endures through time, thus thoughts and actions at different moments of time may pertain to the same self. OK, how do we code that up? Sounds like an AI problem, I guess right?

    But there's actually hidden in there an important element, and that is, when I create an instance, I have to be able to get access to the things that characterize that instance. I won't say that they're thoughts and emotions or things, but what characterizes an instance here, it's the internal parameters that specify what is going on. So in fact what happens inside of an object-oriented system, and particularly in Python's object-oriented system, is the following. When we call init, it's going to create the instance, all right, just as we said before. But in particular, it's going to use self to refer to that instance. Right, so let me say this a little differently.

    I have a class definition. It's actually an object somewhere. It has inside of it all those internal definitions. When I call that class definition, it calls init. Init creates a pointer to the instance. And then it needs to have access to that, so it calls it, passing in self as the pointer to the instance. That is, it says it has access to that piece in memory, and now inside of that piece of memory, I can do things like, as you see here, define self dot x to be the value passed in for x.

    What's that doing? It's saying where's self pointing to? Inside of that structure, create a variable name x, and a value associated with it. Notice what I also do here, I create self dot y, give it a value, and then, oh cool, I can also set up what's the radius and angle for this point, by just doing a little bit of work. OK, in fact if you look at what it does there, just put the pointer over here, it says, get the value of x that I just stored away, square it, add it to the value of y squared that I just stored away, and then take square root, pass it back out. So I just computed the radius of that particular thing. Right? Compute the angle the same way, just using the appropriate things. So the idea is that self will always point to the particular instance.

    Now you might say, why? Why do it this way? Well, basically because it was a design choice when the creators of Python decided to create the language, they basically said, we're always going to have an explicit pointer to the instance. Some other object-oriented programming languages do not provide that pointer. This is kind of nice in my view, I don't know if John, you'd agree, but this is explicit. It actually lets you see how to get access to that pointer so you know what you're referring to. But it's simply design choice. So another way saying it again is, when I call the class definition, by default I'm going to look to see is there an init method there, and if there is, I'm going to use it. First argument by convention is always self, because it has to point to the instance, and then I pass, in this case, another couple of arguments in.

    OK, now, if I actually do this, and I'm going to show you the example, I just, what did I type over there, I got p was a c point. If I want to get values back out, I could in fact simply send to that instance a message, in this case I could say p dot x. In fact let's do it. If I do that over here -- aha -- it gets me back the value. Now let me spend just a second to say, what was this actually doing? p is an instance. It knows, or has stored away, and in fact let's look at it, if we look at what p does, p says -- it says reading through a little bit of this stuff here, it says -- it's a kind of Cartesian point, it's an instance, there's actually the memory location that it's at, that's why I say this idea of it's an instant at a specific spot. It knows that it came from this class, c point. So when I type, I'm sorry, I shouldn't say type, when I write, although I would have typed it, p dot x, here's what basically happens. p is an instance, it's being sent a message, in this case the message x, it says I want the x-value back out. p knows that it is a kind of Cartesian point, it actually goes and gets, if you like, the class definition up here. And is able to then say, inside of that class definition, find the value of x. All right, now, that's one of the ways we could get things out, but in fact it's really not a good way.

    A better way to do this would be the following. If I could type. What did I just do there? One of the things that I defined inside my class definition here was an internal method. That method has a name, obviously, and what does it do? It's going to go off and get the values of x and y attached to this thing and return them to me. And that's one of the things I want. I would like my classes to have methods. So you can access the values of the specific instance. Now, this is still a nuance, why would I like to do this? Well this is leading up to why I want to gather things together in classes to start with. It's perfectly legal in Python to type that in and get the value back out. As I said, I would prefer to do something that uses an accessor that I just wrote. So p dot Cartesian is a kind of accessor, it's getting access to the data. And here's why I'd like to have it. Right now, I still have the problem that those classes, those instances of classes, are exposed. What do I mean by that? Here's something I could do. Let's do it in fact. OK. What point in the plane does p now point to? X-axis is foobar y-axis ought to be foobass something else, right? I know it looks like a simple and silly little example, but at the moment, I still have the ability to go in and change the values of the parameters by that little definition. And this makes no sense. And this is because I don't have something I would really like to have, which is data hiding.

    So you'll see lots of definitions of this. I think of data hiding as basically saying, one can only access instance values, or, we'll call them that, instance values through defined methods. And that's a wonderful thing to have because it gives you that modularity, that encapsulation that basically says, when I create a point, the only way I can get at the values, is by using one of the defined methods, in this case it could be Cartesian, and get all the pieces of that. Unfortunately, Python doesn't do this. Which is really a shame. Or another way of saying it is, please don't do that. Don't go in and change the values of things by using the direct access. Have the computational hygiene, if you like, to only go through accessors, only go through methods that are actually provided to you as you do this. I actually don't remember, John, C++ does have data hiding, I think, right?

    PROFESSOR 2: And not only shouldn't you change it, you shouldn't even read it.

    PROFESSOR: Exactly. What you're going to see in a second I violated in some of my code, which Professor Guttag is going to yell at me shortly because I should have done it through accessors, but, he's exactly right. A good, hygienic way of doing this is, not only do I not go in and change things except through a pre-defined method, I shouldn't read it other than through a pre-defined method. I should use Cartesian or polar to pull out those pieces of it.

    Once I've got that, you notice I can now define a polar point, same way. Notice I've now solved one of my problems, which is, in each one of these cases here, I'm creating both x y and radius angle values inside of there. If it's in polar form I passed in a radius and angle and I'll compute what the x- and y- value is. If its in Cartesian form I'll pass in an x and y and compute what a radius and angle is. But it now says that in any, in no matter what kind of form I made it from, I can get out that kind of information. So for example I defined p, remember back over here, as a Cartesian point, but I can actually ask for its polar form. It's there accessible to me. OK, this is great. Just to drive home one more reason why I don't want to have changes to the values other than through pre-defined things. Notice what happens if I do the following. I could say I want to change the radius of this particular thing. OK, perfectly reasonable thing to do. And if I go look at the polar form of this, OK, good, looks right, right? It's now got a different radius, same angle, so I just changed the radius of it.

    Oh, but what happened to the Cartesian form. I should have done this earlier by typing the Cartesian form earlier, so let me go back to where I was, sorry for that, let me go make this a 1 again. If I look at the Cartesian, oh, I did have the Cartesian form, don't mind me while I mutter to myself here quietly. Yeah, that's right, I did screw that up badly.

    All right, we try one more time, here we go, let's try one more time. We'll make p a new point, ok? There's the Cartesian representation of it, which is right, (1,2). Here's the polar representation of it, some random set of numbers which makes sense. If I now say, I'm going to go ahead and change the radius of this, something, my polar form did it right, but what happened to the Cartesian form? Ah yes, didn't change. Which makes sense if you think of my code. I didn't have anything in there that says, if you change one of these values, other values depend on it, and I want to make that change to it. So this is one more example of stressing why I only want to come access to the instances through defined methods. Because I could've built that in, it says if you change the value of this thing, by the way you need to change recompute those other values in order to make this hold up.

    OK, so what else do I have then in my little class definitions here? So, I've got an init in both cases. I don't have to put an init in, but it's again, usually a good idea to put that in originally. I've got and init that says, when you create an instance, here's what you do. Notice that that typically also defines for me what the internal variables are, what the internal characteristics of the class are going to be. Again, I could have some other functions to compute things, but this is typically the place where I'm going to put them in. So this is giving me now that template, better way of saying it, all right, a template now, for a point is x, y, radius, angle. And I can see that in those pieces there. And then I've got some things that get me back out information about them. But I got a couple of other of these strange looking things in there with underbars to them. So let's look at what some of the traditional methods for classes are in Python. I have init. This is what's actually going to create the instance, instantiate it, create what the set of variable values are for it. OK, I have another one in there, underbar, underbar, str. Anybody have a sense of what that's doing? What's s -- sorry, I heard something, sorry go ahead.

    STUDENT: Display what I have.

    PROFESSOR: Displaying what I have. Thank you. Yeah, I was going to say, think about what does str do, in general? It converts things into a string type. How do we typically print things, we convert them to strings. So str is basically telling us how we want to have it printed out. OK, in fact if we look at this, if I say, print of p, it prints it out in that form. Now this is actually a poor way to do it, because you might say, well, it's just the list. But remember, it wasn't a list. What does it do? It says, if I want to print out something I built in Cartesian form up here, says, again, I'm going to pass it in a pointer to the instance, that self thing, and then I'm going to return a string that I combine together with an open and close paren, a comma in the middle, and getting the x-value and the y-value and converting them into strings before I put the whole thing together. So it gives me basically my printed representation. OK. What else do I have in here? Well, I have cmp. My handout's wrong, which I discovered this morning after I printed them all out. So the version I'd like you to have uses, that, greater than rather than equals that I had in my handout. What's cmp doing as a method? Yeah?

    STUDENT: Comparing values?

    PROFESSOR: Yeah, comparing values, right? And again, it's similar to what cmp would do generically in Python. It's a way of doing comparisons. So this is doing comparisons. Now, I put a version up there, I have no idea if this is the right way to do comparisons or not. I said both the x- and y- coordinates are bigger, then I'm going to return something to it. And I think in the polar one I said, if, what did I do there, I said, yeah, again if the x and y are greater than the other one, I'm going to return them to it. The version in the handout, what was that actually doing? You could look at the handout. Well I think it was comparing, are they the same? So that would actually be another method I could put in. Underbar underbar eq, underbar underbar. Would be a default or generic way of doing, are these things the same? OK, in each case, what these things are doing, is they're doing, what sometimes gets referred to as operator overloading. I know you don't remember that far back, but in about the second lecture I made a joke of Professor Guttag which, you know, you didn't laugh at, he didn't laugh at, that's okay. In which I said, you know, I didn't like the fact that things like plus are overloaded, because you can use plus to add strings, you can use plus to add numbers, you can use plus to add floats. And he quite correctly, because he's more senior than I am, more experienced than I am, said it's actually a good thing. And he's right. Most of the time.

    The reason I say that is, by having operator overloading I can use 1 generic interface to all of the objects that I want to use. So it makes sense to be able to say, look for many methods I do want to have a way of doing comparison, and I don't have to remember, at top level, what the name of the comparison method was. I can simply use the built-in Sc -- about to say Scheme again -- the built-in Python comparison operation. Say, are these 2 things the same? Same thing with cmp, that's just saying greater than, and greater than now can apply to strings, it can apply to floats, it could apply to points, it could add other pieces into it. So there are some downsides, in my view, to doing operator overloading, but there's some real pluses. And the main one is, I get to just decide, how do I want to use this, and call it. Yes, ma'am?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Right, cmp other, so how would I call this? A good question. Here's the way I would call it. Let me give you, I'm going to create, a polar point, I'm going to call it q, and we'll give it some random values. OK, and now I want to know, is p greater than q? Now happens to return true here, but the question is, where's the other come from? P is a particular object type. When I try and evaluate that expression of greater than, is going to go into the class to say greater than is a comp method. So let me say it very carefully here. When I evaluate, yeah, when I evaluate this, p is an instance of a point, in this case it was actually a Cartesian point, it sends a message to the instance, which sends a message to the class, to get the cmp method from the class. And that then gets applied to itself, just p, and one other argument, which is the second piece there, so other points to the second argument that was present. OK. John?

    PROFESSOR 2: -- other, it could have said who or zort or --

    PROFESSOR: Yeah, sorry, that was part of the question, I could have a picked foobar could put anything in here. It's simply, notice the form of it here is, it's going to take two arguments, and you're right, self is the original instance. This says, I need a second argument to it, and that second argument better be a point so I can do the comparison. Yes ma'am?

    STUDENT: [INAUDIBLE]

    PROFESSOR: What do you think happens? Sorry, the question was, what happens if I said p is less than q? Got it, yes? Seems pretty obvious, right? Next time I bring the right glasses. It's still calling cmp, but it's knowing that cmp is just reversing the order of the arguments. Ok, which makes sense. If greater than takes, expects, arguments in order x y, less than simply takes greater than, but with the arguments reversed. OK, so I don't have to, it's a great question, I don't have to create a second one for cmp. Cmp is just saying, is this bigger than, and if I want to reverse it, it goes the other way. Question?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Or equal equal? Let's try equal equal because I didn't define it here. It says they're not the same, and boy, I need help on this one, John, it's not, there's no pre-defined eq in there.

    PROFESSOR 2: So, what cmp does, and maybe this isn't exactly the right way to write is, is cmp actually returns 1 of 3 values. A 0, minus a positive value, zero or a negative value, depending upon whether it's less than, equal, or greater than.

    PROFESSOR: Right.

    PROFESSOR2: So it's not really a Boolean-valued function. It has 3 possible values it could return.

    PROFESSOR: And so in this case, it's using the same piece, but it's returning that middle value that says they're actually the same. Right, one the things you can see now is, we start building up classes, we get these methods. So you can actually say, how do I know which methods are associated with the class? For that, we can call dir. And what it does, is it gives me back a listing of all the things, all the methods, that are associated with it. Some of which I built: cmp, init, str. And there, notice, are the internal definitions and there are the internal variables. And in fact I should've said, we often call those things fields. So inside of an instance, associated with an instance, we have both methods and fields. These are both altogether called attributes of the instance. And then there were a couple of other ones in there that I hadn't actually dealt with.

    The reason I want to point this out to you is, if we go back up to the kinds of data objects we started with, floats, ints, strings, they actually behave the same way. They are instances of a class, and associated with that class is a set of methods. So for example, I can say, what are all the methods associated with the number, or the integer 1? And you probably recognize some of them in there, right, absolute value, add, comp, cors, well we didn't do cors, we did a bunch of other things. It could also say, what are the methods associated with the string, 1. I'm sure you can quickly graph it, but notice they aren't the same. That makes sense. We have some set of things we want to do with strings, and different set of things we want to do with numbers. But underlying Python is the same idea. These are instances of a class, and associated with that class are a set of methods, things that I can deal with. So this is a handy way of being able to see, what are in fact the methods that are available if I don't happen to remember them, and want to go back to them.

    OK, I want to spend the last few minutes just showing you a couple of other things that we can do in here. Let me see where I want to go with this. So let's add one more piece to this. OK, now that I've got points, I might want to do something with points. So an easy thing to do in planar geometry is I want to make a line segment. It's got a start point, it's got an end point. Right, if you want to think of it back over here. There's a line segment, it's got a starting point and ending point. Well, I can do the same thing. And the reason I want to use this as an example is, here's my little definition of segment. Again, it's got an initializer, or an instance creator, right there. Takes a start and an end point, just going to bind local variable names start and end to those pieces. But notice now, those aren't just simple things like numbers, those are actually points. And that's where the modularity comes in. Now I have the ability to say, I've got a new class, I can create instances of a line segment, and it's elements are themselves instances of points. OK? And then what might I want to do with the segment? I might want to get the length of the segment. And I know it's kind of, you can see it on your handout, it has the rest of the pieces over here.

    Ok, what's the geometry say? The length of a line segment? Well, it's Pythagoras, right? I take the difference in the x-values, squared, the difference in the y-values, squared, add them up, take the square root of that. Notice what this says to do. It says if I want to get the length of a segment, going to pass in that instance, it says from that instance, get the start point, that's the thing I just found. And then from that start point, get the x-value. Same thing, from that instance, get the endpoint, from that end point get the x-value, square. Add the same thing to the y-values, squared, take the square root. Yes, ma'am?

    STUDENT: So are you entering a tuple in for start and end?

    PROFESSOR: No. I'm entering -- well, let's look at the example right down here. In fact, let me uncomment it so we can look at it. All right. I'm going to uncomment that. So notice what I'm going to do. I'm going to build, this case, a Cartesian point, I'm going to build a second Cartesian point, and my segment passes in those class instances. All right, they're not tuples, they're simply an instance with some structuring. And in fact if I go off and run this, OK, what I was printing here was s 1 dot length, and that's -- What is it doing? S 1 is a segment. It has inside of it pointers to 2 points which are instances. And when I call length on this, it takes that starting point, sends it the message saying give me your x-coordinate, takes the endpoint, says give me your x-coordinate, and add them together. Now, I prefaced this a few minutes ago about saying Professor Guttag wasn't going to like me. He doesn't like me generally, but that's between he and I. He beats me regularly at tennis, which is why I don't like him. Sorry, John. This is being taped, which is really good, isn't it? So why am I saying that? I said that if I was really hygienic, and you can now wonder about how often do I shower? If I was really hygienic. I would only ever access the values through a method. And I'm cheating here, right, because what am I doing? I'm taking advantage of the fact that start is going to be a point, and I'm just directly saying, give me your x-value. So I don't know don't, John, I would argue if I'd written this better, I would have had a method that returned the x- and the y- value, and it would be cleaner to go after it that way. This is nice shorthand, all right, but it's something that in fact I probably would want to do differently.

    Why would I want to do it differently? Imagine that I've written code like this, written a bunch of code. And I originally decided I was going to have as points, it's going to have internal values of an x and a y. And then somewhere along the line, I decide to store things in a different representation. If I had had a clean interface, that I had a specific method to get those values out, I wouldn't have to change anything. Other than that interface. But here, if I decide I'm going to store things not in x and y, but with some other set of names, for example, I've gotta go back into these pieces of code that use the points, and change them. So I've lost modularity. I'd really like to have that modularity that says, I'm only going to get access to the values, not by calling their names, but by calling some specific method to get access to their names. You could argue, well, x is in some sense inherently a method, but it's not nearly as clean as what I would like.

    And the last piece I want you to see here, and then I'll let you go is, notice now how that encapsulation, that binding things together has really helped me. Given the abstraction, the notion of a point as an instance with some values, I can now start building segments. And I could now extend that. I could have, you know, polygonal figures, that are a sequence of segments. And I would be able to simply bury away the details of how those other instances are created from how I want to use them by simply calling methods on the classes. We'll come back to this next time.
  • 26
    MIT Lecture 16: Encapsulation, Inheritance, Shadowing
    50:23
    Lecture 16: Encapsulation, inheritance, shadowing Instructors: Prof. Eric Grimson, Prof. John Guttag View the complete course at: http://ocw.mit.edu/6-00F...

    LECTURE TRANSCRIPT

    PROFESSOR: Last lecture we were talking about classes, and object-oriented programming, and we're going to come back to it today. I'm going to remind you, we were talking about it because we suggested it is a really powerful way of structuring systems, and that's really why we want to use it, It's a very common way of structuring systems. So today I'm going to pick up on a bunch of more nuanced, or more complex if you like, ways of leveraging the power of classes. But we're going to see a bunch of examples that are going to give us a sense. I'm going to talk about inheritance, we're going to talk about shadowing, we're going to talk about iterators. But before get to it, I want to start by just highlighting, sort of, what was the point of classes? So I'll remind you.

    A class, I said, was basically a template for an abstract data type. And this was really to drive home this idea of modularity. I want the ability to say, I've got a set of things that naturally belong together, I'm going to cluster them together, I want to treat it like it's a primitive, I want to treat it like it's a float or an int or a string. Is this going to be a point or a segment or something different like that. So it's really a way, as I said, of just trying to cluster data together. And this is a notion of modularity slash abstraction where I'm treating them as primitives. But the second thing we talked about is that we also have a set of methods, using the special name method because we're talking classes. But basically functions that are designed to deal with this data structure. We're trying to group those together as well. So we cluster data and methods.

    Second key thing we said was, in the ideal case, which unfortunately Python isn't, but we'll come back to that, in the ideal case, we would have data hiding, and by data hiding, which is sort of a version of encapsulation, what we meant was that you could only get to the internal pieces of that data structure through a proscribed method. Proscribed meaning it's something I set up. So data hiding saying, you would only access the parts through a method. And as we said, unfortunately Python does not enforce this. Meaning that I could create one of these data structures, ideally I'd have a method, that I'm going to see some examples of that I used to get the parts out, unfortunately in Python you could take the name the instance dot some internal variable you'll get it back. It is exposed. And this is actually just not a good idea. So I suggested in my very bad humor, that you practice computational hygiene and you only use appropriate methods to get the parts out. OK didn't laugh the joke last time, you're not going to laugh at it this time, I don't blame you. All right, and then the last piece of this is that we said the class is a template. When we call that class, it makes an instance. So class is used to make instances, meaning particular versions, of that structure, and we said inside the instances we have a set of attributes. Internal variables, methods, that are going to belong to that structure.

    OK, so with that in mind, here's what I want to do. I'm going to show you a set of examples, and I want to warn you ahead of time, the code handout today is a little longer than normal because we want to build essentially an extended example of a sequence of examples of classes. We're going to see the idea, of which we're gonna talk about, of inheritance or hierarchy, in which we can have classes that are specializations of other classes. We're gonna see how we can inherit methods, how we can shadow methods, how we can use methods in a variety of ways. So this is a way of suggesting you may find it more convenient to put notes on the code handout rather than in your own notes. Do whatever you like, but I just wanted to alert you, we're going to go through a little more code than normal.

    So, the little environment I'm going to build is an environment of people. I'll build a simple little simulation of people. So I'm going to start off with the first class, which I've got up on the screen, and it's on your handout as well, which is I'm going to build an instance, or a class rather, of persons. I'm going to draw a diagram, which I'm gonna try and see if I can do well, over here, of the different objects we're going to have. So I've got, a class, and by the way a class is an object. Instances are also objects, but classes are objects. We're gonna see why we want that in a second. Because I'm gonna build an object, sorry a class, called a person. Now, let's walk through some of the pieces here. The first one is, there's something a little different. Remember last time we had that keyword class and then a name, that name, in this case, person says this is the name for the class, and then we would have just had the semicolon and a bunch of internal things. Here I've got something in parens, and I want to stress this is not a variable. All right, this is not a def, this is a class. I'm going to come back to it, but what this is basically saying is that the person class is going to inherit from another class, which in this case is just the built-in Python object class. Hold on to that thought, it's going to make more sense when we look at a little more interesting example, but I want to highlight that. All right now, if we do this, as I said before, we can create a version of a person, let me just call it per, person.

    OK? And what we said last time is, when we wanted to create an instance inside of this class definition, we've got one of those built-in things called init. I'm gonna again remind you, some of the methods we have, Underbar underbar init is going to be the thing that creates the instance. Actually slightly misspeaking, actually Python creates the instance, but it's one thing that fills it in. So in this case, I'm going to give it 2 arguments: Frank Foobar Now, you might have said, wait a minute, init here has 3 arguments: self, family name, and first name. So again, just to remind you, what we said happens here is that when I call this class, person, I'm creating an instance. We'll draw a little instance diagram down here. I'm going to give it the name per. And I should have said inside of person, we've got a set of things. We've got our underbar underbar init, we've got, what else do I have up there? Family name. And a bunch of other methods, down to say.

    What happens inside of Python is, when we called the class definition, person, it creates an instance, there it is. Think of it as a pointer to a spot in memory, and then what we do is, we call, or find, that init method, up here, and we apply it. And the first argument self, points to the instance. So this object here is what self looks at. Now you can see what init's going to do. It says, oh, inside of self, which is pointing to here, let me bind a variable, which was, can read that very carefully, it's family underbar name, to the value I passed in, which was 4. Same thing with first name. OK, so the reason I'm stressing this is, self we do not supply explicitly, it is supplied as pointing to the instance, it's giving us that piece of memory. And that is what then gets created. So here's, now, the instance for per. OK, and I put a little label on there, I'm going to call that an isALink, because it is an instance of that class. God bless you.

    All right, so once we got this, let's look at what we can do with person. That's why I built person here. And as I said, I've already bound basically, those two pieces. If I want to get a value out, I can give person, or per, rather, this instance, a messaging. In this case I want to get family, what did I say, family name out, now, again I want to stress, what is happening here? per is an instance, it's this thing here. When I say per dot family name, I'm sending it a message, in essence what that does is, it says, from here it's going to go up the chain to this class object and find the appropriate method, which was family name. It is then going to apply that to self, which points to this instance. And that allows it, therefore, is you can see on the code, to look up under self, what's the binding for family name, and print it back up. So self is always going to point to the instance I want and I can use it. OK what else do we have in here? We can get the first name, that's not particularly interesting.

    We've got 2 other special methods: that's cmp and str. All right, cmp is our comparison method. And since I, I was about to say I blew it last time, I misspoke last time, a wonderful phrase that politicians like to use, I misspoke last time. Let me clarify again what cmp will do. Underbar underbar cmp is going to be the method you're going to use to compare two instances of an object. Now, let's back up for second. If I wanted to test equality, in fact I could use underbar underbar eq, under under. It's natural to think about an equality tester as returning a Boolean, it's either gonna be true or false, because something's either equal to or not. In many languages, comparisons also return Booleans, which is why I went down this slippery slope. For many languages, either it's greater than or it's not. But Python is different. Python use cmp, in fact it has a built in cmp, which is what we're relying on here. Where am I, right there. And what cmp returns is 1 of 3 values. Given 2 objects, it says if the first one is less than the second one, it returns -1, if it's equal it returns 0, if it's greater than, it returns 1.

    So it allows you this broader range of comparisons. And if you think about it, cmp, you could apply on integers, you could apply it on floats, apply it on strings. So it's overloaded, it has the ability to do all of those. And in this case what we're saying is, given 2 objects, let's create a tuple of the first, sorry, family and first name of ourselves, and other is another object, family and first name of that, and then just use cmp to compare them. All right, so it's going to use the base pieces. OK, so it gives me a way of doing comparisons. And str we saw last time as well, this is cmp does comparison, and str is our printed representation.

    OK. So what we've got now, is a simple little class. We've also got two methods there. I want to look at them, we're gonna come back to them, but they start to highlight things we can do with our classes. So I've built one simple version of it here, which is per. And notice I've got another method, right up here, called say. And say takes two arguments, for the moment the second argument, or the first argument's, not going to make a lot of sense, but say takes two arguments besides itself. It's going to take another object to which it's saying something and the thing to say. Since I only have one object here, I'm going to have person talk to himself. You may have met a few other undergraduates who have this behavior. I'll have him talk to himself and say, just some random message the faculty members occasionally worry about. OK, what does this thing do? Now you're going to see some of the power of this. Again, remember, I'm down here, I'm sending this the message say, it's going to go up the chain to find the say message in person. And what does say do, it says given another object and some string, it's going to return, oh, and interesting things, part of which you can't see on the screen. First what it does, is it gets first name of self. Remember self is pointing to this instance, so it's simply looks up that binding, which is Frank. It's going to create a string in which it adds to that the family name of self, and then another thing that says to, and then ah, I'm now going to send a message to the other object, saying give me your first name. Going to add that to the second piece, and you can see in this case it happens to be the same first and family name. And then at the end of it, which you can't see here but you can see in your handout, I just append the whole string, so it spits it out.

    What's the point of this, other than I can get it to say things? Notice, I can now reference values of the instance. But I can also get values of other instances, by sending in a message. And that's why we have that form right there. And then it glued all together. If you think about this for a second, you might say, wait a minute, actually you might have said wait a minute a while ago, why am I just using the variable name there in the function over here? Well in fact, I could've used the function here, first name open close, right? It would have done the same thing. But because I know I'm inside the instance, it's perfectly reasonable to just look up the value. OK, I could have, although I don't want you to do it, have done the same thing there and used underbar, sorry, first name underbar, sorry, first underbar name, but that's really breaking this contract that I want to happen. I should send the message to get the method back out. So again the standard practices is if you know you're inside the object, you can just access the values. If you're doing it with any other objects, send it a message to get it out.

    OK, now, that gives you an ability to say, let's look at one more example here, and then we're going to start building our hierarchy, which is, that this person can also sing. And we've got a little sing method here. And notice what it does, it's going to sing to somebody, I guess you're part of the Chorallaries. You're going to sing something, and notice what it does, it's simply going to use its say method, but add at the end of whatever's being said, just tra la la at the end. So this is now an example of a method using another method. Why would you want that? It's nice modularly. I have one method that's doing saying, I have another method that's just building on it. So if I have is person sing to themselves, not a highly recommended activity, it would help if I had it sing to itself, not sing to sing, sorry about that. Notice what it does. Looks like exactly like a say method, except it's got tra la la at the end. Don't worry I'm not going to sing to you. I'll simply say the words. Power of this, other than the silly examples. You see how I can access variables of the instance, how I can access variables of other instances, going to come back to that, and how I can use versions of my own methods to implement other methods. In this case sing is using say as part of what it wants to get out.

    OK, so we got a simple little example. Now, let's start adding some other pieces to this. OK, and what do I want to add. Find my spot here. OK, we're going to add an MIT person. Sorry, machine is -- do this, let's go down. OK so I'm going to add an MIT person. Look at the code for second. Aha! Notice what this says. MIT person says it inherits from person. That is, that's the first thing in parens up there. It says, you know, class of MIT person is person. What that is saying is, that this is a specialization of the person class. Or another way of saying it is, we have a super class, in this case it's person. And we have a subclass, in this case its MIT person. And we're going to walk through some examples, but what it says is that that subclass of MIT person can inherit the attributes of the person class. Can inherit the methods, it can inherit variables.

    OK, what does MIT person do? Well, here's 1 of the new things it does. It has a local variable called next id num, which is initially set to 0. See that up there. And then it's got some methods, it's got an init method, a get id method, a few other things. OK, let's run this. In particular, I go back down to this one. Let me just uncomment this and do it here. Assuming my machine will do what I want it to do, which it really doesn't seem to want to do today. Try one more time. Thank you, yep. Still not doing it for me, John. OK, we type it. No idea what Python doesn't like me today, but it doesn't. So we're gonna define p 1, I've lost my keyboard, indeed I have. Try one more time. p 1 MIT person, see how fast I can type here -- OK, now, let's look at what the code does, because again it's going to highlight some things. I called MIT person, push this up slightly, it's going to create an instance down here, I called p 1. And when I would do that, I'm gonna initialize it. So I've got, right up here, an initializer, init for MIT person, takes in the family name and the first name. Notice what it does. Huh. It says, if I'm sitting here at MIT person, I'm going to go up and inherit from person its init function and call it. And what am I calling it on? I'm calling it on self, which is pointing to this object, so I've still got it, and then I'm then going to apply the base initialization. And that does exactly what you'd expect, which is just going to create a binding for family name down here. As well as some other things. So this is an example of inheritance. MIT person inherits the init method from person, can get access to by simply referring to it, and I refer to it right there. And it's take the person class, get its init and apply it to my instance plus those things. So I'm just using the same piece of code

    Notice the second thing it does. It says inside of self, I'm going to bind the local variable id name to the value of next id name in MIT person. Self is down here, id num, sorry, not id name. I'm going to bind that to the value that I find my going up to here, which is 0, and having done that, I simply increment that value. OK? So what has this done? It says I now have captured in the class, a local variable that I can keep track of. And when I use it, every time I generate an example, let me build another one. I make p 2 another MIT person. OK, I can do things like saying, what is the id number for each of these. First one is 0, second one is 1, which makes sense, right? I'm just incrementing a global variable. Now, things I want you to see about this. Now that I've got a beginning of a hierarchy, I have this notion of inheritance. I can ask a function inside one class to use a function from a class that it can reach by going up the chain. I just did it there. I can ask it to go get values of variables, right, so that looks good. What else do we have in person or MIT person? Well, we can get the id number, we just did. We have a thing to do with this string. Notice it's going to print out something a little different. In fact, there's a kind of funky form there. Which just says, if I want to print it out, I'm gonna create, what this says to do is, I'm gonna create an output template that has that structure to it, but where I see that percent s I'm going to substitute this value for the first one, that value for the second. So if I say, what is p 1? It says ok, MIT person Fred Smith. On the other hand, if I said, what is per, which is that thing I build earlier, it had a different string method, which is just print out person, those pieces.

    All right, one last piece to this and we're going to add to it. Suppose I want Fred to say something. Say something to Jane. OK, he said it. Where's the say method? OK, Fred is an instance of an MIT person. where's the say method? Well, there isn't one there, but again, that's where the hierarchy comes in. Fred is this object here, I'm sending it the message say. That turns into going up the chain to this object, which is the class object, and saying find a say method and apply it to that instance. Fudge-knuckle, it ain't here. Don't worry about it, because it says if I can't find one there, I'm going to go up the chain to this method, sorry to this class, and look for a method there. Which there was one, I have a say method. It's going to use that say method. Apply to it. Well, you might say, OK, what happens if it isn't there? Well, that's where, remember I defined person to be an instance of an object, it will go up the chain one last time to the base object in Python to see is there a method there or not. Probably isn't a say method for an object, so at that point it's going to raise an exception or throw an error. But now you again see this idea that the inheritance lets you capture methods.

    Now you might say, why not just put a say method inside of MIT person? Well, if you wanted it to do something different, that would be the right thing to do. But the whole notion here's that I'm capturing modularity, I've got base methods up in my base class. If I just want to use them I'm just going to inherit them by following that chain, if you like, basically up the track. OK, so we've got an MIT person, we can use that. Let's add a little bit more to our hierarchy here. I'm going to create, if I can do this right, a specialization of an MIT person, which is an undergraduate. A special kind of MIT person. All right, so if I go back up here, even though my thing is not going to let me do it, let's build an undergraduate. OK, there's the class definition for an undergrad. We're just starting to see some of the pieces, right, so in an undergraduate, where am I here, an undergraduate. OK, it's also got an initialization function. So if I call undergrad, I'm gonna make an undergrad here, again let me go back down here, line ug 2 it's making undergrad, Jane Doe. Now, what happens when I do the initialization here? Notice what goes on. It simply calls the person initialization method. All right, so I'm down here. I'm going to call the person initialization method, what did do? Sorry, the MIT person method, it calls the person method. Just walking up the chain, that's going to do exactly what I did with all the other ones, so I now have a family name and a first name. So I can, for example, say family name and get it back out. All right?

    And then, other things that I can do, well I can set what year the person's in, I can figure out what year they're in, there's this unfortunate overflow error if you've hung around too long, but that's not going to happen to you. And I've now got a say method here, so let's look what happens if I ask the undergraduate to say something. OK, it's not a realistic dialogue I know, but, what did this method do? I asked this object to do a say. And notice what it does. It simply passes it back up to MIT person. There's that inheritance again. It's saying, I'm going to have my base say method say something. I'm going to say it to a person, but all I'm going to do because undergraduates in my experience, at least, are always very polite, I'm going to put "Excuse me but" at the front of it. OK, what am I trying to show you here? I know the jokes are awful, but what am I trying to show you here? That I can simply pass up the chain to get it. In fact, what method does the final say here? What class does it come from? Person class, yes, thank you. It goes all the way up to person, right, because MIT person didn't have a say. So I can simply walk up the chain until I find the method I want to have.

    Now this is an example of shadowing. Not a great example, but it's a beginning example of shadowing, in that this same method for an undergraduate, shadows the base say method, it happens to call it, but it changes it. It puts "Excuse me but" at the front, before it goes on to do something. Now again, I could have decided here to actually copy what the original say method did, stitch all the other things together. But again, that loses my modularity. I'd really to only have to change it in one place. So by putting my say method up in person, I can add these nuances to it, and it lets me have something that has that variation. If I decide I want to change what say does, I only have to change it in one place. It is in the person class definition, and everything else will follow through for free.

    OK, so now I've got an undergrad, right? Let's look at a couple of variations of what happens here. So first of all, I can -- yes?

    PROFESSOR 2: Shadowing here is often sometimes called overriding.

    PROFESSOR: Yes, thank you, because I'm going to do a pure example of shadowing in a second, John right. Also called overriding. Part of the reason I like the phrase shadow is, if you think about it as looking at it from this direction, you see this version of init before you see the other ones, or you see that version of say, but it is overriding the base say example. OK, so I can say, what does p 1, sorry, yes, what does undergrad look like? And I said wait a minute, MIT person, not undergrad, is that right? Well, where's the str method? I didn't define one in undergrad, so it again tracks up the chain and finds the str method here, so it's OK undergrads are MIT people most the time, so it's perfectly fine.

    OK, now, I have built into this also these cmp methods. So I've got two examples. I've got undergrad, or ug. And then I've got poor old Frank Foobar back there, per person. So suppose I want to compare them? What do you think happens here? Compare sounds weird, right, I compare an undergraduate to a person. I don't know what that's doing, some kind of weird psychological thing, but what do you think happens in terms of the code here if I run this. I know it's a little hard because you got a lot of code to look at. Do I have a cmp method defined somewhere? Yeah. So, it's hard to know what it's going to do, but let's look at it. Hmm. Now sometimes I type things and I got errors I don't expect, this one I did expect. So what happened here? Well let's talk about what happens if I do that comparison I was doing, what was I doing? Ug greater than per? What unwinds into is, I'm going to send to ug, that instance, a cmp method. This is really going to become something like ug dot under under cmp under under applied to per. I think that's close.

    What does that do? It says starting in ug, I'm going to look for the first cmp method I could find, which is actually sitting here. I had a cmp method in MIT person. If you look at your code, what does it do? It looks up the id numbers to compare them. Well the, ug has an id number because it was created along this chamber. Remember per over here was just created as a person. It doesn't have an id number, so that's why it complaints. Ok, happens if I do that? Compare per to ug. How many people think I get an error? Wow. How many people think I'm going to get either true or false out of this? A few brave hands. Why? Can I ask you, please? Why do you think I'm going to get a, doesn't matter whether it's true or false, why am I going to have something work this time that didn't work last time?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Yeah, exactly. And in case you didn't hear it, thank you, great answer, sorry, terrible throw. In this case I'm using per, that's the first part, so it's not symmetric. It's gonna use per to do the look up. And as it was said there, per over here goes up and finds a cmp method here which it can apply. In that case, it simply looked at, remember, it took the tuples of first and last name which are both defined here, and did some comparison on that. So this is a way of again pointing out to you that the things are not always symmetric, and I have to be careful about where do I find the methods as I want to use them.

    Ok? All right. Let's add, I'm gonna do two more classes here. Let's add one more class, some people debate whether these are really people or not, but we're going to add a class called a professor. OK. Now what am I doing? I'm creating another version of class down here. Which again is an instance, or a subclass, sorry, not an instance, a subclass of an MIT person. I see that because I built it to be there. Again I've got an initialization that's going to call the person initialization, which we know is going to go up -- I keep saying that -- going to call the MIT person initialization, which is going to go up and call this one. So again I'm going to be able to find names. And I do a couple of other different things here. I'm gonna pass in a rank, full professor, associate professor, assistant professor, which I'm just going to bind locally. But I'm gonna add one other piece here, which is I'm gonna add a little dictionary on teaching. So when I create a professor, I'm gonna associate with it a dictionary that says, what have you been teaching?

    And then notice the methods I create. I've got a method here called add teaching, takes, obviously a pointer to the instance. A term, which will just be a string, and a subject. And let's look at what it does right here. OK. In fact the call I'm going to make, I'm not certain I'm going to be able to get away with it, my machine is still wonderfully broken, all right, it is, let me just show you what the calls would look like. As you can see here I'm not going to be able to do them. But I'm going to add teaching, as a method call with this with a string for term, and a subject number. What is this going to do? Yeah, I know I'm just worried if I restart Python, I may not be able to pull the thing back in, so I'm going to try and wing it, John, and see if I can make it happen.

    Right, what does that teaching do? It's got one of those try except methods. So what does it say it's going to do? It's going to go into the dictionary associated with teaching, under the value of term, and get out a list. And it's going to append to the end of the list the new subject. So it's going to be stored in there, is then going to be term, and a list of what I taught, in case I teach more than one thing each term. It's going to do that, but notice it's a try. If in fact there is no term currently in the dictionary, started out empty, it's going to throw an error, sorry, not throw an error, it's going to raise an exception. Which is a key error, in which case notice what I'm going to do, I'm not going to treat it as an error. I'm simply going to say, in that case, just start off with an empty, with an initial list with just that subject in and put it in the dictionary. As I add more things in, I'll just keep adding things to this dictionary under that term. And if I want to find out what I'm doing, well I can use get teaching, which says given the term, find the thing in the dictionary under that term and return it. If I get an error, I'm going to raise it, which says there is nothing for that term, and in that case I guess I'm just going to return none.

    OK? And then the other two pieces we're going to have here, and we want to look at a little more carefully, I just wanted to show you that example, is a professor can lecture, and a professor can say something. Look at the say method, because this now add one more nuance to what we want to do here. And I think in interest of making this go, let me actually, since I'm not going to get my machine to do this right, let me create a couple of professors. If I look at what that is, it's an MIT person because I didn't have any separate string thing there, and we will create a more important professor. What rank do you want, John? Do you want to stay full?

    PROFESSOR 2: Undergraduate.

    PROFESSOR: Undergraduate, right, a lot more fun I agree. Sorry about that, and we can again just see what that looks like. And that of course, we'll print out, he's also an MIT person. But now here's what I want to do. I want to say something to my good colleague Professor Guttag. Actually I'm going to start a separate -- I'm going to say something to a smart undergraduate. So if I say, remember we have ug defined as an undergraduate, let me do something a little different here. Well let, me do it that way. It says, I don't understand why you say you were enjoying 6.00. Not a good thing to say, right, but if I say to my good colleague Professor Guttag. I have to spell say right, I know, I need help with this, what can I say? We flatter each other all the time. It's part of what makes us feel good about ourselves. Why is the sky blue? I enjoyed your paper, but why is the sky blue?

    OK, terrible examples, but what's going on here? One more piece that I want to add. Here's my say method for professor, and now I'm actually taking advantage of to whom I am saying something. Notice again, what does it do? There's the self argument, that's just pointing to the instance of me. I'm passing in another argument, going to call it to who, in one case it was ug, in one case it was Guttag. And then the thing I want to say, ah, look what it does, it says, check the type. And the type is going to take that instance, I had an instance, for example, of a professor down here, and it's going to pick up what type of object it is. So if the type of the person I'm speaking to is undergrad, let's pause for second. Remember I started away back saying we're building abstract data types. Well, here's a great example of how I'm using exactly that, right? I've got int, I've got float, I now have ug, it's a type. So it's says if the object to whom I'm speaking is an undergrad, then use the same method from person where I'm going to put this on the front. On the other hand, if the object to whom I'm speaking is a professor, then I'm going to tag this on the front and use the underlying say method. On the other hand, if I'm speaking to somebody else, I'm just going to go lecture. All right, and when a professor lectures, they just put it's obvious on the end of things, as you may have noticed.

    What's the point I want you to see here? I'm now using the instances to help me to find what the code should do. I'm looking at the type. If the type is this, do that. If the type is this, do something different, ok? And I can now sort of build those pieces up. OK, I said one more class. Notice what we're doing. I know they're silly examples, but, sorry, they are cleverly designed examples to highlight key points. What I'm trying to do is show you how we have methods inherit methods, how have message shadow methods, how we have methods override methods, how we can use instances as types to define what the method should do.

    Let me show you one last class, because I'm gonna have one more piece that we want to use. And the last class is, sort of, once you've got a set of professors, you can have an aggregate of them. And I don't know, if a group of geese are gaggle, I don't know what a set of professors are, John. Flamers? I, you know, we've got to figure out what the right collective noun here is. We're going to call them a faculty for lack of a better term, right? Now the reason I want to show you this example is, this class, notice, it only is going to inherit from object. It actually makes sense. This is going to be a collection of things, but it's not a subclass of a particular kind of person. And what I want the faculty to do, is to be able to gather together a set of faculty. So if I go down here, grab this for second, and pull it down so you can see it. It looks like I'm not going to be able to run this because my machine is broken, but basically I'm gonna define a set of professors, and then I'm gonna create a new class called faculty. There's the definition of it. It's got an init. You can kind of see what it does. It's going to set up an internal variable called names, which is initially an empty list, internal variable called ids, which is empty, an internal variable called members, which is empty, and another special variable called place, which we're going to come back to in a second, initially bound to none.

    OK, I've got a method called add which I'm going to use down here to add professors to the course 6 faculty. Here's what I want to add to do. First of all, notice I'm going to check the type. If this is not a professor, I'm gonna raise an error, a type error, it's the wrong type of object to pass in. The second thing I'm gonna do is say, if that's okay, then let me go off and get the id number. Now remember, that's right up here, so I'm asking the instance of the professor to go up and get the id number. And I want to make sure I only have one instance of each professor in my faculty, so if the id number is in the list of ids already, I'm going to raise an error, sorry, raise an exception as well, saying I've got a duplicate id. OK? And the reason that's going to come up is, notice what I do now. Inside of the instant self, I take the variable names and I add to it the family name of the person I just added. OK, notice the form. I'm using the method, there's the parens to get the family name of the person. I'm just adding it to the list. I've got the id number, I've added the ids, and I add the object itself into members. So as I do this, what am I doing? I'm creating a list, actually several lists: a list of ids, a list of the actual instances, and a list of the family names. And as a cost I want to add, that's why I can check and see, is this in here already or not?

    Now, the last reason I want to do this is, I want to be able to support things like that. This is now different, right, this instance is a collection. I want to be able to do things like, for all the things in that collection, do something, like print out the family names. And to do that, I need two special forms: iter and next. OK, now let me see if I can say this cleanly. Whenever I use a for, in structure, even if it was on just a normal list you built, what Python is doing is returning an, what is called an iterator. Which is something that we talked earlier. It's keeping track of where are you in the list, and how do I get to the next thing in the list?

    I'm going to do the same thing here, and I'm going to create it for this particular structure. So this little thing iter, when I call a for something in, one of these instances, it calls iter, and notice what it does. It initializes place to 0. That was that variable I had up there. That's basically saying I'm at the beginning of the list. It's a pointer to the beginning of the list, and it returns self. Just gives me back a pointer to the instance. That now allows me at each step in that loop to call next. And what does next do? Next says, check to see if that value is too long, if it's longer than, for example, the list of names, raise an exception called stop iteration, which the for loop will use to say OK, I'm done. I'm going to break out of the for loop. Otherwise, what am I going to do? I'll increment place by 1, that's going to move me to the next place in the list, and then in this case I'll just return the instance itself, right? Members is a list of instances, place I've incremented by 1, I take 1 off of it, I get to it. So iter and next work together. Iter creates this method, that's going to give you a pointer to the place in the structure, and then next literally walks along the structure giving you the next element and returning elements in turn so you can do something with it.

    Right, so now what that says is, I can have classes that just have local variables. I can have classes that get methods from other variables, and I can also have classes that are collections. And I've supported that by adding in this last piece. OK once you have all of that, in principle we could start doing some fun things. So let's see what happens if we try and make all of this go. And let me, since I'm not going to be able to run it, let me simply do it this way. If I have my undergraduate, ug. I can -- sorry, let's not do it that way -- I can have undergraduate say things like -- all right, what did I just do wrong here? Do I not have undergrad defined? I do. Oh, I didn't have Grimson, sorry, it's me, isn't it? Thank you. The undergraduate very politely asks why he didn't understand, you can have the professor respond. Again, it simply puts a different thing into there. On the other hand, if Professor Guttag asks me something about understanding, I say I really like this paper on, you do not understand, it's a deep paper on programming languages 5, I think, John, isn't it? What else can you do with this thing, right? You can have an undergraduate talk to an undergraduate, in which case they're still polite. Or you could have -- sorry, let me do that the other way -- you could also have an undergraduate simply talk to a normal person. All right, but the good news is you know eventually you get it done, and when you're really done you can have the undergraduate be really happy about this, and so she sings to herself.

    OK it's a little silly, but notice what we've just illustrated. And this is where I want to pull it together. With a simple set of classes, and the following abilities, an ability to inherit methods from subclasses, sorry from superclasses, that is having this hierarchy of things. I can create a fairly complex kind of interaction. I can take advantage of the types of the objects to help me decide what to do. And if you think about that, I know it sounds very straightforward, but you would do exactly that if you were writing earlier code to deal with some numerical problem. All right, if the thing is an integer, do this, if it's a float, do that, if it's a string, do something else. I'm now giving you exactly the same ability, but the types now can be things that you could create. And what I've also got is now the ability to inherit those methods as they go up the chain. So another way of saying it is, things that you want to come away from here, are, in terms of these classes. We now have this idea of encapsulation. I'm gathering together data that naturally belongs as a unit, and I'm gathering together with it methods that apply to that unit. Just like we would have done with float or int. Ideally, we data hide, we don't happen to do it here, which is too bad.

    Basically we've got the idea of encapsulation. The second thing we've got is this idea of inheritance. Inheritance both meaning I can inherit attributes or field values. I can inherit methods by moving up the chain. I can also the shadow or override methods, so that I can specialise. And I do all of that with this nice hierarchy of classes. So what hopefully you've seen, between these two lectures, and we're going to come back to it in some subsequent lectures, is that this is now a different way of just structuring a computational system. Now, you'll also get arguments, polite arguments from faculty members or other experts about which is a better way of doing it. So I'll give you my bias, Professor Guttag will give you his bias next time around. My view, object-oriented system are great when you're trying to model systems that consist of a large number of units that interact in very specific ways. So, modeling a system of people's a great idea. Modeling a system of molecules is probably a great idea. Modeling a system where it is natural to associate things together and where the number of interactions between them is very controlled. These systems work really well. And we'll see some examples of that next week. Thanks.
  • 27
    Assignment 9: Classes and methods
    6 pages
  • 28
    MIT Lecture 17: Computational Models: Random Walk Simulation
    49:23
    Lecture 17: Computational models: random walk simulationInstructors: Prof. Eric Grimson, Prof. John Guttag View the complete course at: http://ocw.mit.edu/6-00F08License: Creative Commons BY-NC-SAMore information at http://ocw.mit.edu/termsMore courses at http://ocw.mit.edu

    LECTURE TRANSCRIPT

    PROFESSOR: OK, we're now, kind of on the home stretch, and we're entering the part of the course, that's actually my favorite part of the course. I can't promise it will be your favorite part of the course, but I hope so, at least for many of you. Throughout the term, we've been talking about ways to solve problems using computation. And one of the key lessons that I hope you're beginning to absorb is that we might use a completely different way to solve a problem with the computer than we would have used if we didn't have a computer handy. In particular, we might often use brute force, which you've never use with a pencil and paper, we might not do the mental gyrations required to try and formulate a closed form solution, but just guess a bunch of answers using successive approximation until we got there. A number of different techniques. And that's really what we're going to be doing for the rest of the term now. Except we won't be talking about algorithms per se, or not very much. Instead we'll be talking about more general techniques for using computers to solve problems that are actually hard. They don't just look hard, they in many cases really are hard. The plan of the next set of lectures is, I want to start with a simple example to give you a flavor of some of these issues. In the course of going through that example, I'll illustrate some both thinking tools and some software tools that can be used for tackling the example. Then abstract from the example to try and put in a more general framework. And then, dive back down and use that framework to tackle a series of other interesting problems.

    Some things I would like you do think about learning along the way, is moving from an informal problem description to a more formal problem statement. So that's, in part, what we did with the optimization problems, right? We looked at an informal description about optimizing your way through the MIT curriculum, and then could formulate it using sigmas and other bizarre notation to try and formalize what we were really trying to do. And we did that as a preamble to writing any code. First understand the problem, formally, and then move on to the code. And we won't always be totally formal. I often like to use the word rigorous instead of formal. Implying that it won't look like math, per se, but it'll be precise and relatively unambiguous. So that will be one thing that I want you to think about as we go through these problems.

    Another thing is inventing computational models. Almost every meaningful program we write is in some sense modeling the actual world. For writing a program to figure out how to keep a bridge from falling down, we're modeling the physics of bridges and wind and things like that. If we're writing a program to try and help us decide what stocks to buy or sell, we're trying to model the stock market in some sense. If we're trying to figure out who's going to win the Super Bowl, we're modeling football teams. But almost every problem, you build a piece of software, and you hope that it has some ability to mimic the actual situation you care about. And it's not the program we care about, per se, it's the world, and the program is merely a mechanism to help us understand the world better. And so we're going to ask the question, have we built a good model, in the computation, that gives us insight into the world?

    As we do this, we'll see that we'll be dealing with and exploiting randomness. Depending upon your outlook of life, one of the sad things in the world or one of the happy things in the world, is that it's unpredictable. And things happen either at random or seemingly at random. It may be that if we had a deep enough understanding at the level of single atoms of the way the world works, we could model the weather and discover that, in fact, the weather patterns are not random but entirely predictable. Since we can't do that, we have to assume that they really are random, and we build models of the weather that assume a certain amount of randomness in it. Maybe if we understood the stock market well enough, we could have predicted the collapse in October. But since nobody does understand it well enough, when people model the stock market they assume that there's a certain amount of randomness, certain amount of stochastic things.

    So, as far as we can observe the world, almost every interesting part of the world has randomness in it. And so when we build models, we will have to build models that are, if we want to be formal, we'll talk about a stochastic, which is just a fancy way of saying they incorporate randomness. So we haven't yet done that this semester, but almost everything we do from here on in, we will deal with a certain amount of randomness. What else are we going to be looking at? We'll be looking at the notion of making sense of data. As we look at, again, modeling the world, what we discover, there's a lot of data out there. If you work at Walmart and you're trying to decide what to stock on the shelves, you're building a model of what the customers might buy under certain circumstances, and that model is going to take account the entire history of what customer's have bought in the past. And that's, if you're Walmart, a lot of data. And so given that you have a lot of data about what's happened in the past, how can we interpret that, for example, to get insight into the future? And we do a lot of that. And so we will, for example, look at how can we draw pictures that help us visualize what's going on with the data, rather than trying to read say, a million numbers and inferring something from them. This is a big part of what people do with computers, is try and figure out how to understand large amounts of data.

    And then finally, as we go through this last third of the course, I want to spend time in evaluating the quality of answers. It's easy to write a program that spits out a number, or string, or anything else. What's hard is to convince yourself that what it's spitting out is actually telling you the truth. And so we're going to look at the question about, you've written a program to do something relatively complicated, you get an answer out. You wouldn't have written the program if you knew what the answer was in advance, right? This is not like a high school physics experiment where you know what the answer should be. Here you went to the trouble of writing the program because you didn't know what the answer was. Now the program gives you an answer, should you believe it, or shouldn't you believe it? There are people who say, well, that's what the computer said, it must be true. By now you all have enough experience with programs that lie to know that's not the case. And so we'll be talking about how do you go about looking at the results, and deciding whether to believe them or not.

    This is a lot of stuff, and we'll be coming back to this. I'll be skipping around from topic this topic. It may seem a little bit random, but there's actually a method to my madness. Part of it is, I believe that repeated exposure to some of these things is a good way to learn it. And so I'll be revisiting the same topic in more and more depth as we go, rather than taking a topic and exhausting it. So if you think about searches, this is more of a breadth first search than a depth first search.

    All right. Think about these things as we now go through the next set of lectures. So, on for the first example. Consider the following situation: a seriously drunken university student, and I emphasize university as opposed to institute here, is standing in the middle of a field. Every second, he takes a step in some direction or another, but, you know, just sort of pretty random, because he's really out of it. But the field is constrained that for the moment we'll assume that, well, we'll come back to that. Now I'm going to ask you just a question. So you've got this student who, every second or so, takes a step in some direction or another. If the student did this for 500 seconds or 1000 seconds, how far do you expect the student would be from where he started? And I say he because most of the drunks are, of course, males. Anybody want to, what do you think?

    STUDENT: Back where he started.

    PROFESSOR: So we have a thing that, on average, and of course since there's randomness it won't be the same every time. Pretty much where he started. Let me ask a question. So if you believe that, you believe he'd be probably the same distance in 1000 seconds as in 500 seconds. Anyone want to posit the counterposition? That in fact, the longer the clock runs, the further the student will be from where he started? Nobody? All right, what do you think?

    STUDENT: [INAUDIBLE]

    PROFESSOR: All right, that's a good answer. So the answer there was, well, if you asked for your best guess of where the student was, the best guess is where the student started. On the other hand, if you ask how far was the student from where he started, the best guess wouldn't be zero. That's a great answer, because it addresses the fact that you have to be very careful what question you're asking. And there are subtle differences between those two questions, and they might have very different answers. So that's part of that first step, going from an informal description, to trying to formalize it or be more rigorous about what is exactly the problem you're trying to solve.

    All right, we'll take a vote. And everyone has to vote here, because it's not a democracy where you're allowed to not vote. And the question I'm asking is, is the expected difference, does the expected distance from the origin grow over time, or remain constant, roughly? Who thinks it grows over time? Who thinks it remains constant? Well, the constants have it. But just because that's the way most people vote, it doesn't make it true. Now let's find out what the actual truth is.

    All right. So let's start by sketching a simple model of the situation. And, one of the things I want to stress as we do these things, in developing anything of this nature, start simple. So start with some simple approximation to the real problem. Check that out, and then, if it turns out not to be a good enough model of the world, add some complications, but don't start with the complicated model. Always start with the simple model. So I'm going to start with the simple model. And I'm going to assume, as we've seen before, that we have Cartesian coordinates and that the player is, or the drunk is, standing on a field that has been cut to resemble a piece of graph paper. They got the groundskeeper from Fenway Park or something, there, and it looks like a beautiful piece of paper. Furthermore, I'm going to assume for simplicity, that the student can only go in one of four directions: north, south, east, or west. OK, we could certainly generalize that, and we will, to something more complicated, but for now we'll keep it simple. And we'll try and go to the board and draw what might happen.

    So the student starts here and takes a step in one direction or another. So as the mathematicians say, without loss of generality, let's just assume that the first step is here. Well what we know for sure, is after one step the student is further from the origin than at the start. But, that doesn't tell us a lot. What happens after the second step? Well, the student could come back to the origin and be closer, that's one possibility, the student could go up here, in which case the student is a little further, the student could go down here, in which case the student is a little further, or the student could go over here, in which case the student is twice as far. So we see for the second step, three times out of four you get further away. What happens in the third step? So let's look at this one. Well, the student could come here, which is closer, could come here, which is closer, could go there, which is further, or could go there, which is further. So the third step, with equal probability, the student is further or closer if the student is here. And we could continue. So as you can see, it gets pretty complicated, as you project how far out it could always get. All right, and this is symmetric to this, but this is yet a different case. So, this sort of says, OK, I'm going to get tired of drawing things on the board.

    So, being who I am, I say, let's write a program to do this. And in fact, what I'm going to write a program to do, is simulate what is called a random walk. And these will be two themes we're going to spend a lot of time on. Simulation, where we try and build the model that pretends it's the real world and simulates what goes on, and a random walk. Now, I'm giving you the classic story about a random walk which you can visualize, at least I hope, but as we'll see, random walks are very general, and are used to address a lot of real problems. So we'll write this program, and I want to start by thinking about designing the structure of the solution. Because one of the things I'm trying to do in this next set of lectures is bring together a lot of the things we have talked about over the course of the semester, about how we go about designing and building programs. Trying to give you a case study, if you will.

    So let's begin in line with what Professor Grimson talked about, about thinking about what might be the appropriate data abstractions. Well, so, I think it would be good to have a location, since after all the whole problem talks about where the drunk is. I decided I also wanted to introduce an abstraction called compass point to capture the notion that the student is going north, south, east, or west. And later, maybe I'll decide that needs to be more complicated and that the student can go north by northwest, or maybe all sorts of things. But that it would probably pay to separate the notion of direction, which is what this is, from location. Maybe I should have called this direction instead of compass point, but I didn't. But, I thought about the problem and said, well, these things really are separate locations. Where you are and where you might head from there are not the same, so let's separate them. Then I said, well of course, there is this notion that the person is in the field, so maybe I want to have field as a separate thing. So think of that as the whole Cartesian plain, as opposed to a point in the plane, which is what the location is. And finally, I better have a drunk, because after all this problem is all about drunks.

    All right, so we're now going to look at some code. I made this code as simple as I could to illustrate the points I wanted to illustrate. This means I left out a lot of things that ought to be included in a good program. So I want to just warn you. So for example, I've already told you that when I build data abstractions, I always put in an underbar underbar str function so that I can print them when I'm debugging. Well, you won't see that in this code, because I felt it just cluttered the code up to make the points. You'll see less defensive programming than I would normally use. But again I wanted to sort of pare things down to the essence, since the essence is, all by itself, probably confusing enough.

    All right. So let's look at it. You have this on your double-sided handout. This is on side 1one So at the top, you'll see that I'm importing three things. Well, math you've heard about. And I'm importing that because I'm going to need a square root. Random, you haven't heard about. This is a package that lets me choose things at random, and I'll show you some of the ways we can use it, but it actually provides something that technically is pseudo-random. Which means that as far as we can tell, it's behaving randomly, but since the computer is in itself a deterministic machine, it's not really random. But it's so close to random that we might as well pretend it is, you can't tell that it isn't. So we'll import random. That will let us make random guesses of things like, give me a random number between and 1. And it will give you a random number. And finally something called Pylab. Anyone here use Matlab? All right, well then, you'll find Pylab kind of comforting. Pylab brings into Python a lot of the features of Matlab And if you haven't used Matlab, don't worry, because we'll explain everything. For the purposes of this set of lectures, the next few lectures, at least, the only thing we're going to get out of this is a bunch of tools for drawing pretty graphs. And we won't get to those today, probably, so don't worry about it. But it's a nice package, which you'll be glad to learn.

    OK, now let's move on to the code, which has got things that should look familiar to you. And I know there used to be a laser -- ah, here's the laser pointer. So we'll see at the top location, it is a class. By now you've gotten as familiar as you want to be in many senses, with classes. It's got an underbar underbar init that gives me, essentially, a point with an x-coordinate and a y-coordinate. It's got get coords, the third function, or method, and what you can see what that does is it's returning a tuple of two values. I could have had it get x and get y, but it turned out, in fact, my first iteration of this it did have a get x and get y, and when I looked at the code that was using it, I realized whenever I got one, I wanted both, and it just seemed kind of silly, so I did something that made the using code a little bit better. It's got get distance, the last method. You saw this in Professor Grimson's lecture, where he used the Pythagorean theorem to basically compute the distance on a hypotenuse, to tell you how far a point was from the origin. That's why I wanted math. And then the one thing it has that it's unlike the example you saw from Professor Grimson, was up here, this move. And this basically takes a point, a location, and an x- and a y- coordinate, and returns another location, in which I've incremented the x and the y, perhaps incrementing by 0. And so now you can see how I'm going to be able to mimic 1 step or any number of steps by using this move. Any questions about this?

    Great, all right. The next one you'll see is compass point. Remember, this was the abstraction that was going to let me conveniently deal with directions. The first thing you'll see is, there's a global variable in the sense that it's external to any of the methods, and you'll note it's not self dot possibles, but just possibles. Because I don't want a new copy of this for every instance. And this tells me the possible directions, which I've abbreviated as n, s, e, and w. And I leave it to you to figure out what those abbreviations stand for. Then I've got init, which takes a point and it first checks to see if the point is in self dot possibles. Should I have bothered saying self? Do I need to write self there? This is a test of classes. It'll work. What do you think? Who thinks I need to write self, and who thinks I don't? Who thinks I need to write self, raise your hand? Who thinks I don't need to write self? The don't needs have it. So in an act of intense bravery, since I have not run it without this, we'll see what happens when it comes time to run the program.

    So if point is in possibles, it will take self dot p t will be assigned p t. Otherwise, I'll raise a value error. And this is a little piece of programming, and what I typically do at a minimum, when I raise these exceptions, is make sure that it says where it's coming from. So this says it's gonna raise this value error in the method compass point dot underbar underbar init, underbar underbar. This is so when I run the program, and I see a message, I don't have to scratch my head figuring out where did things go wrong? So just a convention I follow a lot. And then I've got move here. And I'll move self some distance, and you can see what I'm doing. If self is north, I'm going to return in distance. So I'm now getting a tuple, which will basically be used to say, all right, we're going to implement x by and y by distance, which is what you think of about moving due north. And if it's south, we'll increment x by 0, and increment y by minus distance, heading down the graph. And similarly for east and west. And in the sad event I call this with something that's not north, south, east, or west, again I'll raise an exception. OK, little bit at a time. So I hope that nothing here looks complicated. In fact, it should all look kind of boring.

    Now we'll get to field, which should look a little less boring. So now I've got a field. And, well, before I get to field, I want to go back to something about the first 2 abstractions: compass point and location. Whenever we design one of these abstractions, we're making some things possible and some things impossible. We're making decisions, and so what decisions are encapsulated in these first two classes? Well, one decision is that it's a 2-dimensional space. I basically said we've got x and y and that's all we've got. This student cannot fly. The student cannot dig a hole and go into the ground, we're only moving in 2 dimensions, so that's fixed. The other decision I made is that this student can only move in 1 of 4 directions. So, couple of good things here. One, I've simplified the world, which will make it easier for us to see what's going on. But two, I know where I've made those decisions, and later if I want to go back and say, you know it'd be fun if the student could fly, let's say we're a drunken pigeon instead of a drunken student. Or it would be a good idea to realize that, in fact, they might head off at any weird angle. I know what code I have to change. It's encapsulated in a single place. So that's why possibles is part of compass point, rather than scattered throughout the program. So this is nice way to think about it.

    All right, let's move on and look at field. So field, it's got an init, it takes a self, a drunk, and a location, and puts the drunk in the field at that location. It can move. So let's look at what happens when we try and move self in some direction, in some distance. We say the old location is the current location, and then x c and y c, think of that as x-change and y-change, get c p dot move of dist. Where is it gonna find c p dot move? It's going to use compass point dot move, because as we'll see, c p is an object of type, of class, compass point. So this is a normal and typical way we structure things with classes. The move of one class is defined using the move of another class. This gives us the modularity we so prize. And then I'll say self dot location is old loc dot move of x c and x c. Now this is oldloc dot move is going to get the move from class location. And you'll remember, that was the thing that just added appropriate values to x and y. So I'll use compass point to get those values, and then oldloc to get the new location. Then I'll be able to get the loc and I'll be able to get the drunk.

    So, let me ask the same question about field I've asked about the others? What interesting decisions, if any, have I embodied in the way I've designed and implemented this abstraction? There's at least one pretty interesting decision here. Just interesting to me because my first version was far more complicated. Well, how many drunks can I have in a field at a time? We have an answer which will not be recorded, because it was merely raising a finger, but it happened to be the correct number of fingers: one. And it was this finger that got pointed at me. We've embodied the fact that you can have one drunk, exactly one drunk in the field. This is a solitary alcoholic. Later we might say, well, it would be fun to put a whole bunch of drunken students in and watch what happens when they bump into each other. And we'll actually later give you a problem set, not with students, but with other things where there might be more than one in a field. But here, I've made that decision. But again, I know where I've made it, and if later I go back and say let's put a bunch of them in, I don't have to change compass point, I don't have to change location, I only have to change field. And we'll also see, I don't have to change drunk. So again, it's very nice that the class structure, the modularity let's us have decisions in exactly one place in our code. Usually important.

    All right, now let's look at drunk. So the drunk has a name. And, like everything else, a move operation. This is the most complicated and interesting move. Because here is where I'm encapsulating the decisions about what the drunk actually does. That the drunk, for example, doesn't head north and just keep on going. So, let's look at what happens here. It's got three parameters: self, the field the drunk is moving in, and time. How long the drunk is going to move. And you may notice something that you haven't seen before, at least some of you, time equals one is sitting up there. There. Python allows us to have what are called default values for parameters. What this says is that if I call this method without that last argument, rather than getting an error message saying it expects three arguments it only got two, it chooses the default value, in this case one for the third argument. This is actually a pretty useful programming paradigm, because there's often a very sensible default. And it can simplify things. It's not an intrinsically interesting or important part of what I'm showing you here. The reason I'm showing it is, you'll be getting a problem set in which default values are there because we're using some other things. And as you bring in libraries and modules from elsewhere, you'll find that there are a lot of these things. And in fact a lot of the functions you've already been using for the built-in types happen to have default values. For example, when you look at things in the init part of a range, it's actually choosing, say, as a start. But don't worry about it. This just says if you don't pass it a time, use 1.

    And then what it does, is it says, if field dot get drunk is not equal to self, raise an exception. OK, you've asked me to move a drunk, it doesn't happen to be in the field. That's not very interesting. But now we come to the interesting part. For i in range time, here by the way range does have a default value of 0, and since I didn't supply it, it used it. Point equals compass point, and here's the most interesting thing, random dot choice of compass point dot possibles. Random dot choice is a function in the random module which we've imported and it takes as an argument a sequence, a list of some sort, a container, and it picks a random element out of that. So here we're passing it in, four possible values, and it's just going to pick one of them at random. So you can see by my having encapsulated the possibles array in compass point, I don't have to worry in drunk about how many possible directions this person could head off in. And then it calls field dot move, with the resultant point, and in this case one. All right?

    We've now finished all of this sort of groundwork that we need to go and actually build the simulation that will model our problem. So let's look at how we use all of this. I've now got a function perform trial, which will take a time in a field. So I'll get a starting point of f dot getloc, wherever the drunk happens to be at this point in time in the field. And then for t in range 1 to time plus 1, I'm going to call field dot get drunk, that will give me the drunk that's in the field, and then I'll get the move function for that drunk. So what you see here is I can actually, in this dot notation, take advantage of the fact, where are we, we're down here, I've got to get far enough away that I can see it -- Where was I?

    STUDENT: You were in the right place right there, down slightly.

    PROFESSOR: -- down slightly, here, right. I've called the function, which has returned an object, f dot get drunk returns an object of class drunk. And then I can select the method associated with that object, the move method, and move the drunk. This gets back to a point that we've emphasized before, that these objects are first class citizens in Python. You can use functions and methods to generate them, and use them just as if you'd typed them. So that will get me that, and then I'll call the move of that drunk with the field. Then I'll get the location, the new location, and then I'll say, distance equals new loc dot get distance of start, how far is the new location from wherever the starting location was? I'll append it to a list, and then I'll return it. So now I have a list, and in the list I have how far away the drunk is after each time step. I'm just collecting the list of distances. So I don't have a list of locations, I don't know where, I can't plot the trajectory, but I can plot the distances.

    OK, now, let's put it all together. So I'll say the drunk is equal to, is drunk of Homer Simpson. I had thought about using the name of someone on campus and decided that, since it was going to be taped for OpenCourseWare, I didn't want to go there. Sometimes I lack courage. Then for i in range 3, and all this says is I'm going to test, I'm going to run the simulation three different times, f is equal to field of drunk and location 0, 0, starting in the middle of the field. Distances equal perform trial 500, 500 steps on that field, and the rest of it is magic you don't need to understand because we'll come back to next lecture, which is, how do I put all this in a pretty picture? So ignore all of that for now, and that's just the Pylab stuff for plotting the pictures. All right, we're there, let's run it. Huh. Remember the change we made? Guess what? Well, we know how to fix that. And we'll come back to it and ask what's really going on here. What does self refer to here? It refers to the class compass point, rather than an instance of the class. Again, driving home the point that classes are objects just like everything else.

    STUDENT: [INAUDIBLE]

    PROFESSOR: Oh, sorry. Thank you. It did it have in here, right. All right, now let's run it, see what we get. We get a picture. Well, so we see that, at least for these three tests, the majority of the public was wrong. It seems that the longer we run it, the further from the origin Homer seems to be getting. Well, let's try it again. Maybe we just got unlucky three times. You can see from the divergence here, that, of course, a lot of things go on. I hope you can also see the advantage of being able to plot it rather than having to look at those arrays. Well, we get different answers, but you know what? The trend still seems pretty clear. It does seem over time, that we wander further away. I know, further away. So, looks at least for the moment, that perhaps we were wrong.

    Well, this is rather silly for me to keep running it over and over again. So clearly what I ought to do is, do it in a more organized way. So now if we go back, and see the, kind of the right way to do it, I'm going to get rid of this stuff here, which was used just to stop the previous computation. I'm going to write a function called perform sim. And what that does, is it takes the time and the number of trials that we wanted to do. It takes distlist equals the empty list in this case, to start with, and then it is basically going to call perform trial, the function we already looked at over and over again. And get a whole bunch of lists back, a list each time it calls it. And then, compute an average. Well, not yet, so it says, it says d equals drunk, field start in zero, distances equals perform trial, append the new list, and what it does, is it returns a list of lists. Where each list is what we had for the previous trials. Make sense? And then I'm going to have 1 more function call, answer a question. Which takes the maximum time and the number of trials, and it's going to do some statistics on all of the lists. We'll come back to this a week from today. If you are, have absolutely nothing to do for the next week, I'll give you a hint. I have salted a bug here in this latest code, and you might have some fun looking at what that bug is.
  • 29
    Assignment 10: Object-oriented programming; graphical user interface for word game
    6 pages
  • 30
    MIT Lecture 18: Presenting Simulation Results, Pylab, Plotting
    52:54
    Lecture 18: Presenting simulation results, Pylab, plottingInstructors: Prof. Eric Grimson, Prof. John Guttag View the complete course at: http://ocw.mit.edu/6-00F08License: Creative Commons BY-NC-SAMore information at http://ocw.mit.edu/termsMore courses at http://ocw.mit.edu

    PROFESSOR: You'll recall, at least some of you will recall, that last time we ended up looking at a simulation of a drunken university student wandering randomly in a field. I'm going to return to that. Before I get to the interesting part, I do want to call your attention to something that some people seemed a little bit confused by last time. So you remember that we had this thing called perform trial. And the way we basically tested the drunk, what would happen, is we would call this, passing in a time, the number of steps, the amount of time that the drunk was wondering, and the field. And there was just 1 line, that I blazed past, probably too quickly. So what that line is saying, is we, from the field, we're going to call the method get drunk, which is going to return us a drunk. And then we're going to call the move method for that drunk, passing the field as an argument. Now the question is, what would have happened if I had left off these parentheses? There. A volunteer? What would, we will do the experiment in a minute. And by the way, I want to emphasize the importance of experimentation. I was kind of surprised, for example, to get email from people saying, well, what was the correct answer to question 3 on the quiz? And my sort of response was, why don't you type it and find out?

    When in doubt, run the experiment. So, what's going to happen, what would have happened, had I failed to type that pair of parentheses? Anybody? Well, what is, get drunk? It is a method. And remember, in Python, methods, like classes, are themselves objects. So get drunk would have a returned an object which was a method, and then, well, let's try it. And the error message will tell us everything we need to know. Function has no attribute move. Sure enough, the method does not have an attribute move. The instance of the class has the attribute. And so what those parentheses did, was tell me to invoke the method get drunk, so now instead of that being the method itself, it's the result returned by the method. Which is an instance of class drunk, and that instance has an attribute move, which is itself a method. This is a very common programming paradigm, and it's important to sort of lock into your heads, the distinction between the method and an invocation of the method. Because you can write either, and sometimes you won't get an error message, you'll just get the wrong answer. And that's kind of bad. That make sense to everybody? Does it not make sense to anybody, maybe is the question I should ask?

    All right, we'll go on, and we'll see more examples of this kind of thing. Now last week, I ran this several times to see what would happen, and we saw that in fact contrary to everyone's, most everyone's, expectation, the longer we ran the simulation, the further the drunk was from the starting point. And we saw that by plotting how far the drunk was after each time unit. We ran it several times, we got different answers. And at that point we should ask ourselves, is that really the way to go about answering the original question? Which was, how far should we expect the drunk to be? The answer is no. I don't want to sit there at the keyboard typing in 400 examples and then, in my head, trying to figure out what's quote, typical, unquote. Instead, I need to organize my simulation so that it runs a number of trials for me and summarizes the results. All the simulations we're going to look at it, in fact almost all the simulations anyone will ever write, sort of, have the same kind of structure. We start with an inner loop that simulates one trial.

    That's what we have here, right, we have happen to have a function. So when I say inner loop, what I mean is, not that I'll write a program a bunch of nested loops, but that I'll write a program with some function calls, and down at sort of the bottom of the stack will be perform trial. Which stimulates one trial of some number of seconds, in this case. And then I'll quote, enclose, unquote the inner loop in another loop that conducts an appropriate number of trials. Now a little bit later in the term, we'll get to the question of, how do we know what an appropriate number is? For today, we'll just say, a lot. And we'll talk about that a little bit more. And then finally, what we want to do, is calculate and present some relevant statistics about the trials. And we'll talk about what's relevant. I want to emphasize today the presentation of them. Last time we looked at a graph, which I think you'll all agree, was a lot prettier to look at an array of, say, 1000 numbers.

    All right. So now, on your handout and on the screen, you'll see the way we've done this. So perform trial, we've already seen, so we know what the inner loop looks like. We can ignore first test, that was just my taking the code that I had in line last time and putting it in a function. And now what you'll see in the handout is perform sim and answer question. So, let's look it those. So perform sim, sim for simulation, takes the amount of time, the number of steps for each trial, plus the number of trials. It starts with an empty list, dist short for distances here, saying so far we haven't run any trials, we don't know what any of the distances are. And then for trial in range number of trials, it creates a drunk, and I'm putting the trial as part of the drunk's name, just so we can make sure that the drunks are all different. Then I'm going to create a field with that drunk in it at location 0,0. And then I'm going to call perform trial with the time in that field to get the distances.

    What does perform trial return? We looked at it earlier. What is perform trial returning? Somebody? Surely someone can figure this one out. I have a whole new bag of candy here, post-Halloween. What kind of thing is it returning? Is it returning a number? If you think it's returning a number, oh, ok, you gonna tell me what kind of thing it's returning, please? A list, thank you. And it's a list of what? I'm giving you the candy on spec, assuming you'll give me the right answer. List of distances, exactly. So how far way it is after each time step? This is exactly the list that we graphed last time.

    So perform sim will get that list and append it to the list. So dist list will be a list of lists, each element will be a list of distances, right? OK, so that's good. And that's now step 2. In some sense, all of this is irrelevant to the actual question we started with, in some sense. This is just the structure. Then I'm going to write a function called answer question, or ans quest, designed to actually address the original question. So it, too, is going to create a list. This is a list called means. Then it's going to call perform sim, to get this list of lists, then it will go through that and calculate the means and create a list of means. And then it will plot it, and we'll get to, before this lecture's over, how the plotting works. So I'm following exactly that structure here. It calls this function, which runs an appropriate number of trials by calling that function, and then we'll calculate and present some statistics.

    So now let's run it. All right, so what have I done? I typed an inadvertent chara -- ah, yes, I typed an s which I didn't intend to type. It's going to take a little while, it's loading Pylab. Now it's running the simulation. All right, and here's a picture. So, when we ran it, let's look at the code for a minute here, what we can see is at the bottom, I called ans quest, saying each trial should be 500 steps, and run 100 trials. And then we'll plot this graph. Graph is lurking somewhere in there, it is. And one of the nice things we'll see is, it's kind of smooth, And we'll come back to this, but the fact that it's sort of smooth makes me feel that running 100 trials might actually be enough to give me a consistent answer. You know, if it had been bouncing up and down as we went, then we'd say, jeez, no trend here. Seeing a relatively smooth trend makes me feel somewhat comfortable that we're actually getting an appropriate answer. And that if I were to run 500 trials, the line would be smoother, but it would look kind of the same. Because it doesn't look like it's moving here, in arbitrary directions large amounts. It's not like the stock market.

    Should I be happy? I've sort of done what I wanted, I kind of I think I have an answer now, which is 500 steps, it should be four and a half units away from the origin. What do you think? Who think this is the right answer? So who thinks it's a wrong answer, raise your hand? All right, TAs, what do you guys think? Putting you on the spot. Right answer or wrong answer? They think it's right. Well, shame on them. Let's remember, rack our brains to a week ago, when we ran a bunch of individual tests. And let's see what we get if we do that. And the point here is, it's always good to check. My recollection, when I looked at this, was that something was amiss. Because I kind of remember, when I ran the test last time, we were more like 40 away than four away. Well all right, let's try it. We'll, sometimes happens, all right, I'm going to have to restart Idol here, just, as you all, at least all who use Macintoshes know, this happens sometimes, it's not catastrophic. Sigh. So this reminds me of the old joke. That a computer scientist, a mechanical engineer, were riding in a car and the car stalled, stopped running. And the mechanical engineer said I know what to do, let's go out and check the carburetor, and look at the engine. The computer scientist said, no that's the wrong thing to do. What you ought to do is, let's turn off the key, get out of the car, shut the doors, open the doors, get back in and restart it. And sure enough, it worked. So when in doubt, reboot.

    So, we'll come down, we'll do that, and we're going to call first test here, and see what that gives us. And we'll, for the moment, ignore that. Well, look at this. We ran a bunch of Homer's random walks, and maybe it isn't 40, but not even one of them was four. So now we see is, we've run two tests, and we've gotten inconsistent answers. Well, we don't know which one is wrong. We know that one of them is wrong. We don't even know that, maybe we just got unlucky with these five tests. But odds are, something is wrong, that there's a bug here. And, we have to figure out which one. So how would we go about doing that? Well, I'm going to do what I always recommend people do. Which was, find a really simple example, One for which I actually know the answer. So what would be a good example for which I might know the answer? Give me the simplest example of a simulation of the random walk I could run, where you're confident you know what the answer is. Yeah? one step, exactly, and what's the answer after one step? One. She can't catch and talk at the same time. Exactly. So we know if we simulate it, one, the drunk has moved in some direction, and is going to be exactly one step from the origin. So now we can go and see what we get. So let's do that. And we'll change this to be one, and we'll change this to be one. We'll see what the answer is.

    Well. 50? Well, kind of makes me worry. 1. All right, so we see that the simple test of Homer gives me the right answer. We don't know it's always the right answer, we know it's, at least for this 1. But we know the other 1 was just way off. Interestingly, unlike the previous time, instead of being way too low, it's way too high. So that gives us some pause for thought. And now we need to go in and debug it. So let's go and debug it. And seems to me the right thing to do is to go here. Oh, boy, I'm going to have to restart it again, not good. And we'll put an intermediate value. Actually, maybe we'll do it. What would be a nice thing to do here? We're going to come here. Well, let's, you know, we want to go somewhere halfway through the program, print some intermediate value that will give us some information. So, this might be a good place. And, what should we print? Well, what values do you think you should get here? Pardon?

    STUDENT: The total distance so far.

    PROFESSOR: The total distance so far. So that would be a good thing to print. And what do we think it should be? We'll comment this 1 out since we think that works, and just to be safe, let's not even run 100 trials let's, run one trial, or two trials maybe. See we get. and then 2. W, was sort of what we expected the first time around, but 2? How did you get to be 2? Anyone want to see what's going on here? So we see, right here we have the wrong answer. Well, maybe we should see what things looked like before this? Is it the lists are wrong? What am I doing wrong here? I'll bet someone can figure this out. Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Well, I'm adding them up, fair enough. But so tot looks OK. So, all right, maybe we should take a look at means. Right? Let's take a look at what that looks like. Not bad. All right, so maybe my example's too simple. Let's try a little bit bigger. Hmmm -- 2.5? All right, so now I know what's going wrong is, somehow not that I'm messing up tot, but that I'm computing the mean incorrectly. Where am I computing the mean? They're only two expressions here. There's tot, we've checked that. So there must be a problem with the divisor, that's the only thing that's left. Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly right. I should be dividing by the length of the list. The number of things I'm adding to tot. So I just, inadvertently, divided by, I have a list of lists. And what I really wanted to do is, divide by the number of lists. Because I'm computing the mean for each list, adding it to total, and then at the end I need to divide by the number of lists who's means I computed, not by the length of 1 of the lists, right? So now, let's see what happens if we run it. Now, we get some output printed, which I really didn't want, but it happens. Well this looks a lot better. Right? Sure enough, it's 1. All right, so now I'm feeling better. I'm going to get rid of this print statement, if we're gonna run a more extensive test. And now we can go back to our original question. And run it.

    Well, this looks a lot more consistent with what we saw before. It says that on average, you should be around 20. So we feel pretty good about that. Now, just to feel even better, I'm going to double the number of trials and see what that tells us. And it's still around 20. Line a little smoother. And if I where do 1000 trials will get a little smoother, and it would still be around 20. Maybe slightly different each time, but consistent with what we saw before, when we ran the other program. We can feel that we're actually doing something useful. And so now we can conclude, and would actually be the correct conclusion, that we know about how far this random drunk is going to move in 500 steps. And if you want to know how far he would move in 1000 steps, we could try that, too.

    All right. What are the lessons here? One lesson is to look at the labels on the axes. Because if we just looked at it without noticing these numbers, it looks the same. Right? This doesn't look any different, in some sense, than when the numbers were four. So you can't just look at the shape of the curve, you have to look at the values. So what does that tell me? It tells me that a responsible person will always label the axes. As I have done here, not only giving you the numbers, but telling you it's the distance. I hate it when I look at graphs and I have to guess what the x- and y- axes are. Here it says time versus distance, and you also notice I put a title on it. So there's a point there. And look, when you're doing it.

    Ask if the answer make sense. One of the things we'll see as we go on, is you can get all your statistics right, and still get the wrong answer because of a consistent bug. And so always just say, do I believe it, or is this so counterintuitive that I'm suspicious? And as part of that ask, is it consistent with other evidence? In this case we had the evidence of watching an individual walk. Now those two things were not consistent, don't know which is wrong, but it must be one of them. And then the final point I wanted to make, is that you can be pretty systematic about debugging, And in particular, debug with a simple example. Right, instead of trying to debug 500 steps and 100 trials, I said, all right, let's look at one step and four trials, five trials. OK, where in my head I knew what it should look like, and then I could check it. All right.

    Jumping up a level or three of abstraction now. What we've done, is we've introduced the notion of a random walk in the context of a pretty contrived example. But in fact, it's worth knowing that random walks are used all over the place to solve real problems, deal with real phenomena. So for example, if you look at something like Brownian motion, which can be used to model the path traced by a molecule as it travels in a liquid or a gas. Typically, people who do that model it using a random walk. And, depending upon, say the density of the gas or the liquid, the size of the molecules, they change parameters in the simulation, how far it, say, goes in each unit time and things like that. But they use a random walk to try and model what will really happened. People have attempted, for several hundred years now, to use, well, maybe a 150 years, to use random walks to model the stock market. There was a very famous book called A Random Walk Down Wall Street, that argued that things happened as a, random walk was a good way to model things. There's a lot of evidence that says that's wrong, but people continue to attempt to do it.

    They use it a lot in biology to do things like model kinetics. So, the kinetics of a protein, DNA strand exchange, things of that nature. A separation of macro-molecules, the movement of microorganisms all of those things are done in biology. And do that. People use it to model evolution. They look at mutations as kind of a random event. So, we'll come back to this, but random walks are used over and over and over again in the sciences, the social sciences and therefore a very useful thing to notice about. All right, we're going to come back to that. We're going to even come back to our drunken student and look at other kinds of random walks other than the kind we just looked at.

    Before I do that, though, I wanted back up and take the magic out of plotting. So we've gone from the sublime, of what random walks are good for, to in some sense the ridiculous, the actual syntax for plotting things. And maybe it's not ridiculous, but it's boring. But you need it, so let's look at it. So we're doing this using a package called Pylab, which is in itself built on a package called Pylab, either pronounced num p or num pi, you can choose your pronunciation as you prefer. This basically gives you a lot of operations on numbers, numbered things, and on top of that, someone bill Pylab which is designed to provide a Python interface to a lot of the functionality you get in Matlab. And in particular, we're going to be using today the plotting functionality that comes with Matlab, or the version of it. So we're going to say, from Pylab import star, that's just so I don't have to type Pylab dot plot every time. And I'm going import random which we're going to use later. So let's look at it now.

    First thing we're going to do is plot 1, 2, 3, 4, and then 1, 2, 3, and then 5, 6, 7, 8. And then at the very bottom, you'll see this line show. That's going to annoy the heck out of you throughout the semester, the rest of the semester. Because what happens is, Pylab produces all these beautiful plots, and then does not display them until you type show. So remember, at the end of every program, kind of, the last thing you should execute should be show. You don't want to execute it in the middle, because what happens in the middle is it, in an interactive mode at least, it just stops. And displays the graphs, and until you make the plots go away, it won't execute the next line. Which is why I've tucked the show at the very bottom of my script here. Inevitably, you will forget to type show. You will ask a TA, how come my graphs aren't appearing in the screen, and the TA will say, did you do show? And you'll go -- but it happens to all of us.

    All right, so let's try it. See what we get. So sure enough, it's plotted the values 1, 2, 3, 4, and 5, 6, 7, 8, on the x- and y- axis. Two things I want you to notice here. One Is, that both plots showed up on the same figure. Which is sometimes what you want, and sometimes not what you want. You'll notice that also happened with the random walk we looked at, where when I plotted five different walks for Homer they all showed up superimposed on top of one another. The other thing I want you to notice, is the x-axis runs from to 3. So you might have kind of thought, that what we would see is a 45 degree angle on these things. But of course, Python, when not instructed otherwise, always starts at zero. Since when we called plot, I gave it only the y-values, it used default values for x. It was smart enough to say, since we have four y-values, we should need four x-values, and I'll choose the integers 0, 1, 2, 3 as those values. Now you don't have to do that. We could do this instead. Let's try this one. What did I just do? Let's comment these two out, if we could only get there. This is highly annoying. Let's hope it doesn't tell me that I have to -- All right, so let's go here. We'll get rid of those guys, and we'll try this one. We'll plot 1, 2, 3, 4 against 1, 4, 9, 16. OK? So now, it's using 1, 2, 3, 4 as the x-axis, and the y-axis I gave it. First x then y.

    Now it looks a little funny, right, you might have not expected it to look like this. You'll notice they're these little inflection points here. Well, because what it's really doing is, I gave it a small number of points, only four. It's found those four points, and it's connected them, each point by a straight line. And since the points are kind of spread out, the line has little bumps in it. That makes sense to everyone? Now, it's often deceptive to plot things this way, where you think you have a continuous function when in fact you just have a few miscellaneous points. So let's look at another example. Here, what I'm going to do, is I've called figure, and remember, this is Pylab dot figure, which says, create a new figure. So instead of putting this new curve on the same figure as the old curve, start a new one. And furthermore, I've got this obscure little thing at the end of it. After you give it the x- and y- values, you can give it some instructions about how you want to plot points, or anything else. In this case, what this little string says is, each point should be represented as a red o. r for red, o for o.

    I'm not asking you to remember this, what you will discover, the good news is there's very good documentation on this. And so you'll find in the reading of pointer to plots, and it will tell you everything you need to know, all of the wizardry and the magic you can put in these strings that tell you how to do things. These are basically the same strings borrowed from Matlab. And now if we run it. Figure one is the same figure we saw before. But figure two has not connected the dots, not drawn a line, it's actually planted each, or plotted, excuse me, each point as a red circle.

    Now when I look at this, there's something that's not very pleasing about this. That in particular, I know I plotted four points, but it a quick glance it looks like they're only three. And that's because it's taking this fourth point and stuck it way up there in the corner where I missed it. It's there. But it's so close to the edge of the graph that it's kind of hard to see. So I can fix that by executing the command axis, which tells it how far I want it to be. And this says, I want 1 axis to go from 0 to 6, and the other to 20. We'll do that, and also to avoid boring you, we'll do more at the same time.

    We'll put some labels on these things. I'm going to put that the title of the graph is going to be earnings, and that the x-axis will be labelled days, and the y-axis will be labelled dollars. So earnings dollars against days. OK, now let's see what happens when we do this. Well, we get the same ugly figure one as before, and now you can see figure two I've moved the axes so that my graph will show up in the middle rather than at the edges, and therefore easier to read. I put a title in the top, and I put labels on the axes. Every graph that I ask you to do this course, I want you to put a title on it and to label your axes so we know what we're reading. Again, nothing very deep here, this is really just syntax, just to give you an idea of the sorts of things you can do.

    All right. now we get to something a little bit more interesting. Let's look at this code here. So far, what I've been passing to the plot function for the x- and y- values are lists. In fact, what Pylab uses is something it gets from NumPy which are not lists really, but what it calls arrays. Now, truth be told, most programming languages use array to mean something quite different. But, in NumPy an array is basically a matrix. On which we can do some interesting things. So for example, when I say x-axes equals array 1, 2, 3, 4. Array is a type, so array applied to the list is just like applying float to an int. If I apply float to an int, it turns it into a floating point number. If I apply array to a list, it turns it into an array. Once it's an array, as we'll see, we can do some very interesting things with it. Now, in addition to getting an array by coercing a the list, which is probably the most common way to get it, by the way. Because you build up a list in simulations of the sort we looked at, and then you might want to change it to an array to perform some operations on it. You can get an array directly with aRange. This is just like the range function we've been using all term, but whereas the range function gives you a list of ints, this gives you an array of ints. But the nice thing about an array is, I can perform operations on it like this. So if I say y-axis equals x-axis raised to the third power, that's something I can't do with a list. I get an error message if I try that with a list. What that will do is, will point-wise, take each element in the array and cube it.

    So the nice thing about arrays is, you can use them to do the kinds of things you, if you ever took a linear algebra course, you learned about doing. You can multiply an array times an array, You can multiply an array times an integer. And sometimes that's a very convenient thing to do. It's not what this course is about, I don't want to emphasize it. I just want you to know it's there so if in some subsequent life you want to do more complicated manipulations, you'll know that that's possible. So let's run this and see what we get. So the first thing to look at, is we'll ignore the figure for the moment. And we've seen that when I printed test and I printed x-axis, they look the same, they are the same. And in fact, I can do this interesting thing now. Print test double equals x-axis. You might have thought that would return a single value, true. Instead it returns a list, where it's done a point-wise comparison of each element. So when we deal with arrays, they don't behave like lists.

    And you can imagine that it might be very convenient to be able to do this. Answers the question, are all the elements the same? Or which ones are the same? So you can imagine doing some very clever things with these. And certainly, if you can convert problems to vectors of this sort, you can really perform what's almost magical. And then when we look at the figure, which should be tucked away somewhere here, what did I do with the figure? Did I make it go away? Well, I think I did one of those ugly things and made it go away again. Oh, no, there it is. All right. And sure enough here, we're plotting a cubic. All right. Nothing very important to observe about any of that, other than that arrays are really quite interesting and can be very valuable.

    Finally, the thing I want to show you is that there are a lot of things we can do that are more interesting than what we've done. So now I'm going to use that random, which I brought in before, and show you that we can plot things other than simply curves. In this case, I'm going to plot a histogram. And what this histogram is going to do is, I'm going to throw a pair of dice a large number of times, and add up the sum, and see what I get. So, for die values equals 1 through 6, for i in range 10,000, a lot of dice. I'm just going to append random choice, we've seen this before, of the two dice, and their sum. And then I'm going to plot a histogram, Pylab dot hist instead of plot, and we'll get something quite different. A nice little histogram showing me the values I get, and we will come back to this later and talk about why this is called a normal distribution.

    LECTURE TRANSCRIPT

    PROFESSOR: You'll recall, at least some of you will recall, that last time we ended up looking at a simulation of a drunken university student wandering randomly in a field. I'm going to return to that. Before I get to the interesting part, I do want to call your attention to something that some people seemed a little bit confused by last time. So you remember that we had this thing called perform trial. And the way we basically tested the drunk, what would happen, is we would call this, passing in a time, the number of steps, the amount of time that the drunk was wondering, and the field. And there was just 1 line, that I blazed past, probably too quickly. So what that line is saying, is we, from the field, we're going to call the method get drunk, which is going to return us a drunk. And then we're going to call the move method for that drunk, passing the field as an argument. Now the question is, what would have happened if I had left off these parentheses? There. A volunteer? What would, we will do the experiment in a minute. And by the way, I want to emphasize the importance of experimentation. I was kind of surprised, for example, to get email from people saying, well, what was the correct answer to question 3 on the quiz? And my sort of response was, why don't you type it and find out?

    When in doubt, run the experiment. So, what's going to happen, what would have happened, had I failed to type that pair of parentheses? Anybody? Well, what is, get drunk? It is a method. And remember, in Python, methods, like classes, are themselves objects. So get drunk would have a returned an object which was a method, and then, well, let's try it. And the error message will tell us everything we need to know. Function has no attribute move. Sure enough, the method does not have an attribute move. The instance of the class has the attribute. And so what those parentheses did, was tell me to invoke the method get drunk, so now instead of that being the method itself, it's the result returned by the method. Which is an instance of class drunk, and that instance has an attribute move, which is itself a method. This is a very common programming paradigm, and it's important to sort of lock into your heads, the distinction between the method and an invocation of the method. Because you can write either, and sometimes you won't get an error message, you'll just get the wrong answer. And that's kind of bad. That make sense to everybody? Does it not make sense to anybody, maybe is the question I should ask?

    All right, we'll go on, and we'll see more examples of this kind of thing. Now last week, I ran this several times to see what would happen, and we saw that in fact contrary to everyone's, most everyone's, expectation, the longer we ran the simulation, the further the drunk was from the starting point. And we saw that by plotting how far the drunk was after each time unit. We ran it several times, we got different answers. And at that point we should ask ourselves, is that really the way to go about answering the original question? Which was, how far should we expect the drunk to be? The answer is no. I don't want to sit there at the keyboard typing in 400 examples and then, in my head, trying to figure out what's quote, typical, unquote. Instead, I need to organize my simulation so that it runs a number of trials for me and summarizes the results. All the simulations we're going to look at it, in fact almost all the simulations anyone will ever write, sort of, have the same kind of structure. We start with an inner loop that simulates one trial.

    That's what we have here, right, we have happen to have a function. So when I say inner loop, what I mean is, not that I'll write a program a bunch of nested loops, but that I'll write a program with some function calls, and down at sort of the bottom of the stack will be perform trial. Which stimulates one trial of some number of seconds, in this case. And then I'll quote, enclose, unquote the inner loop in another loop that conducts an appropriate number of trials. Now a little bit later in the term, we'll get to the question of, how do we know what an appropriate number is? For today, we'll just say, a lot. And we'll talk about that a little bit more. And then finally, what we want to do, is calculate and present some relevant statistics about the trials. And we'll talk about what's relevant. I want to emphasize today the presentation of them. Last time we looked at a graph, which I think you'll all agree, was a lot prettier to look at an array of, say, 1000 numbers.

    All right. So now, on your handout and on the screen, you'll see the way we've done this. So perform trial, we've already seen, so we know what the inner loop looks like. We can ignore first test, that was just my taking the code that I had in line last time and putting it in a function. And now what you'll see in the handout is perform sim and answer question. So, let's look it those. So perform sim, sim for simulation, takes the amount of time, the number of steps for each trial, plus the number of trials. It starts with an empty list, dist short for distances here, saying so far we haven't run any trials, we don't know what any of the distances are. And then for trial in range number of trials, it creates a drunk, and I'm putting the trial as part of the drunk's name, just so we can make sure that the drunks are all different. Then I'm going to create a field with that drunk in it at location 0,0. And then I'm going to call perform trial with the time in that field to get the distances.

    What does perform trial return? We looked at it earlier. What is perform trial returning? Somebody? Surely someone can figure this one out. I have a whole new bag of candy here, post-Halloween. What kind of thing is it returning? Is it returning a number? If you think it's returning a number, oh, ok, you gonna tell me what kind of thing it's returning, please? A list, thank you. And it's a list of what? I'm giving you the candy on spec, assuming you'll give me the right answer. List of distances, exactly. So how far way it is after each time step? This is exactly the list that we graphed last time.

    So perform sim will get that list and append it to the list. So dist list will be a list of lists, each element will be a list of distances, right? OK, so that's good. And that's now step 2. In some sense, all of this is irrelevant to the actual question we started with, in some sense. This is just the structure. Then I'm going to write a function called answer question, or ans quest, designed to actually address the original question. So it, too, is going to create a list. This is a list called means. Then it's going to call perform sim, to get this list of lists, then it will go through that and calculate the means and create a list of means. And then it will plot it, and we'll get to, before this lecture's over, how the plotting works. So I'm following exactly that structure here. It calls this function, which runs an appropriate number of trials by calling that function, and then we'll calculate and present some statistics.

    So now let's run it. All right, so what have I done? I typed an inadvertent chara -- ah, yes, I typed an s which I didn't intend to type. It's going to take a little while, it's loading Pylab. Now it's running the simulation. All right, and here's a picture. So, when we ran it, let's look at the code for a minute here, what we can see is at the bottom, I called ans quest, saying each trial should be 500 steps, and run 100 trials. And then we'll plot this graph. Graph is lurking somewhere in there, it is. And one of the nice things we'll see is, it's kind of smooth, And we'll come back to this, but the fact that it's sort of smooth makes me feel that running 100 trials might actually be enough to give me a consistent answer. You know, if it had been bouncing up and down as we went, then we'd say, jeez, no trend here. Seeing a relatively smooth trend makes me feel somewhat comfortable that we're actually getting an appropriate answer. And that if I were to run 500 trials, the line would be smoother, but it would look kind of the same. Because it doesn't look like it's moving here, in arbitrary directions large amounts. It's not like the stock market.

    Should I be happy? I've sort of done what I wanted, I kind of I think I have an answer now, which is 500 steps, it should be four and a half units away from the origin. What do you think? Who think this is the right answer? So who thinks it's a wrong answer, raise your hand? All right, TAs, what do you guys think? Putting you on the spot. Right answer or wrong answer? They think it's right. Well, shame on them. Let's remember, rack our brains to a week ago, when we ran a bunch of individual tests. And let's see what we get if we do that. And the point here is, it's always good to check. My recollection, when I looked at this, was that something was amiss. Because I kind of remember, when I ran the test last time, we were more like 40 away than four away. Well all right, let's try it. We'll, sometimes happens, all right, I'm going to have to restart Idol here, just, as you all, at least all who use Macintoshes know, this happens sometimes, it's not catastrophic. Sigh. So this reminds me of the old joke. That a computer scientist, a mechanical engineer, were riding in a car and the car stalled, stopped running. And the mechanical engineer said I know what to do, let's go out and check the carburetor, and look at the engine. The computer scientist said, no that's the wrong thing to do. What you ought to do is, let's turn off the key, get out of the car, shut the doors, open the doors, get back in and restart it. And sure enough, it worked. So when in doubt, reboot.

    So, we'll come down, we'll do that, and we're going to call first test here, and see what that gives us. And we'll, for the moment, ignore that. Well, look at this. We ran a bunch of Homer's random walks, and maybe it isn't 40, but not even one of them was four. So now we see is, we've run two tests, and we've gotten inconsistent answers. Well, we don't know which one is wrong. We know that one of them is wrong. We don't even know that, maybe we just got unlucky with these five tests. But odds are, something is wrong, that there's a bug here. And, we have to figure out which one. So how would we go about doing that? Well, I'm going to do what I always recommend people do. Which was, find a really simple example, One for which I actually know the answer. So what would be a good example for which I might know the answer? Give me the simplest example of a simulation of the random walk I could run, where you're confident you know what the answer is. Yeah? one step, exactly, and what's the answer after one step? One. She can't catch and talk at the same time. Exactly. So we know if we simulate it, one, the drunk has moved in some direction, and is going to be exactly one step from the origin. So now we can go and see what we get. So let's do that. And we'll change this to be one, and we'll change this to be one. We'll see what the answer is.

    Well. 50? Well, kind of makes me worry. 1. All right, so we see that the simple test of Homer gives me the right answer. We don't know it's always the right answer, we know it's, at least for this 1. But we know the other 1 was just way off. Interestingly, unlike the previous time, instead of being way too low, it's way too high. So that gives us some pause for thought. And now we need to go in and debug it. So let's go and debug it. And seems to me the right thing to do is to go here. Oh, boy, I'm going to have to restart it again, not good. And we'll put an intermediate value. Actually, maybe we'll do it. What would be a nice thing to do here? We're going to come here. Well, let's, you know, we want to go somewhere halfway through the program, print some intermediate value that will give us some information. So, this might be a good place. And, what should we print? Well, what values do you think you should get here? Pardon?

    STUDENT: The total distance so far.

    PROFESSOR: The total distance so far. So that would be a good thing to print. And what do we think it should be? We'll comment this 1 out since we think that works, and just to be safe, let's not even run 100 trials let's, run one trial, or two trials maybe. See we get. and then 2. W, was sort of what we expected the first time around, but 2? How did you get to be 2? Anyone want to see what's going on here? So we see, right here we have the wrong answer. Well, maybe we should see what things looked like before this? Is it the lists are wrong? What am I doing wrong here? I'll bet someone can figure this out. Pardon?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Well, I'm adding them up, fair enough. But so tot looks OK. So, all right, maybe we should take a look at means. Right? Let's take a look at what that looks like. Not bad. All right, so maybe my example's too simple. Let's try a little bit bigger. Hmmm -- 2.5? All right, so now I know what's going wrong is, somehow not that I'm messing up tot, but that I'm computing the mean incorrectly. Where am I computing the mean? They're only two expressions here. There's tot, we've checked that. So there must be a problem with the divisor, that's the only thing that's left. Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly right. I should be dividing by the length of the list. The number of things I'm adding to tot. So I just, inadvertently, divided by, I have a list of lists. And what I really wanted to do is, divide by the number of lists. Because I'm computing the mean for each list, adding it to total, and then at the end I need to divide by the number of lists who's means I computed, not by the length of 1 of the lists, right? So now, let's see what happens if we run it. Now, we get some output printed, which I really didn't want, but it happens. Well this looks a lot better. Right? Sure enough, it's 1. All right, so now I'm feeling better. I'm going to get rid of this print statement, if we're gonna run a more extensive test. And now we can go back to our original question. And run it.

    Well, this looks a lot more consistent with what we saw before. It says that on average, you should be around 20. So we feel pretty good about that. Now, just to feel even better, I'm going to double the number of trials and see what that tells us. And it's still around 20. Line a little smoother. And if I where do 1000 trials will get a little smoother, and it would still be around 20. Maybe slightly different each time, but consistent with what we saw before, when we ran the other program. We can feel that we're actually doing something useful. And so now we can conclude, and would actually be the correct conclusion, that we know about how far this random drunk is going to move in 500 steps. And if you want to know how far he would move in 1000 steps, we could try that, too.

    All right. What are the lessons here? One lesson is to look at the labels on the axes. Because if we just looked at it without noticing these numbers, it looks the same. Right? This doesn't look any different, in some sense, than when the numbers were four. So you can't just look at the shape of the curve, you have to look at the values. So what does that tell me? It tells me that a responsible person will always label the axes. As I have done here, not only giving you the numbers, but telling you it's the distance. I hate it when I look at graphs and I have to guess what the x- and y- axes are. Here it says time versus distance, and you also notice I put a title on it. So there's a point there. And look, when you're doing it.

    Ask if the answer make sense. One of the things we'll see as we go on, is you can get all your statistics right, and still get the wrong answer because of a consistent bug. And so always just say, do I believe it, or is this so counterintuitive that I'm suspicious? And as part of that ask, is it consistent with other evidence? In this case we had the evidence of watching an individual walk. Now those two things were not consistent, don't know which is wrong, but it must be one of them. And then the final point I wanted to make, is that you can be pretty systematic about debugging, And in particular, debug with a simple example. Right, instead of trying to debug 500 steps and 100 trials, I said, all right, let's look at one step and four trials, five trials. OK, where in my head I knew what it should look like, and then I could check it. All right.

    Jumping up a level or three of abstraction now. What we've done, is we've introduced the notion of a random walk in the context of a pretty contrived example. But in fact, it's worth knowing that random walks are used all over the place to solve real problems, deal with real phenomena. So for example, if you look at something like Brownian motion, which can be used to model the path traced by a molecule as it travels in a liquid or a gas. Typically, people who do that model it using a random walk. And, depending upon, say the density of the gas or the liquid, the size of the molecules, they change parameters in the simulation, how far it, say, goes in each unit time and things like that. But they use a random walk to try and model what will really happened. People have attempted, for several hundred years now, to use, well, maybe a 150 years, to use random walks to model the stock market. There was a very famous book called A Random Walk Down Wall Street, that argued that things happened as a, random walk was a good way to model things. There's a lot of evidence that says that's wrong, but people continue to attempt to do it.

    They use it a lot in biology to do things like model kinetics. So, the kinetics of a protein, DNA strand exchange, things of that nature. A separation of macro-molecules, the movement of microorganisms all of those things are done in biology. And do that. People use it to model evolution. They look at mutations as kind of a random event. So, we'll come back to this, but random walks are used over and over and over again in the sciences, the social sciences and therefore a very useful thing to notice about. All right, we're going to come back to that. We're going to even come back to our drunken student and look at other kinds of random walks other than the kind we just looked at.

    Before I do that, though, I wanted back up and take the magic out of plotting. So we've gone from the sublime, of what random walks are good for, to in some sense the ridiculous, the actual syntax for plotting things. And maybe it's not ridiculous, but it's boring. But you need it, so let's look at it. So we're doing this using a package called Pylab, which is in itself built on a package called Pylab, either pronounced num p or num pi, you can choose your pronunciation as you prefer. This basically gives you a lot of operations on numbers, numbered things, and on top of that, someone bill Pylab which is designed to provide a Python interface to a lot of the functionality you get in Matlab. And in particular, we're going to be using today the plotting functionality that comes with Matlab, or the version of it. So we're going to say, from Pylab import star, that's just so I don't have to type Pylab dot plot every time. And I'm going import random which we're going to use later. So let's look at it now.

    First thing we're going to do is plot 1, 2, 3, 4, and then 1, 2, 3, and then 5, 6, 7, 8. And then at the very bottom, you'll see this line show. That's going to annoy the heck out of you throughout the semester, the rest of the semester. Because what happens is, Pylab produces all these beautiful plots, and then does not display them until you type show. So remember, at the end of every program, kind of, the last thing you should execute should be show. You don't want to execute it in the middle, because what happens in the middle is it, in an interactive mode at least, it just stops. And displays the graphs, and until you make the plots go away, it won't execute the next line. Which is why I've tucked the show at the very bottom of my script here. Inevitably, you will forget to type show. You will ask a TA, how come my graphs aren't appearing in the screen, and the TA will say, did you do show? And you'll go -- but it happens to all of us.

    All right, so let's try it. See what we get. So sure enough, it's plotted the values 1, 2, 3, 4, and 5, 6, 7, 8, on the x- and y- axis. Two things I want you to notice here. One Is, that both plots showed up on the same figure. Which is sometimes what you want, and sometimes not what you want. You'll notice that also happened with the random walk we looked at, where when I plotted five different walks for Homer they all showed up superimposed on top of one another. The other thing I want you to notice, is the x-axis runs from to 3. So you might have kind of thought, that what we would see is a 45 degree angle on these things. But of course, Python, when not instructed otherwise, always starts at zero. Since when we called plot, I gave it only the y-values, it used default values for x. It was smart enough to say, since we have four y-values, we should need four x-values, and I'll choose the integers 0, 1, 2, 3 as those values. Now you don't have to do that. We could do this instead. Let's try this one. What did I just do? Let's comment these two out, if we could only get there. This is highly annoying. Let's hope it doesn't tell me that I have to -- All right, so let's go here. We'll get rid of those guys, and we'll try this one. We'll plot 1, 2, 3, 4 against 1, 4, 9, 16. OK? So now, it's using 1, 2, 3, 4 as the x-axis, and the y-axis I gave it. First x then y.

    Now it looks a little funny, right, you might have not expected it to look like this. You'll notice they're these little inflection points here. Well, because what it's really doing is, I gave it a small number of points, only four. It's found those four points, and it's connected them, each point by a straight line. And since the points are kind of spread out, the line has little bumps in it. That makes sense to everyone? Now, it's often deceptive to plot things this way, where you think you have a continuous function when in fact you just have a few miscellaneous points. So let's look at another example. Here, what I'm going to do, is I've called figure, and remember, this is Pylab dot figure, which says, create a new figure. So instead of putting this new curve on the same figure as the old curve, start a new one. And furthermore, I've got this obscure little thing at the end of it. After you give it the x- and y- values, you can give it some instructions about how you want to plot points, or anything else. In this case, what this little string says is, each point should be represented as a red o. r for red, o for o.

    I'm not asking you to remember this, what you will discover, the good news is there's very good documentation on this. And so you'll find in the reading of pointer to plots, and it will tell you everything you need to know, all of the wizardry and the magic you can put in these strings that tell you how to do things. These are basically the same strings borrowed from Matlab. And now if we run it. Figure one is the same figure we saw before. But figure two has not connected the dots, not drawn a line, it's actually planted each, or plotted, excuse me, each point as a red circle.

    Now when I look at this, there's something that's not very pleasing about this. That in particular, I know I plotted four points, but it a quick glance it looks like they're only three. And that's because it's taking this fourth point and stuck it way up there in the corner where I missed it. It's there. But it's so close to the edge of the graph that it's kind of hard to see. So I can fix that by executing the command axis, which tells it how far I want it to be. And this says, I want 1 axis to go from 0 to 6, and the other to 20. We'll do that, and also to avoid boring you, we'll do more at the same time.

    We'll put some labels on these things. I'm going to put that the title of the graph is going to be earnings, and that the x-axis will be labelled days, and the y-axis will be labelled dollars. So earnings dollars against days. OK, now let's see what happens when we do this. Well, we get the same ugly figure one as before, and now you can see figure two I've moved the axes so that my graph will show up in the middle rather than at the edges, and therefore easier to read. I put a title in the top, and I put labels on the axes. Every graph that I ask you to do this course, I want you to put a title on it and to label your axes so we know what we're reading. Again, nothing very deep here, this is really just syntax, just to give you an idea of the sorts of things you can do.

    All right. now we get to something a little bit more interesting. Let's look at this code here. So far, what I've been passing to the plot function for the x- and y- values are lists. In fact, what Pylab uses is something it gets from NumPy which are not lists really, but what it calls arrays. Now, truth be told, most programming languages use array to mean something quite different. But, in NumPy an array is basically a matrix. On which we can do some interesting things. So for example, when I say x-axes equals array 1, 2, 3, 4. Array is a type, so array applied to the list is just like applying float to an int. If I apply float to an int, it turns it into a floating point number. If I apply array to a list, it turns it into an array. Once it's an array, as we'll see, we can do some very interesting things with it. Now, in addition to getting an array by coercing a the list, which is probably the most common way to get it, by the way. Because you build up a list in simulations of the sort we looked at, and then you might want to change it to an array to perform some operations on it. You can get an array directly with aRange. This is just like the range function we've been using all term, but whereas the range function gives you a list of ints, this gives you an array of ints. But the nice thing about an array is, I can perform operations on it like this. So if I say y-axis equals x-axis raised to the third power, that's something I can't do with a list. I get an error message if I try that with a list. What that will do is, will point-wise, take each element in the array and cube it.

    So the nice thing about arrays is, you can use them to do the kinds of things you, if you ever took a linear algebra course, you learned about doing. You can multiply an array times an array, You can multiply an array times an integer. And sometimes that's a very convenient thing to do. It's not what this course is about, I don't want to emphasize it. I just want you to know it's there so if in some subsequent life you want to do more complicated manipulations, you'll know that that's possible. So let's run this and see what we get. So the first thing to look at, is we'll ignore the figure for the moment. And we've seen that when I printed test and I printed x-axis, they look the same, they are the same. And in fact, I can do this interesting thing now. Print test double equals x-axis. You might have thought that would return a single value, true. Instead it returns a list, where it's done a point-wise comparison of each element. So when we deal with arrays, they don't behave like lists.

    And you can imagine that it might be very convenient to be able to do this. Answers the question, are all the elements the same? Or which ones are the same? So you can imagine doing some very clever things with these. And certainly, if you can convert problems to vectors of this sort, you can really perform what's almost magical. And then when we look at the figure, which should be tucked away somewhere here, what did I do with the figure? Did I make it go away? Well, I think I did one of those ugly things and made it go away again. Oh, no, there it is. All right. And sure enough here, we're plotting a cubic. All right. Nothing very important to observe about any of that, other than that arrays are really quite interesting and can be very valuable.

    Finally, the thing I want to show you is that there are a lot of things we can do that are more interesting than what we've done. So now I'm going to use that random, which I brought in before, and show you that we can plot things other than simply curves. In this case, I'm going to plot a histogram. And what this histogram is going to do is, I'm going to throw a pair of dice a large number of times, and add up the sum, and see what I get. So, for die values equals 1 through 6, for i in range 10,000, a lot of dice. I'm just going to append random choice, we've seen this before, of the two dice, and their sum. And then I'm going to plot a histogram, Pylab dot hist instead of plot, and we'll get something quite different. A nice little histogram showing me the values I get, and we will come back to this later and talk about why this is called a normal distribution.
  • 31
    Assignment 11: Simulating robots
    11 pages
  • 32
    MIT Lecture 19: Biased Random Walks, Distributions
    49:53
    Lecture 19: Biased random walks, distributionsInstructors: Prof. Eric Grimson, Prof. John Guttag View the complete course at: http://ocw.mit.edu/6-00F08License: Creative Commons BY-NC-SAMore information at http://ocw.mit.edu/termsMore courses at http://ocw.mit.edu

    LECTURE TRANSCRIPT

    PROFESSOR: So you may recall that, in the last lecture, we more or less solved the drunken student problem, looking at a random walk. I now want to move on and discuss some variants of the random walk problem that are collectively known as biased random walks. So the notion here is, the walk is still stochastic but there is some bias in the direction, so the movements are not uniformly distributed or equally distributed in all directions. As I go through this, I want you to sort of, think in advance about what the take-home message should be. One, and this is probably the most important part for today, is I want to illustrate how by designing our programs in a nice way around classes, we can change them to do something else writing a minimal amount of code. And the idea here is that classes are not there just to provide some syntax that makes life complicated, but to actually provide a mechanism that lets us structure our programs in a way that we can modify them later. I also want to give you a bit more experience looking at data, and understanding how, when you get the results of a simulation or anything else that provides you with data, you can plot, them study the plots, and try and use those to develop an understanding about what's going on. And finally, I want to just get you thinking more about random behavior. Because we're going to talk a lot more this semester and in general throughout your careers, you'll see a lot of uses of stochastic things.

    All right, so let's think about these biased walks. So for example, assume that the drunk grew up in South Florida, and really hates the New England winter. Well there might be a bias that even though the drunk is not in total control, she's kind of wandering southward. So there's a bias when taking a step to more likely go south than certainly north. Or imagine that the drunk is photosensitive and moves either towards the sun or away from the sun, or some such thing. So we're going to look at different things of that nature and think about, I'll come back to Chutes and Ladders later in the lecture, you can speculate on why this is here. And we'll look at that. So, this is on the front page of your handout here.

    I've taken the random walk program we looked at last time and changed it a bit. And in particular, I've changed the way the drunk works. So let's look at the code. Wow, haven't even done anything it's -- see how this works. So now, drunk itself is a very small class, one I don't actually intend to ever instantiate. I've pared it down, it has only two methods in it. It has an init, which is the same as before, and a move which is the same as before, almost the same. Now I'm going to have the class usual drunk. That's the drunk we looked at the last time. And this usual drunk will behave just like the drunk we looked at last time. In here, I'm going to override the move method of the inherited superclass, and what this is going to do is, it's going to make the same random choice it made before and then it's going to call drunk dot move with the new compass point. So all I've done is, I've taken a couple of lines of code out of the previous implementation and moved it to the subclass. So it's now, excuse me, in the subclass I'm making the decision about which direction to move. Once I've made that decision, then I can call the drunk in the superclass. Notice here, when I've made the call, drunk dot move. So usually you're expecting me to write something like d dot move. The object. Here, instead I'm specifying which class to get the move from, So this is saying, don't call my local move, call the move from the superclass. There are a number of other syntaxes I could've used to accomplish the same thing. I used this because it seemed the most straightforward.

    Now let's compare the usual drunk to the cold drunk. So the cold drunk, again, I've overridden the move, and now it does something different. It gets the random compass point as before, but if it happens to be south, it calls drunk dot move with twice the distance. So whenever I get the notion I'm moving south, because I love the warmth, I actually move twice as fast. So instead of taking one step, which I do to the other compass points, if I happen to be going south I'll take two steps. So I'm biased towards moving southward. And then I call drunk dot move. Otherwise, I call drunk dot move with the distances before. So I've isolated into the subclass this notion of being heat-seeking. We've got a third sub class, ew drunk, known for east west drunk. And here, what I've done is, I choose to compass point and then while it's not either east or west, I choose again. So this drunk can only move eastward or westward. Can't move north or south. So again you'll see I'm able to have three kinds of drunks. Who as we will see, exhibit rather different behavior from each other with a minimal amount of change to the code.

    If we look down at the bottom, we'll see that the code is pretty much exactly what it was last time. By the way, this doesn't quite match the code in your handout. I decided to make the plots a little bit prettier with better titles and things, but when I tried to put these in your handouts it wouldn't fit. So I have a little more compact things, but this is, I think, a better way to write it. But essentially, perform trial, perform sim, answer quest, are all identical to what they were before. So because I structured my program in such a way that the drunk's behavior was independent of all the code for the generating the trials and plotting the results, I didn't have to change any of that. And this is the key to good design. Is to isolate decisions in small parts of your program, so when as inevitable happens, you want to enhance the program, or you discover you've done something wrong, the number changes you make is not large. OK, one thing we do need to look at down here is, it's not quite the same as before, because we'll see that there's an extra parameter to these things called drunk type. So if we start with answer question, it takes as before, the time, the number of trials, a title, and this one extra thing called the drunk type. And then when it goes on and it calls perform sim, it passes the drunk type. And this is the key line here. Before, you may recall we just said d equals drunk with the arguments. Here, I say d equals drunk type.

    So what is drunk type? Drunk type is itself a type. We can use types, you know variables can have as their values types, parameters can have as their values types. So I'm just passing the type around so that here, when it comes time to get a drunk, I get a drunk of the appropriate type. So I can now answer the question about a usual drunk, or a cold drunk, or a ew drunk. This is very common kind of programming paradigm. We've talked about it before, but this is yet another example of how we use polymorphism. Polymorphism. We write our code in such a way that it works with many types of objects. Here we're writing code that will work with any subtype of drunk. Now I'd better not pass it a float. You know, if I answer quest, and instead of something like usual drunk or cold drunk or e w drunk, I pass it float or int, the world will come crashing down around my head. But I haven't, so you'll see down here, I pass it num steps, number trials, and usual drunk. Which is the name of? A class. Any questions about this? Yes?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Louder, please?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Good, good question. The question is, does a drunk type make a new drunk? Now, I have so much candy to choose from, I don't know what to throw you. Do you like Tootsie Rolls, Snickers, Tootsie Roll Pops, Dots, what's your choice? Snickers. OK, so what is it? Down here nothing is created when that call was made. It was if I had typed the word float or int. I'm merely saying, here is a type, and the parameter should now be bound to that type. Now any time you use the parameter, it's just as if you had written the string, in this case, usual drunk. So when I get up to here, think about, it will take, the variable drunk type, evaluate it, it will evaluate to a type usual drunk, say, in the first call. And then it will, just as if I had written usual drunk, invoke the init method of drunk, and create a new drunk, but this drunk will be of type usual drunk rather than a drunk. Does that make sense? All right, so this avoids my having to write a whole bunch of tests, I could have passed a string, and said if the string said usual drunk, then d equals something. But here instead of that, I'm actually passing the type itself. OK? So other than the fact that I have to pass the type around, this is exactly the same as before.

    Now just for fun, we saw that last time when I asked you what did usual drunk do, there was some confusion. Not confusion, just a bad guess about how the drunk would wander. So, let's try it again. What do you think about the cold drunk? Will the cold drunk get further from the origin than the usual drunk? Not further from the origin? Who thinks the cold drunk will be further away the end of 500 steps? Who thinks the cold drunk will not be further away? And I'll bet we can all agr -- got that right, I think -- and I'll bet you'll all agree that, with higher probability, the cold drunk will be south of the origin than north of the origin. More interesting, how about the ew drunk? So this is a drunk that only goes east or west, east or west. Will this drunk, well, first of all, what's the expected distance? Do you think it should be close to zero or not close to zero? Who thinks it should be close to 0? Who thinks it won't be close to 0? Well you guys have all fallen for the same trick you fell for last time. Fooled you twice. Be careful, one more and you're out. As we'll see. But it's interesting.

    So let's run this program and see what happens. I can take a just a second or two and we should get a bunch of plots. All right, so here's the usual drunk. 100 trials and just as before, wandered away. Moderately smooth, but of course little ups and downs, maybe suggesting averaging over 100 trials is not quite enough. I felt a little better if it were smoother, but the trend is pretty clear here, I think we probably don't doubt it. Well, look at cold drunk. Like a shot heading south. So here we see, there's bias, that if you get south you go twice as far makes a huge difference. Instead of being roughly 20 steps away, it's roughly 120 steps away. So we see, and this is something to think about, a small bias can end up making a huge difference over time. This is why casinos do so well. They don't have to have a very much of a bias in their favor on each roll of the dice in order to accumulate a lot of money over time. It's why if you just weigh your dice a little bit, so that they're not fair, you'll do very very well. I'm not recommending that. Kind of interesting. The e w drunk is pretty much the same distance away as the usual drunk. This is why we run these simulations, is because we learn things. Now, I have to confess, it fooled me, too. The first time I ran this, I looked at this and said, wait a minute, that's not what I expected. So I scratched my head, and thought about, all right, why is it behaving this way? This happens me a lot in my own research, that I crunch some data, do some plots, look at it, get surprised, and on the basis of that I learned something about the underlying system I'm modeling. Because it provokes me to think in ways I had not thought before I saw the data. This is an important lesson. You don't ignore the data, you try and explain the data.

    So when I thought about what happened, I realize that while I might have started at the origin, and after the first step I'm equally likely to be here or here. Once I've taken the first step, now my expected location is not zero, right? The first step, my expected location was zero. Because equally likely to be plus 1 or minus 1. But once I've taken a step and I happen to be here, my expected location, if I'm here, is plus 1, it's not 0. Because half the time I'll get back to 0, but half the time I'll get to 2. So once I have moved away from where I've started, the expectation of where I'll end up has changed. This is kind of a profound thing to think about, because when we look at it, this kind of behavior explains a lot of surprising results in life. Where, if you get a small run of random events that take you far from where you started, it's hard to get back. So if you think about the stock market, for example, if you happen to get unlucky and have a few days or weeks or months, as the case may be, where you get a run of bad days, it's really hard to get back if all the movements after that are random. Because you've established a new baseline. And as we'll see later, when you start doing things at random, it's likely you'll get a few runs in one direction or another, and then you've got a new baseline, and it's unlikely to get back to where you were originally. We'll see a lot of examples of this. But that's what's happening here, and that's why it looks the way it does. That makes sense to people?

    Well, so I had this theory, as I often do when I have theories, I don't believe them, so I decided I better write some more code to check it out. So, what I did is, I took the previous simulation and now instead of looking at different kinds of drunks, I got more aggressive in thinking about my analysis of the data. So again, there's a lesson here. I decided I wanted to analyze my data in new ways. And the beauty of it was, I could do that without changing either any of the code about the drunk, or any they of the code about performing the simulation, really. All I had to do was do a different kind of analysis to a first approximation. So, let's look at what I've done here. What I've decided to do here is not only plot, and again, I think you'll see this code, not only plot how far from the origin the drunk is over time, but I'm going to actually plot two other things. I'm going to use a scatter plot, I think I use scatter in this one, I do, to show where the drunk is at each period of time. Actually, just all the places the drunk ends up in each trial. This will give me a visualized way just see, OK, in my, say, 100 trials, how many the drunk ended up near the origin, how many of them ended up really far away? And just, allow me to visualize what's going on.

    And then I'm going to also use a histogram to sort of summarize some things. To get a look at the distribution of the data. We've been just looking at average results, but it's often the case that the average does not really tell the whole story. You want to ask, well, how many of the drunks ended up really far away? How many of the drunks ended up where they started? You can't tell that from the averages. So you use a distribution, which tells you information about how many people ended up where. So we again saw that, in a less pleasant context, maybe, in looking at the quizzes, remember I sent you some plots about the quizzes, and I didn't just say, here was the average. I showed you the distribution of grades. That allowed you to see, not only where you were relative to the average, but whether you were an outlier. Was your grade much better than the average? Or were there a whole bunch of people around where you are? So there's a lot of information there, that will help us understand what's really going on.

    So we'll run this now. Again, there's nothing very interesting here about how I did it, I just used some of these Pylab dot hist, which gives me a histogram, Pylab dot scatter, which gives me a scatter plot. I'll take 500 steps, 400 trials. I've done some more trials here, so we get some smoother lines, better distributions. And then I'm going to look at the usual drunk and ew drunk only, just to save some time. At least I think I am. My computer runs faster when it's plugged in. All right, but we have our figures here, let's look at them. So here's figure one, the usual drunk. Well, all right, as we've seen this 1000 times, or at least six. So doing what it usually does, not quite 20. This is kind of interesting. This is the final locations of the 400 trials of the usual drunk.

    So we see some information here, that we really couldn't get from the averages. That there are more of these points near the middle, but there are certainly plenty scattered around. And we get to see that, they're pretty much symmetric around where we started. This is what we would have hoped, right? If we had looked at it, and we had seen all the points up here, or most of them up here, we would have said, wait a minute, there's something wrong with my code. This was supposed to not be a biased walk, but an unbiased walk. And if I'd seen a bias in this population, I should have gotten nervous that I had some problems with my implementation. My simulation. So in addition to helping me understand results, this gives me some confidence that I really am doing an unbiased walk, right? Some over here, some over here. Not exactly the same, but close enough that I feel pretty comfortable.

    And if I look at the distribution, which probably I could've if I worked really hard, gotten from the scatter plot, but I really didn't want to, we'll see that most of the drunks actually do end up pretty close to the origin. But that there are a few that are pretty far away. And again, it's close to symmetric around zero. Which gives me a good feeling that things are working the way they should. This, by the way, is what's called a normal distribution. Because once upon a time, people believed this was the way things usually happened. This is called either normal or Gaussian. The mathematician Gauss was one of the first people to write about this distribution. We'll come back to this later, and among other things, we'll try and see whether normal is really normal. But you see it a lot of times. And basically what it says is that most of the values hang out around the average, the middle. And there are a few outliers, and as we get further and further out, fewer and fewer points occur. So you'll see these kind of curves, of distributions, occurring a lot. Again, we'll come back to distributions in a week, or actually in a lecture or two. See a little bit more today, and then more later on.

    All right, here's our east west drunk, again as before, drifting off to around 18. Well, look at these dots. Well, this makes me feel, that at least I know that the drunk is only moving east and west. And again you'll see, pretty dense in the middle and a few outliers. Managed to get a little further east than west, but that happens. East is always more attractive. What we might have guessed. And if we look at the distribution, again we can look at it, and I'm just giving you the east west values here, and again it's about the same. So, what do I want you to take away from these graphs? Different plots show you different things. So I'm not going to try and claim that of these three different plots we did, one is the right one to do. What I'm going to assert is that you want to do all of them. Try and visualize the data in multiple ways. And sometimes it's much easier to, even if different plots contain exactly the same information, and in some sense of the scatter plot and the histogram do contain exactly the same information. In fact, there's more information in the scatter plot, in some sense, because I see each one individually. But it's, for me, a lot harder to see. If I'd ask you from the scatter plot, are they normally distributed? You might have had a hard time answering it. Particularly for the usual drunk, where you just saw a lot of points. So the idea of some summarizing it in the histogram makes it much easier to make sense of the data, and see what you're doing, and to try and learn the lesson you're trying to learn. OK, this makes sense to people?

    All right, let's move on. Now we'll go back to this lovely picture we had before. Any of you ever play the game Chutes and Ladders? Raise your hand? OK, some things never go out of style. Well, imagine your poor drunk wandering through this board, and every once in a while, because I'm kind of a, not a nice person, I've eliminated all the ladders, and have only some chutes. And every once in a while the drunk hits a chute and goes whoosh. And in fact, let's, for the sake of simplicity here, assume that the chutes are going to, every once in a while, the drunk will hit a bad spot, and go right back to the origin. So this poor, say, heat-seeking drunk who's trying to head south, does it for a while and maybe gets zipped right back to where he or she started. All right, I've now told you I want to put some, make this happen. What part of the code do you think I should change? Somebody?

    STUDENT: The field.

    PROFESSOR: The field, absolutely. Those are good to throw short distances. The field, right, because what we're talking about here is not a property of the drunk, or a property of the simulation, but a property of the space in which the drunk is wandering. So again, we see that by structuring the initial program around the natural abstractions that occur in the problem, natural kinds of changes can be localized. So let's go and change the field. So I'm going to have the field I had before plus this thing called an odd field. Odd field will be a subclass of field. This is not odd as in odd or even, this is odd as in strange. So I'm going to first define where my chutes are. So is chute will take, will get the coordinates, and assign it to x and y, so it gets the coordinates of a place, and then will return abs x minus abs y is equal to zero. So, where are my chutes going to occur? Somebody? Yeah, kind of radiating out from the origin. Any place, so I'll have my origin, here. Got my graph. And all along here, any place x and y are equal, there'll be a chute. Now, I have to be a little bit careful that it doesn't happen here, or we won't get anywhere.

    So, and then move, in odd field, will call field dot move, which is unchanged from what it was before. And then it will say, if self dot is chute, set the location back to 0,0. So the poor drunk will move as before, and if he or she hits this wormhole, get instantly teleported back to the origin. Very small change, but it will give us rather different behaviors, at least I think it will. And then if we look at how we use it, not much changes. Except, what do you think is going to have to change? Where will I have to make the change, somebody? You can actually see the code here. I'll have to get a different field, right? So, at the place where I instantiate a field, instead of instantiating a field, I'll instantiate an odd field. Now if I'd had 16 different kinds of odd fields, or even two, I might well have done what I did with drunk. But in order to sort of minimize things, I've done something much smaller. So let's see, where did we get the field? I don't even remember. So, here's what I do when I'm looking at my program and I want to see where something is. I use the text editor, and I'm going to search for it. Let's see. Well, it's not there. There it is.

    So here when I get my field, I get my drunk, and then I get a field which is an odd field. All right, anyone want to speculate what will happen when I run this one? Think the drunk will be closer or further from the origin when we're done? Who thinks closer? Who thinks further? Who thinks no difference? Well, you're right, I mean it's sort of logical if we think about it, that if every once in a while you get to zipped, back it's going to be harder to get further away. We'll see different behaviors, by the way, it depends on the drunk, right? That's true for the usual drunk, but maybe not for the east west drunk. We can see what happens. So let's try some. We'll do it for the usual drunk, which is the more interesting case to start with. So you'll see whereas before we were up just shy of 20 most of the time, here we don't get very much much, further. We do get steadily further, but at a lot slower pace. What do you think it means that, before you remember, we saw a pretty smooth line. Here we see what may look you like a fat line, but is in fact a line with a lot of wiggles in it. What does that imply, about what's going on in the simulation, the fact that line is jagged rather than smooth as it was before? Pardon? Can't hear you?

    STUDENT: That it jumps.

    PROFESSOR: Well, certainly it happens because of the jumps, but what else is going on? But what is it, sort of, imply more, whoa, sorry about that, more generally? Right, what it's saying, is that because of the jumps, there's a lot more variation from trial to trial. Not surprising, right, because you sometimes you hit these wormholes and sometimes you don't. So let's look at this other figure. This is kind of interesting. So here's the scatter plot. And if we zoom in on it, what we see here is, there are essentially holes, you know, these alleys where no points lie. Just as we might have guessed from here. The only way you would get a point lying on these alleys, if it happened to be the very last step of the simulation, not even then, because the last thing we do is go back to the origin. So we can look at this, and say, well, sure enough, this is keeping people off of a certain part of the field. All right, some of you will be happy to know we are, for the moment, actually maybe for the whole rest of the term, leaving the notion of drunks. Though not of random walks. All right, any questions about what's going on here? And again, I would urge you to sort of study the code and the way it's been factored to accomplish these things. And play with it, and look at what you get from the different plots.

    I now want to pull back from this specific example, and spend a few minutes talking about simulation in general. Computer simulation really grew hand in hand with the development of the computers, from the very beginning. The first large-scale deployment of computer simulation was as part of the Manhattan Project. This was done during the war to model the process of nuclear detonation. And, in fact, what they did was a simulation of 12 hard spheres, and what would happen when they would bump into each other. It was a huge step forward to do it with simulation, since the whole project had been stalled by their attempt to do this analytically. They were unable to actually solve the problem analytically. And it was only when they hit upon the notion of using a computer, a relatively new tool in those days, to simulate it, that they were able to get the answers they needed. And they did it using something called a Monte Carlo simulation. Now in fact, that's just what we've been doing with the random walk, the Monte Carlo simulation, and I'll come back to that in a minute. In general, the thing to think about, is we use simulation when we really can't get a closed form analytic solution. If you can put down a system of equations and easily solve it to get an answer, that's usually the right thing to do. But when we can't easily do that, the right thing to do is typically to fall back on simulation. Sometimes even when we can do it analytically, simulation has some advantages, as we'll be seeing as we go forward.

    What we're typically doing when we're simulating anything, is we're attempting to generate a sample of representative scenarios. Because an exhaustive enumeration of all possible states would be impossible. So again, if sometimes you can't solve it analytically, you can exhaustively enumerate the space and then see what's going on. But again, usually you can't. And so we look for a sample, and the key is, it's gotta be representative. Of what we would get in reality. We'll see an example of that shortly. So simulation attempts to build an experimental device that will act like the real system in important aspects. So I always think of a simulation as an experimental device. And every time I run it, I'm running an experiment designed to give me some information about the real world. These things are enormously popular, so if you were to get on Google and look at simulation. So for, example, we could Google simulation finance. You know, we see we get about 3,000,000 hits. We can do biology. We get twice as many hits, showing the relative importance in the world of biology and finance. For all you Course 7 majors, we should make sure that should compare biology to physics. Oh dear, we'll see that physics is even more important than biology. And I know we have a lot of Course 2 students in here, so let's try mechanical. Oh, lot of those, too. And if I did baseball, or football, we'd get more than any of them. Showing the real importance of things in the world.

    As we look at simulations, I want you to keep in mind that they are typically descriptive not prescriptive. So by that I mean, they describe a situation, they don't tell you what the answer is. So another way to think about this is, a simulation is not an optimization procedure. When we looked at optimization, that was prescriptive. We ran an optimization algorithm, and it gave us the best possible solution. A simulation typically doesn't give you this best possible solution, but if you give it the starting point, it will tell you the consequences of that. Now, can we use simulation to do optimization? Absolutely. For example, we can use it to do guess and check. Where we guess a, probably not to get a truly optimal solution, but to get a good solution, we can guess various possibilities, simulate them, and see. People do that all the time. If they want to see what's the right number of checkout counters at the supermarket, or what's the right airline schedule to use? They'll make a guess, they'll simulate it, and they'll say all right, this was better than this, so we'll choose it. All right, I'm going to stop here. Next time we're going to get more deeply into the actual stochastics, the probability distributions, and start understanding what's going on under the covers.
  • 33
    MIT Lecture 20: Monte Carlo Simulations, Estimating pi
    47:55
    Lecture 20: Monte Carlo simulations, estimating piInstructors: Prof. Eric Grimson, Prof. John Guttag View the complete course at: http://ocw.mit.edu/6-00F08License: Creative Commons BY-NC-SAMore information at http://ocw.mit.edu/termsMore courses at http://ocw.mit.edu

    LECTURE TRANSCRIPT

    PROFESSOR: All right, so today we're returning to simulations. And I'm going to do at first, a little bit more abstractly, and then come back to some details. So they're different ways to classify simulation models. The first is whether it's stochastic or deterministic. And the difference here is in a deterministic simulation, you should get the same result every time you run it. And there's a lot of uses we'll see for deterministic simulations. And then there's stochastic simulations, where the answer will differ from run to run because there's an element of randomness in it. So here if you run it again and again you get the same outcome every time, here you may not. So, for example, the problem set that's due today -- is that a stochastic or deterministic simulation? Somebody? Stochastic, exactly. And that's what we're going to focus on in this class, because one of the interesting questions we'll see about stochastic simulations is, how often do have to run them before you believe the answer? And that turns out to be a very important issue. You run it once, you get an answer, you can't take it to the bank. Because the next time you run it, you may get a completely different answer. So that will get us a little bit into the whole issue of statistical analysis.

    Another interesting dichotomy is static vs dynamic. We'll look at both, but will spend more time on dynamic models. So the issue -- it's not my phone. If it's your mother, you could feel free to take it, otherwise -- OK, no problem. Inevitable. In a dynamic situation, time plays a role. And you look at how things evolve over time. In a static simulation, there is no issue with time. We'll be looking at both, but most of the time we'll be focusing on dynamic ones. So an example of this kind of thing would be a queuing network model. This is one of the most popular and important kinds of dynamic simulations. Where you try and look at how queues, a fancy word for lines, evolve over time. So for example, people who are trying to decide how many lanes should be in a highway, or how far apart the exits should be, or what should the ratio of Fast Lane tolls to manually staffed tolls should be. All use queuing networks to try and answer that question. And we'll look at some examples of these later because they are very important. Particularly for things related to scheduling and planning.

    A third dichotomy is discrete vs continuous. Imagine, for example, trying to analyze the flow of traffic along the highway. One way to do it, is to try and have a simulation which models each vehicle. That would be a discrete simulation, because you've got different parts. Alternatively, you might decide to treat traffic as a flow, kind of like water flowing through things, where changes in the flow can be described by differential equations. That would lead to a continuous model. Another example is, a lot of effort has gone into analyzing the way blood flows through the human body. You can try and model it discretely, where you take each red blood cell, each white blood cell, and look at how they move, or simulate how they move. Or you could treat it continuously and say, well, we're just going to treat blood as a fluid, not made up of discrete components, and write some equations to model how that fluid goes through and then simulate that. In this course, we're going to be focusing mostly on discrete simulations.

    Now if we think about the random walk we looked at, indeed it was stochastic, dynamic, and discrete. The random walk was an example of what's called a Monte Carlo simulation. This term was coined there. Anyone know what that is? Anyone want to guess? It's the casino, in Monaco, in Monte Carlo. It was at one time, before there was even a Las Vegas, the most famous casino in the world certainly. Still one of the more opulent ones as you can see. And unlike Las Vegas, it's real opulence, as opposed to faux opulence. And this term Monte Carlo simulation, was coined by Ulam and Metropolis, two mathematicians, back in 1949, in reference to the fact that at Monte Carlos, people bet on roulette wheels, and cards on a table, games of chance, where there was randomness, and things are discrete, in some sense. And they decided, well, this is just like gambling, and so they called them Monte Carlos simulations.

    What Is it that makes this approach work? And, in some sense, I won't go into a lot of the math, but I would like to get some concepts across. This is an application of what are called inferential statistics. You have some sample size, some number of points, and from that you try to infer something more general. We always depend upon one property when we do this. And that property is that, a randomly chosen sample tends to exhibit the same properties as the population from which it is drawn. So you take a population of anything, red balls and black balls, or students, or steps, and at random draw some sample, and you assume that that sample has properties similar to the entire population. So if I were to go around this room and choose some random sample of you guys and write down your hair color, we would be assuming that the fraction of you with blonde hair in that sample would be the same as the fraction of you with blonde hair in the whole class. That's kind of what this means. And the same would be true of black hair, auburn hair, etc. So consider, for example, flipping a coin. And if I were to flip it some number of times, say 100 times, you might be able to, from the proportion of heads and tails, be able to infer whether or not the coin was fair. That is to say, half the times it would be heads, in half the times it would be that tails, or whether it was unfair, that it was somehow weighted, so that heads would come up more than tails. And you might say if we did this 100 times and looked at the results, then we could make a decision about what would happen in general when we looked at the coin.

    So let's look in an example of doing that. So I wrote a little program, it's on your handout, to flip a coin. So this looks like the simulations we looked at before. I've got flip trials, which says that the number of heads and tails is 0 for i in x range. What is x range? So normally you would have written, for i in range zero to num flips. What range does, is it creates a list, in this case from to 99 and goes through the list of one at a time. That's fine, but supposed num flips were a billion. Well, range would create a list with a billion numbers in it. Which would take a lot of space in the computer. And it's kind of wasted. What x range says is, don't bother creating the list just go through the, in this case, the numbers one at a time. So it's much more efficient than range. It will behave the same way as far as the answers you get, but it doesn't use as much space. And since some of the simulations we'll be doing will have lots of trials, or lots of flips, it's worth the trouble to use x range instead of range. Yeah? Pardon?

    STUDENT: Like, why would we ever use range instead of x range?

    PROFESSOR: No good reason, when dealing with numbers, unless you wanted to do something different with the list. But, there's no good reason. The right answer for the purposes of today is, no good reason. I typically use x range all the time if I'm thinking about it. It was just something that didn't seem worth introducing earlier in this semester. But good question. Certainly deserving of a piece of candy. All right, so for i in x range, coin is equal some random integer or 1. If coin is equal to then heads, else tails. Well, that's pretty easy. And then all I'm going to do here is go through and flip it a bunch of times, and we'll get some answer, and do some plots.

    So let's look at an example. We'll try -- we'll flip 100 coins, we'll do 100 trials and see what we get. Error in multi-line statement. All right, what have I done wrong here? Obviously did something by accident, edited something I did not intend edit. Anyone spot what I did wrong? Pardon? The parentheses. I typed where I didn't intend. Which line? Down at the bottom? Obviously my, here, yes, I deleted that. Thank you. All right, so we have a couple of figures here. Figure one, I'm showing a histogram. The number of trials on the y-axis and the difference between heads and tails , do I have more of one than the other on the x-axis. And so we what we could see out of my 100 trials, somewhere around 22 of them came out the same, close to the same. But way over here we've got some funny ones. 100 and there was a difference of 25. Pretty big difference.

    Another way to look at the same data, and I'm doing this just to show that there are different ways of looking at data, is here, what I've plotted is each trial, the percent difference. So out of 100 flips. And this is normalizing it, because if I flip a million coins, I might expect the difference to be pretty big in absolute terms, but maybe very small as a percentage of a million. And so here, we can again see that as these stochastic kinds of things, there's a pretty big difference, right? We've got one where it was over 25 percent, and several where it's zero. So the point here, we can see from this graph, that if I'd done only one trial and just assumed that was the answer as to whether my coin was weighted or not, I could really have fooled myself. So the the main point is that you need to be careful when you're doing these kinds of things. And this green line here is the mean. So it says on average, the difference was seven precent.

    Well suppose, maybe, instead of, flipping 100, I were to flip 1,000. Well, doesn't seem to want to notice it. One more try and then I'll just restart it, which is always the safest thing as we've discussed before. Well, we won't panic. Sometimes this helps. If not, here we go. So let's say we wanted to flip 1,000 coins. So now what do we think? Is the difference going to be bigger or smaller than when we flipped 100? Is the average difference between heads and tails bigger or smaller with 1,000 flips than with 100 flips? Well, the percentage will be smaller, but in absolute terms, it's probably going to be bigger, right? Because I've got more chances to stray. But we'll find out. So here we see that the mean difference is somewhere in the twenties, which was much higher than the mean difference for 100 flips. On the other hand, if we look at the percentage, we see it's much lower. Instead of seven percent, it's around two and a half percent in the main.

    There's something else interesting to observe in figure two, relative to when we looked at with 100 flips. What else is pretty interesting about the difference between these two figures, if you can remember the other one? Yeah?

    STUDENT: There are no zeros?

    PROFESSOR: There are no zeros. That's right, as it happens there were no zeros. Not so surprising that it didn't ever come out exactly 500, 500. What else? What was the biggest difference, percentage-wise, we saw last time? Over 25. So notice how much narrower the range is here. Instead of ranging from 2 to 25 or something like that, it ranges from to 7, or maybe 7 and a little. So, by flipping more coins, the experiment becomes more reproduce-able. I'm looks like the same because of scaling, but in fact the range is much narrower. Each experiment tends to give an answer closer to all the other experiments. That's a good thing. It should give you confidence that the answers are getting pretty close to right. That they're not bouncing all over the place. And if I were to flip a million coins, we would find the range would get very tight. So notice that even though there's similar information in the histogram and the plot, different things leap out at you, as you look at it.

    All right, we could ask a lot of other interesting questions about coins here. But, we'll come back to this in a minute and look at some other questions. I want to talk again a little bit more generally. It's kind of easy to think about running a simulation to predict the future. So in some sense, we look at this, and this predicts what might happen if I flipped 1,000 coins. That the most likely event would be that I'd have something under 10 in the difference between heads and tails, but that it's not terribly unlikely that I might have close to 70 as a difference. And if I ran more than 100 trials I'd see more, but this helps me predict what might happen.

    Now we don't always use simulations to predict what might happen. We sometimes actually use simulations to understand the current state of the world. So for example, if I told you that we are going to flip three coins, and I wanted you to predict the probability that all three would be either heads, or all three would be tails. Well, if you'd studied any probability, you could know how to do that. If you hadn't studied probability, you would say, well, that's OK, we have a simulation right here. Let's just do it. Here we go again. And so let's try it. Let's flip three coins, and let's do it 4,000 times here. Well, that's kind of hard to read. It's pretty dense. But we can see that the mean here is 50. And, this is a little easier to read. This tells us that, how many times will the difference, right, be zero 3,000 out of 4,000. Is that right? What do you think? Do you believe this? Have I done the right thing? three coins, 4,000 flips, how often should they all be heads, or how often should they all be tails? What does this tell us? It tells us one-fourth of the time they'll all be-- the difference between, wait a minute, how can the difference between -- something's wrong with my code, right? Because I only have two possible values. I hadn't expected this. I obviously messed something up. Pardon?

    STUDENT: It's right.

    PROFESSOR: It's right, because?

    STUDENT: Because you had an odd number of flips, and when you split them --

    PROFESSOR: Pardon?

    STUDENT: When you split an odd number --

    PROFESSOR: Exactly. So it is correct. And it gives us what we want. But now, let's think about a different situation. Anybody got a coin here? Anyone give me three coins? I can trust somebody, I hope. What a cheap -- anybody got silver dollars, would be preferable? All right, look at this. She's very carefully given me three pennies. She had big, big money in that purse, too, but she didn't want me to have it. All right, so I'm going to take these three pennies, jiggle them up, and now ask you, what's the probability that all three of them are heads? Anyone want to tell me? It's either or 1, right? And I can actually look at you and tell you exactly which it is. And you can't see which it is. So, how should you think about what the probability it? Well, you might as well assume that it's whatever this graph tells you it is. Because the fact that you don't have access to the information, means that you really might as well treat the present as if it's the future. That it's unknown. And so in fact we frequently, when there's data out there that we don't have access to, we use simulations and probabilities to estimate, make our best guess, about the current state of the world. And so, in fact, guessing the value of the current state, is really no different from predicting the value of a future state when you don't have the information.

    In general, all right, now, just to show that your precautions were unnecessary. Where was I? Right. In general, when we're trying to predict the future, or in this case, guess the present, we have to use information we already have to make our prediction or our best guess. So to do that, we have to always ask the question, is past behavior a good prediction of future behavior? So if I flip a coin 1,000 times and count up the heads and tails, is that a good prediction what will happen the next time? This is a step people often omit, in doing these predictions. See the recent meltdown of the financial system. Where people had lots of stochastic simulations predicting what the market would do, and they were all wrong, because they were all based upon assuming samples drawn from the past would predict the future. So, as we build these models, that's the question you always have to ask yourself. Is, in some sense, this true? Because usually what we're doing is, we're choosing a random sample from the past and hoping it predicts the future. And that is to say, is the population we have available the same has the one in the future. So it's easy to see how one might use these kinds of simulations to figure out things that are inherently stochastic. So for example, to predict a poker hand. What's the probability of my getting a full house when I draw this card from the deck? To predict the probability of coming up with a particular kind of poker hand. Is a full house more probable than a straight? Or not? Well, you can deal out lots of cards, and count them up, just as Ulam suggested for Solitaire. And that's often what we do.

    Interestingly enough though, we can use randomized techniques to get solutions to problems that are not inherently stochastic. And that's what I want to do now. So, consider for example, pi. Many of you have probably heard of this. For thousands of years, literally, people have known that there is a constant, pi, associated with circles such that pi times the radius squared equals the area. And they've known that pi times the diameter is equal to the circumference. So, back in the days of the Egyptian pharaohs, it was known that such a constant existed. In fact, it didn't acquire the name pi until the 18th century. And so they called it other things, but they knew it existed. And for thousands of years, people have speculated on what it's value was. Sometime around 1650 BC, the Egyptians said that pi was 3.16, something called the Rhind Papyrus, something they found. Many years later, about 1,000 years later, the Bible said pi was three. And I quote, this is describing the specifications for the Great Temple of Solomon. "He made a molten sea of 10 cubits from brim to brim, round in compass, and 5 cubit the height thereof, and a line of 30 cubits did compass it round about." So, all right, so what we've got here is, we've got everything we need to plug into these equations and solve for pi, and it comes out three. And it does this in more than one place in the Bible. I will not comment on the theological implications of this assertion. Sarah Palin might. And Mike Huckabee certainly would.

    The first theoretical calculation of pi was carried out by Archimedes, a great Greek mathematician from Syracuse, that was about somewhere around 250 BC. And he said that pi was somewhere between 223 divided by 71, and 22 divided by 7. This was amazingly profound. He knew he didn't know what the answer was, but he had a way to give an upper and a lower bound, and say it was somewhere between these two values. And in fact if you calculate it, the average of those two values is 3.1418. Not bad for the time. This was not by measurement, he actually had a very interesting way of calculating it. All right, so this is where it stood, for years and years, because of course people forgot the Rhind Papyrus, and they forgot Archimedes, and they believed the Bible, and so three was used for a long time. People sort of knew it wasn't right, but still. Then quite interestingly, Buffon and Laplace, two great French mathematicians, actually people had better estimates using Archimedes' methods long before they came along, proposed a way to do it using a simulation. Now, since Laplace lived between 1749 and 1827, it was not a computer simulation. So I'm going to show you, basically, the way that he proposed to do it. the basic idea was you take a square, assume that's a square, and you inscribe in it a quarter of a circle. So here, you have the radius of the square r. And then you get some person to, he used needles, but I'm going to use darts, to throw darts at the shape. And some number of the darts will land in the circle part, and some number of the darts will land out here, in the part of the square that's not inscribed by the circle. And then we can look at the ratio of the darts in the shaded area divided by the total number of darts in the square. And that's equal to the shaded area divided by the area of the square. The notion being, if they're landing at random in these places, the proportion here and not here will depend upon the relative areas. And that certainly makes sense. If this were half the area of the square, then you'd expect half the darts to land in here.

    And then as you can see in your handout, a little simple algebra can take this, plus pi r squared equals the area, you can solve for pi, and you can get that pi is equal to 4, and I'll write it, h, where h is the hit ratio, the number falling in here. So people sort of see why that should work intuitively? And that it's a very clever idea to use randomness to find a value that there's nothing random about. So we can now do the experiment. I need volunteers to throw darts. Come on. Come on up. I need more volunteers. I have a lot of darts. Anybody else? Anybody? All right, then since you're all in the front row, you get stuck. So now we'll try it. And you guys, we'll see how many of you hit in the circle, and how many of you hit there. Go ahead, on the count of 3, everybody throw. 1, 2, 3. Ohh! He did that on purpose. You'll notice Professor Grimson isn't here today, and that's because I told him he was going to have to hold the dart board. Well, what we see here is, we ignore the ones that missed all together. And we'll see that, truly, I'm assuming these are random throws. We have one here and two here. Well, your eyes will tell you that's the wrong ratio. Which suggests that having students throw darts is not the best way to solve this problem. And so you will see in your handout a computer simulation of it.

    So let's look at that. So this is find pi. So at the beginning of this code, by the way, it's not on your handout, is some magic. I got tired of looking at big numbers without commas separating the thousands places. You've see me in other lectures counting the number of zeros. What we have here is, that just tells it I have to have two versions, one for the Mac, and one for the PC. To set some variables that had to write integers, things in general, and I'm saying here, do it the way you would do it in the United States in English. And UTF8 is just an extended character code. Anyway, you don't need to learn anything about this magic, but it's just a handy little way to make the numbers easier to read. All right, so let's let's try and look at it. There's not much interesting to see here. I've done this little thing, format ints, that uses this magic to say grouping equals true, that means put a comma in the thousand places. But again, you can ignore all that. The interesting part, is that from Pylab I import star, import random in math. As some of you observed, the order these imports matters. I think I sent out an email yesterday explaining what was going on here. This was one of the things that I knew, and probably should've mentioned. But since I knew it, I thought everyone knew. Silly me. It was of course a dumb thing to think.

    And then I'm going to throw a bunch of darts. The other thing you'll notice is, throw darts has a parameter called should plot. And that's because when I throw a billion darts, I really don't want to try and take the time to plot a billion points. So let's first look at a little example. We'll try throwing 10,000 darts. And it gives me an estimated value of pi of 3.16. And what we'll see here, is that the number of darts thrown, the estimate changes, right? When I threw one dart, the estimate of pi was 4. I threw my second dart, it dropped all the way to 3. And then it bounced around a while, and then at the end, it starts to really stabilize around the true value. You'll notice, by the way, that what I've got here is a logarithmic x-axis. If you look at the code, you'll see I've told it to be semi log x. And that's because I wanted you to be able to see what was happening early on, where it was fluctuating. But out here it's kind of boring, because the fluctuations are so small. So that was a good way to do it. All right now. Do I think I have enough samples here? Well, I don't want you to cheat and look at the estimate and say no, you don't, because you know that's not the right answer. And, it's not even as good as Archimedes did. But how could you sort of look at the data, and get a sense that maybe this is not the right answer? Well, even at the end, if we look at it, it's still wiggling around a fair amount. We can zoom in. And it's bouncing up and down here. I'm in a region, but it's sort of makes us think that maybe it hasn't stabilized, right? You'd like it to not be moving very much. Now, by the way, the other thing we could've looked at, when we ran it, let's run it again, probably get a different answer by the way. Yeah, notice the different answer here. Turns out to be a better answer, but it's just an accident, right?

    Notice in the beginning it fluctuates wildly, and it fluctuates less wildly at the end. Why is that? And don't just say because it's close to right and it knows it. Why do the mathematics of this, in some sense, tell us it has to fluctuate less wildly at the end? Yes?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly, exactly right. If I've only thrown two darts, the third dart can have a big difference in the average value. But if I've thrown a million darts, the million and first can't matter very much. And what this tells me is, as I want ever more digits of precision, I have to run a lot more trials to get there. And that's often true, that simulations can get you in the neighborhood quickly, but the more precision you want, the number of steps grows quite quickly. Now, the fact that I got such different answers the two times I ran this suggests strongly that I shouldn't believe either answer. Right? So we need to do something else. So let's try something else. Let's try throwing a lot more darts here, and see what we get. Now if you look at my code, you'll see I'm printing intermediate values. Every million darts, I'm printing the value. And I did that because the first time I ran this on a big number, I was afraid I had an infinite loop and my program was not working. So I just said, all right, let's put a print statement in the loop, so I could see that it's making progress. And then I decided it was just kind of nice to look at it, to see what was going on here. So now you see that if I throw 10 million darts, I'm starting to get a much better estimate. You'll also see, as predicted, that as I get further out, the value of the estimate changes less and less with each million new darts, because it's a smaller fraction of the total darts. But it's getting a lot better. Still not quite there. Let's just see what happens, I can throw in another one. This is going to take a little while. So I can talk while it's running. Oops, what did I do here?

    So, it's going to keep on going and going and going. And then if we were to run it with this number of darts several times over, we would discover that we got answers that were very, very similar. From that we can take comfort, statistically, that we're really getting close to the same answer every time, so we've probably thrown enough darts to feel comfortable that we're doing what's statistically the right thing. And that there maybe isn't a lot of point in throwing more darts. Does that mean that we have the right answer? No, not necessarily, and that's what we're going to look at next week.PROFESSOR: All right, so today we're returning to simulations. And I'm going to do at first, a little bit more abstractly, and then come back to some details. So they're different ways to classify simulation models. The first is whether it's stochastic or deterministic. And the difference here is in a deterministic simulation, you should get the same result every time you run it. And there's a lot of uses we'll see for deterministic simulations. And then there's stochastic simulations, where the answer will differ from run to run because there's an element of randomness in it. So here if you run it again and again you get the same outcome every time, here you may not. So, for example, the problem set that's due today -- is that a stochastic or deterministic simulation? Somebody? Stochastic, exactly. And that's what we're going to focus on in this class, because one of the interesting questions we'll see about stochastic simulations is, how often do have to run them before you believe the answer? And that turns out to be a very important issue. You run it once, you get an answer, you can't take it to the bank. Because the next time you run it, you may get a completely different answer. So that will get us a little bit into the whole issue of statistical analysis.

    Another interesting dichotomy is static vs dynamic. We'll look at both, but will spend more time on dynamic models. So the issue -- it's not my phone. If it's your mother, you could feel free to take it, otherwise -- OK, no problem. Inevitable. In a dynamic situation, time plays a role. And you look at how things evolve over time. In a static simulation, there is no issue with time. We'll be looking at both, but most of the time we'll be focusing on dynamic ones. So an example of this kind of thing would be a queuing network model. This is one of the most popular and important kinds of dynamic simulations. Where you try and look at how queues, a fancy word for lines, evolve over time. So for example, people who are trying to decide how many lanes should be in a highway, or how far apart the exits should be, or what should the ratio of Fast Lane tolls to manually staffed tolls should be. All use queuing networks to try and answer that question. And we'll look at some examples of these later because they are very important. Particularly for things related to scheduling and planning.

    A third dichotomy is discrete vs continuous. Imagine, for example, trying to analyze the flow of traffic along the highway. One way to do it, is to try and have a simulation which models each vehicle. That would be a discrete simulation, because you've got different parts. Alternatively, you might decide to treat traffic as a flow, kind of like water flowing through things, where changes in the flow can be described by differential equations. That would lead to a continuous model. Another example is, a lot of effort has gone into analyzing the way blood flows through the human body. You can try and model it discretely, where you take each red blood cell, each white blood cell, and look at how they move, or simulate how they move. Or you could treat it continuously and say, well, we're just going to treat blood as a fluid, not made up of discrete components, and write some equations to model how that fluid goes through and then simulate that. In this course, we're going to be focusing mostly on discrete simulations.

    Now if we think about the random walk we looked at, indeed it was stochastic, dynamic, and discrete. The random walk was an example of what's called a Monte Carlo simulation. This term was coined there. Anyone know what that is? Anyone want to guess? It's the casino, in Monaco, in Monte Carlo. It was at one time, before there was even a Las Vegas, the most famous casino in the world certainly. Still one of the more opulent ones as you can see. And unlike Las Vegas, it's real opulence, as opposed to faux opulence. And this term Monte Carlo simulation, was coined by Ulam and Metropolis, two mathematicians, back in 1949, in reference to the fact that at Monte Carlos, people bet on roulette wheels, and cards on a table, games of chance, where there was randomness, and things are discrete, in some sense. And they decided, well, this is just like gambling, and so they called them Monte Carlos simulations.

    What Is it that makes this approach work? And, in some sense, I won't go into a lot of the math, but I would like to get some concepts across. This is an application of what are called inferential statistics. You have some sample size, some number of points, and from that you try to infer something more general. We always depend upon one property when we do this. And that property is that, a randomly chosen sample tends to exhibit the same properties as the population from which it is drawn. So you take a population of anything, red balls and black balls, or students, or steps, and at random draw some sample, and you assume that that sample has properties similar to the entire population. So if I were to go around this room and choose some random sample of you guys and write down your hair color, we would be assuming that the fraction of you with blonde hair in that sample would be the same as the fraction of you with blonde hair in the whole class. That's kind of what this means. And the same would be true of black hair, auburn hair, etc. So consider, for example, flipping a coin. And if I were to flip it some number of times, say 100 times, you might be able to, from the proportion of heads and tails, be able to infer whether or not the coin was fair. That is to say, half the times it would be heads, in half the times it would be that tails, or whether it was unfair, that it was somehow weighted, so that heads would come up more than tails. And you might say if we did this 100 times and looked at the results, then we could make a decision about what would happen in general when we looked at the coin.

    So let's look in an example of doing that. So I wrote a little program, it's on your handout, to flip a coin. So this looks like the simulations we looked at before. I've got flip trials, which says that the number of heads and tails is 0 for i in x range. What is x range? So normally you would have written, for i in range zero to num flips. What range does, is it creates a list, in this case from to 99 and goes through the list of one at a time. That's fine, but supposed num flips were a billion. Well, range would create a list with a billion numbers in it. Which would take a lot of space in the computer. And it's kind of wasted. What x range says is, don't bother creating the list just go through the, in this case, the numbers one at a time. So it's much more efficient than range. It will behave the same way as far as the answers you get, but it doesn't use as much space. And since some of the simulations we'll be doing will have lots of trials, or lots of flips, it's worth the trouble to use x range instead of range. Yeah? Pardon?

    STUDENT: Like, why would we ever use range instead of x range?

    PROFESSOR: No good reason, when dealing with numbers, unless you wanted to do something different with the list. But, there's no good reason. The right answer for the purposes of today is, no good reason. I typically use x range all the time if I'm thinking about it. It was just something that didn't seem worth introducing earlier in this semester. But good question. Certainly deserving of a piece of candy. All right, so for i in x range, coin is equal some random integer or 1. If coin is equal to then heads, else tails. Well, that's pretty easy. And then all I'm going to do here is go through and flip it a bunch of times, and we'll get some answer, and do some plots.

    So let's look at an example. We'll try -- we'll flip 100 coins, we'll do 100 trials and see what we get. Error in multi-line statement. All right, what have I done wrong here? Obviously did something by accident, edited something I did not intend edit. Anyone spot what I did wrong? Pardon? The parentheses. I typed where I didn't intend. Which line? Down at the bottom? Obviously my, here, yes, I deleted that. Thank you. All right, so we have a couple of figures here. Figure one, I'm showing a histogram. The number of trials on the y-axis and the difference between heads and tails , do I have more of one than the other on the x-axis. And so we what we could see out of my 100 trials, somewhere around 22 of them came out the same, close to the same. But way over here we've got some funny ones. 100 and there was a difference of 25. Pretty big difference.

    Another way to look at the same data, and I'm doing this just to show that there are different ways of looking at data, is here, what I've plotted is each trial, the percent difference. So out of 100 flips. And this is normalizing it, because if I flip a million coins, I might expect the difference to be pretty big in absolute terms, but maybe very small as a percentage of a million. And so here, we can again see that as these stochastic kinds of things, there's a pretty big difference, right? We've got one where it was over 25 percent, and several where it's zero. So the point here, we can see from this graph, that if I'd done only one trial and just assumed that was the answer as to whether my coin was weighted or not, I could really have fooled myself. So the the main point is that you need to be careful when you're doing these kinds of things. And this green line here is the mean. So it says on average, the difference was seven precent.

    Well suppose, maybe, instead of, flipping 100, I were to flip 1,000. Well, doesn't seem to want to notice it. One more try and then I'll just restart it, which is always the safest thing as we've discussed before. Well, we won't panic. Sometimes this helps. If not, here we go. So let's say we wanted to flip 1,000 coins. So now what do we think? Is the difference going to be bigger or smaller than when we flipped 100? Is the average difference between heads and tails bigger or smaller with 1,000 flips than with 100 flips? Well, the percentage will be smaller, but in absolute terms, it's probably going to be bigger, right? Because I've got more chances to stray. But we'll find out. So here we see that the mean difference is somewhere in the twenties, which was much higher than the mean difference for 100 flips. On the other hand, if we look at the percentage, we see it's much lower. Instead of seven percent, it's around two and a half percent in the main.

    There's something else interesting to observe in figure two, relative to when we looked at with 100 flips. What else is pretty interesting about the difference between these two figures, if you can remember the other one? Yeah?

    STUDENT: There are no zeros?

    PROFESSOR: There are no zeros. That's right, as it happens there were no zeros. Not so surprising that it didn't ever come out exactly 500, 500. What else? What was the biggest difference, percentage-wise, we saw last time? Over 25. So notice how much narrower the range is here. Instead of ranging from 2 to 25 or something like that, it ranges from to 7, or maybe 7 and a little. So, by flipping more coins, the experiment becomes more reproduce-able. I'm looks like the same because of scaling, but in fact the range is much narrower. Each experiment tends to give an answer closer to all the other experiments. That's a good thing. It should give you confidence that the answers are getting pretty close to right. That they're not bouncing all over the place. And if I were to flip a million coins, we would find the range would get very tight. So notice that even though there's similar information in the histogram and the plot, different things leap out at you, as you look at it.

    All right, we could ask a lot of other interesting questions about coins here. But, we'll come back to this in a minute and look at some other questions. I want to talk again a little bit more generally. It's kind of easy to think about running a simulation to predict the future. So in some sense, we look at this, and this predicts what might happen if I flipped 1,000 coins. That the most likely event would be that I'd have something under 10 in the difference between heads and tails, but that it's not terribly unlikely that I might have close to 70 as a difference. And if I ran more than 100 trials I'd see more, but this helps me predict what might happen.

    Now we don't always use simulations to predict what might happen. We sometimes actually use simulations to understand the current state of the world. So for example, if I told you that we are going to flip three coins, and I wanted you to predict the probability that all three would be either heads, or all three would be tails. Well, if you'd studied any probability, you could know how to do that. If you hadn't studied probability, you would say, well, that's OK, we have a simulation right here. Let's just do it. Here we go again. And so let's try it. Let's flip three coins, and let's do it 4,000 times here. Well, that's kind of hard to read. It's pretty dense. But we can see that the mean here is 50. And, this is a little easier to read. This tells us that, how many times will the difference, right, be zero 3,000 out of 4,000. Is that right? What do you think? Do you believe this? Have I done the right thing? three coins, 4,000 flips, how often should they all be heads, or how often should they all be tails? What does this tell us? It tells us one-fourth of the time they'll all be-- the difference between, wait a minute, how can the difference between -- something's wrong with my code, right? Because I only have two possible values. I hadn't expected this. I obviously messed something up. Pardon?

    STUDENT: It's right.

    PROFESSOR: It's right, because?

    STUDENT: Because you had an odd number of flips, and when you split them --

    PROFESSOR: Pardon?

    STUDENT: When you split an odd number --

    PROFESSOR: Exactly. So it is correct. And it gives us what we want. But now, let's think about a different situation. Anybody got a coin here? Anyone give me three coins? I can trust somebody, I hope. What a cheap -- anybody got silver dollars, would be preferable? All right, look at this. She's very carefully given me three pennies. She had big, big money in that purse, too, but she didn't want me to have it. All right, so I'm going to take these three pennies, jiggle them up, and now ask you, what's the probability that all three of them are heads? Anyone want to tell me? It's either or 1, right? And I can actually look at you and tell you exactly which it is. And you can't see which it is. So, how should you think about what the probability it? Well, you might as well assume that it's whatever this graph tells you it is. Because the fact that you don't have access to the information, means that you really might as well treat the present as if it's the future. That it's unknown. And so in fact we frequently, when there's data out there that we don't have access to, we use simulations and probabilities to estimate, make our best guess, about the current state of the world. And so, in fact, guessing the value of the current state, is really no different from predicting the value of a future state when you don't have the information.

    In general, all right, now, just to show that your precautions were unnecessary. Where was I? Right. In general, when we're trying to predict the future, or in this case, guess the present, we have to use information we already have to make our prediction or our best guess. So to do that, we have to always ask the question, is past behavior a good prediction of future behavior? So if I flip a coin 1,000 times and count up the heads and tails, is that a good prediction what will happen the next time? This is a step people often omit, in doing these predictions. See the recent meltdown of the financial system. Where people had lots of stochastic simulations predicting what the market would do, and they were all wrong, because they were all based upon assuming samples drawn from the past would predict the future. So, as we build these models, that's the question you always have to ask yourself. Is, in some sense, this true? Because usually what we're doing is, we're choosing a random sample from the past and hoping it predicts the future. And that is to say, is the population we have available the same has the one in the future. So it's easy to see how one might use these kinds of simulations to figure out things that are inherently stochastic. So for example, to predict a poker hand. What's the probability of my getting a full house when I draw this card from the deck? To predict the probability of coming up with a particular kind of poker hand. Is a full house more probable than a straight? Or not? Well, you can deal out lots of cards, and count them up, just as Ulam suggested for Solitaire. And that's often what we do.

    Interestingly enough though, we can use randomized techniques to get solutions to problems that are not inherently stochastic. And that's what I want to do now. So, consider for example, pi. Many of you have probably heard of this. For thousands of years, literally, people have known that there is a constant, pi, associated with circles such that pi times the radius squared equals the area. And they've known that pi times the diameter is equal to the circumference. So, back in the days of the Egyptian pharaohs, it was known that such a constant existed. In fact, it didn't acquire the name pi until the 18th century. And so they called it other things, but they knew it existed. And for thousands of years, people have speculated on what it's value was. Sometime around 1650 BC, the Egyptians said that pi was 3.16, something called the Rhind Papyrus, something they found. Many years later, about 1,000 years later, the Bible said pi was three. And I quote, this is describing the specifications for the Great Temple of Solomon. "He made a molten sea of 10 cubits from brim to brim, round in compass, and 5 cubit the height thereof, and a line of 30 cubits did compass it round about." So, all right, so what we've got here is, we've got everything we need to plug into these equations and solve for pi, and it comes out three. And it does this in more than one place in the Bible. I will not comment on the theological implications of this assertion. Sarah Palin might. And Mike Huckabee certainly would.

    The first theoretical calculation of pi was carried out by Archimedes, a great Greek mathematician from Syracuse, that was about somewhere around 250 BC. And he said that pi was somewhere between 223 divided by 71, and 22 divided by 7. This was amazingly profound. He knew he didn't know what the answer was, but he had a way to give an upper and a lower bound, and say it was somewhere between these two values. And in fact if you calculate it, the average of those two values is 3.1418. Not bad for the time. This was not by measurement, he actually had a very interesting way of calculating it. All right, so this is where it stood, for years and years, because of course people forgot the Rhind Papyrus, and they forgot Archimedes, and they believed the Bible, and so three was used for a long time. People sort of knew it wasn't right, but still. Then quite interestingly, Buffon and Laplace, two great French mathematicians, actually people had better estimates using Archimedes' methods long before they came along, proposed a way to do it using a simulation. Now, since Laplace lived between 1749 and 1827, it was not a computer simulation. So I'm going to show you, basically, the way that he proposed to do it. the basic idea was you take a square, assume that's a square, and you inscribe in it a quarter of a circle. So here, you have the radius of the square r. And then you get some person to, he used needles, but I'm going to use darts, to throw darts at the shape. And some number of the darts will land in the circle part, and some number of the darts will land out here, in the part of the square that's not inscribed by the circle. And then we can look at the ratio of the darts in the shaded area divided by the total number of darts in the square. And that's equal to the shaded area divided by the area of the square. The notion being, if they're landing at random in these places, the proportion here and not here will depend upon the relative areas. And that certainly makes sense. If this were half the area of the square, then you'd expect half the darts to land in here.

    And then as you can see in your handout, a little simple algebra can take this, plus pi r squared equals the area, you can solve for pi, and you can get that pi is equal to 4, and I'll write it, h, where h is the hit ratio, the number falling in here. So people sort of see why that should work intuitively? And that it's a very clever idea to use randomness to find a value that there's nothing random about. So we can now do the experiment. I need volunteers to throw darts. Come on. Come on up. I need more volunteers. I have a lot of darts. Anybody else? Anybody? All right, then since you're all in the front row, you get stuck. So now we'll try it. And you guys, we'll see how many of you hit in the circle, and how many of you hit there. Go ahead, on the count of 3, everybody throw. 1, 2, 3. Ohh! He did that on purpose. You'll notice Professor Grimson isn't here today, and that's because I told him he was going to have to hold the dart board. Well, what we see here is, we ignore the ones that missed all together. And we'll see that, truly, I'm assuming these are random throws. We have one here and two here. Well, your eyes will tell you that's the wrong ratio. Which suggests that having students throw darts is not the best way to solve this problem. And so you will see in your handout a computer simulation of it.

    So let's look at that. So this is find pi. So at the beginning of this code, by the way, it's not on your handout, is some magic. I got tired of looking at big numbers without commas separating the thousands places. You've see me in other lectures counting the number of zeros. What we have here is, that just tells it I have to have two versions, one for the Mac, and one for the PC. To set some variables that had to write integers, things in general, and I'm saying here, do it the way you would do it in the United States in English. And UTF8 is just an extended character code. Anyway, you don't need to learn anything about this magic, but it's just a handy little way to make the numbers easier to read. All right, so let's let's try and look at it. There's not much interesting to see here. I've done this little thing, format ints, that uses this magic to say grouping equals true, that means put a comma in the thousand places. But again, you can ignore all that. The interesting part, is that from Pylab I import star, import random in math. As some of you observed, the order these imports matters. I think I sent out an email yesterday explaining what was going on here. This was one of the things that I knew, and probably should've mentioned. But since I knew it, I thought everyone knew. Silly me. It was of course a dumb thing to think.

    And then I'm going to throw a bunch of darts. The other thing you'll notice is, throw darts has a parameter called should plot. And that's because when I throw a billion darts, I really don't want to try and take the time to plot a billion points. So let's first look at a little example. We'll try throwing 10,000 darts. And it gives me an estimated value of pi of 3.16. And what we'll see here, is that the number of darts thrown, the estimate changes, right? When I threw one dart, the estimate of pi was 4. I threw my second dart, it dropped all the way to 3. And then it bounced around a while, and then at the end, it starts to really stabilize around the true value. You'll notice, by the way, that what I've got here is a logarithmic x-axis. If you look at the code, you'll see I've told it to be semi log x. And that's because I wanted you to be able to see what was happening early on, where it was fluctuating. But out here it's kind of boring, because the fluctuations are so small. So that was a good way to do it. All right now. Do I think I have enough samples here? Well, I don't want you to cheat and look at the estimate and say no, you don't, because you know that's not the right answer. And, it's not even as good as Archimedes did. But how could you sort of look at the data, and get a sense that maybe this is not the right answer? Well, even at the end, if we look at it, it's still wiggling around a fair amount. We can zoom in. And it's bouncing up and down here. I'm in a region, but it's sort of makes us think that maybe it hasn't stabilized, right? You'd like it to not be moving very much. Now, by the way, the other thing we could've looked at, when we ran it, let's run it again, probably get a different answer by the way. Yeah, notice the different answer here. Turns out to be a better answer, but it's just an accident, right?

    Notice in the beginning it fluctuates wildly, and it fluctuates less wildly at the end. Why is that? And don't just say because it's close to right and it knows it. Why do the mathematics of this, in some sense, tell us it has to fluctuate less wildly at the end? Yes?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly, exactly right. If I've only thrown two darts, the third dart can have a big difference in the average value. But if I've thrown a million darts, the million and first can't matter very much. And what this tells me is, as I want ever more digits of precision, I have to run a lot more trials to get there. And that's often true, that simulations can get you in the neighborhood quickly, but the more precision you want, the number of steps grows quite quickly. Now, the fact that I got such different answers the two times I ran this suggests strongly that I shouldn't believe either answer. Right? So we need to do something else. So let's try something else. Let's try throwing a lot more darts here, and see what we get. Now if you look at my code, you'll see I'm printing intermediate values. Every million darts, I'm printing the value. And I did that because the first time I ran this on a big number, I was afraid I had an infinite loop and my program was not working. So I just said, all right, let's put a print statement in the loop, so I could see that it's making progress. And then I decided it was just kind of nice to look at it, to see what was going on here. So now you see that if I throw 10 million darts, I'm starting to get a much better estimate. You'll also see, as predicted, that as I get further out, the value of the estimate changes less and less with each million new darts, because it's a smaller fraction of the total darts. But it's getting a lot better. Still not quite there. Let's just see what happens, I can throw in another one. This is going to take a little while. So I can talk while it's running. Oops, what did I do here?

    So, it's going to keep on going and going and going. And then if we were to run it with this number of darts several times over, we would discover that we got answers that were very, very similar. From that we can take comfort, statistically, that we're really getting close to the same answer every time, so we've probably thrown enough darts to feel comfortable that we're doing what's statistically the right thing. And that there maybe isn't a lot of point in throwing more darts. Does that mean that we have the right answer? No, not necessarily, and that's what we're going to look at next week.
  • 34
    Assignment 12: Simulating virus population dynamics
    12 pages
  • 35
    MIT Lecture 21: Validating simulation results, curve fitting, linear regression
    53:48
    Topics covered: Validating simulation results, curve fitting, linear regression

    LECTURE TRANSCRIPT

    PROFESSOR: So let's start. I have written a number on the board here. Anyone want to speculate what that number represents? Well, you may recall at the end of the last lecture, we were simulating pi, and I started up running it with a billion darts. And when it finally terminated, this was the estimate of pi it gave me with a billion. Not bad, not quite perfect, but still pretty good. In fact when I later ran it with 10 billion darts, which took a rather long time to run, didn't do much better. So it's converging very slowly now near the end.

    When we use an algorithm like that one to perform a Monte Carlo simulation, we're trusting, as I said, that fate will give us an unbiased sample, a sample that would be representative of true random throws. And, indeed in this case, that's a pretty good assumption. The random number generator is not truly random, it's what's called pseudo-random, in that if you start it with the same initial conditions, it will give you the same results. But it's close enough for, at least for government work, and other useful projects. We do have to think about the question, how many samples should we run? Was a billion darts enough? Now since we sort of all started knowing what pi was, we could look at it and say, yeah, pretty good. But suppose we had no clue about the actual value of pi. We still have to think about the question of how many samples? And also, how accurate do we believe our result is, given the number of samples? As you might guess, these two questions are closely related. That, if we know in advance how much accuracy we want, we can sometimes use that to calculate how many samples we need.

    But there's still always the issue. It's never possible to achieve perfect accuracy through sampling. Unless you sample the entire population. No matter how many samples you take, you can never be sure that the sample set is typical until you've checked every last element. So if I went around MIT and sampled 100 students to try and, for example, guess the fraction of students at MIT who are of Chinese descent. Maybe 100 students would be enough, but maybe I would get unlucky and draw the wrong 100. In the sense of, by accident, 100 Chinese descent, or 100 non-Chinese descent, which would give me the wrong answer. And there would be no way I could be sure that I had not drawn a biased sample, unless I really did have the whole population to look at.

    So we can never know that our estimate is correct. Now maybe I took a billion darts, and for some reason got really unlucky and they all ended up inside or outside the circle. But what we can know, is how likely it is that our answer is correct, given the assumptions. And that's the topic we'll spend the next few lectures on, at least one of the topics. It's saying, how can we know how likely it is that our answer is good. But it's always given some set of assumptions, and we have to worry a lot about those assumptions. Now in the case of our pi example, our assumption was that the random number generator was indeed giving us random numbers in the interval to 1. So that was our underlying assumption. Then using that, we looked at a plot, and we saw that after time the answer wasn't changing very much. And we use that to say, OK, it looks like we're actually converging on an answer. And then I ran it again, with another trial, and it converged again at the same place. And the fact that that happened several times led me to at least have some reason to believe that I was actually finding a good approximation of pi.

    That's a good thing to do. It's a necessary thing to do. But it is not sufficient. Because errors can creep into many places. So that kind of technique, and in fact, almost all statistical techniques, are good at establishing, in some sense, the reproduce-ability of the result, and that it is statistically valid, and that there's no error, for example, in the way I'm generating the numbers. Or I didn't get very unlucky. However, they're other places other than bad luck where errors can creep in.

    So let's look at an example here. I've taken the algorithm we looked at last time for finding pi, and I've made a change. You'll remember that we were before using 4 as our multiplier, and here what I've done is, just gone in and replaced 4 by 2. Assuming that I made a programming error. Now let's see what happens when we run it. Well, a bad thing has happened. Sure enough, we ran it and it converged, started to converge, and if I ran 100 trials each one would converge at roughly the same place. Any statistical test I would do, would say that my statistics are sound, I've chosen enough samples, and for some accuracy, it's converting. Everything is perfect, except for what? It's the wrong answer. The moral here, is that just because an answer is statistically valid, does not mean it's the right answer.

    And that's really important to understand, because you see this, and we'll see more examples later, not today, but after Thanksgiving, comes up all the time in the newspapers, in scientific articles, where people do a million tests, do all the statistics right, say here's the answer, and it turns out to be completely wrong. And that's because it was some underlying assumption that went into the decision, that was not true. So here, the assumption is, that I've done my algebra right for computing pi based upon where the darts land. And it turns out, if I put 2 here, my algebra is wrong. Now how could I discover this? Since I've already told you no statistical test is going to help me. What's the obvious thing I should be doing when I get this answer? Somebody? Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Exactly. Checking against reality. I started with the notion that pi had some relation to the area of a circle. So I could use this value of pi, draw a circle with a radius. Do my best to measure the area. I wouldn't need to get a very good, accurate measurement, and I would say, whoa, this isn't even close. And that would tell me I have a problem. So the moral here is, to check results against physical reality. So for example, the current problem set, you're doing a simulation about what happens to viruses when drugs are applied. If you were doing this for a pharmaceutical company, in addition to the simulation, you'd want to run some real experiments. And make sure that things matched. OK, what this suggests, is that we often use simulation, and other computational techniques, to try and model the real world, or the physical world, in which we all live. And we can use data to do that. I now want to go through another set of examples, and we're going to look at the interplay of three things: what happens when you have data, say from measurements, and models that at least claim to explain the data. And then, consequences that follow from the models. This is often the way science works, its the way engineering works, we have some measurements, we have a theory that explains the measurements, and then we write software to explore the consequences of that theory. Including, is it plausible that it's really true?

    So I want to start, as an example, with a classic chosen from 8.01. So I presume, everyone here has taken 8.01? Or in 8.01? Anyone here who's not had an experience with 801? All right, well. I hope you know about springs, because we're going to talk about springs. So if you think about it, I'm now just talking not about springs that have water in them, but springs that you compress, you know, and expand, and things like that. And there's typically something called the spring constant that tells us how stiff the spring is, how much energy it takes to compress this spring. Or equivalently, how much pop the spring has when you're no longer holding it down. Some springs are easy to stretch, they have a small spring constant. Some strings, for example, the ones that hold up an automobile, suspension, are much harder to stretch and compress.

    There's a theory about them called Hooke's Law. And it's quite simple. Force, the amount of force exerted by a spring, is equal to minus some constant times the distance you have compressed the spring. It's minus, because the force is exerted in an opposite direction, trying to spring up. So for example, we could look at it this way. We've got a spring, excuse my art here. And we put some weight on the spring, which has therefore compressed it a little bit. And the spring is exerting some upward force. And the amount of force it's exerting is proportional to the distance x. So, if we believe Hooke's Law, and I give you a spring, how can we find out what this constant is? Well, we can do it by putting a weight on top of the spring. It will compress the spring a certain amount, and then the spring will stop moving. Now gravity would normally have had this weight go all the way down to the bottom, if there was no spring. So clearly the spring is exerting some force in the upward direction, to keep that mass from going down to the table, right? So we know what that force is there. If we compress the spring to a bunch of different distances, by putting, say, different size weights on it, we can then solve for the spring constant, just the way, before, we solved for pi.

    So it just so happens, not quite by accident, that I've got some data from a spring. So let's look at it. So here's some data taken from measuring a spring. This is distance and force, force computed from the mass, basically, right? Because we know that these have to be in balance. And I'm not going to ask you to in your head estimate the constant from these, but what you'll see is, the format is, there's a distance, and then a colon, and then the force. Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Ok, right, yes, thank you. All right, want to repeat that more loudly for everyone?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Right, right, because the x in the equation -- right, here we're getting an equilibrium. OK, so let's look at what happens when we try and examine this. We'll look at spring dot pi. So it's pretty simple. First thing is, I've got a function that reads in the data and parses it. You've all done more complicated parsing of data files than this. So I won't belabor the details. I called it get data rather than get spring data, because I'm going to use the same thing for a lot of other kinds of data. And the only thing I want you to notice, is that it's returning a pair of arrays. OK, not lists. The usual thing is, I'm building them up using lists, because lists have append and arrays don't, and then I'm converting them to arrays so I can do matrix kinds of operations on them.

    So I'll get the distances and the forces. And then I'm just going to plot them, and we'll see what they look like. So let's do that. There they are. Now, if you believe Hooke's Law, you could look at this data, and maybe you wouldn't like it. Because Hooke's Law implies that, in fact, these points should lie in a straight line, right? If I just plug in values here, what am I going to get? A straight line, right? I'm just multiplying k times x. But I don't have a straight line, I have a little scatter of points, kind of it looks like a straight line, but it's not. And why do you think that's true? What's going on here? What could cause this line not to be straight? Have any you ever done a physics experiment? And when you did it, did your results actually match the theory that your high school teacher, say, explained to you. No, and why not. Yeah, you have various kinds of experimental or measurement error, right? Because, when you're doing these experiments, at least I'm not perfect, and I suspect at least most of you are not perfect, you get mistakes. A little bit of error creeps in inevitably. And so, when we acquired this data, sure enough there was measurement error. And so the points are scattered around.

    This is something to be expected. Real data almost never matches the theory precisely. Because there usually is some sort of experimental error that creeps into things. So what should we do about that? Well, what usually people do, when they think about this, is they would look at this data and say, well, let me fit a line to this. Somehow, say, what would be the line that best approximates these points? And then the slope of that line would give me the spring constant.

    So that raises the next question, what do I mean by finding a line that best fits these points? How do we, fit, in this case, a line, to the data? First of all, I should ask the question, why did I say let's fit a line? Maybe I should have said, let's fit a parabola, or let's fit a circle? Why should I had said let's fit a line. Yeah?

    STUDENT: [INAUDIBLE]

    PROFESSOR: Well, how do I know that the plot is a linear function? Pardon? Well, so, two things. One is, I had a theory. You know, I had up there a model, and my model suggested that I expected it to be linear. And so if I'm testing my model, I should and fit a line, my theory, if you will. But also when I look at it, it looks kind of like a line. So you know, if I looked at it, and it didn't look like a line, I might have said, well, my model must be badly broken. So let's try and see if we can fit it. Whenever we try and fit something, we need some sort of an objective function that captures the goodness of a fit. I'm trying to find, this is an optimization problem of the sort that we've looked at before. I'm trying to find a line that optimizes some objective function. So a very simple objective function here, is called the least squares fit. I want to find the line that minimizes the sum of observation sub i, the i'th data point I have, minus what the line, the model, predicts that point should have been, and then I'll square it. So I want to minimize this value. I want to find the line that gives me the smallest value for this.

    Why do you think I'm squaring the difference? What would happen if I didn't square the difference? Yeah? Positive and negative errors might cancel each other out. And in judging the quality of the fit, I don't really care deeply -- you're going to get very fat the way you're collecting candy here -- I don't care deeply whether the error is, which side, it is, just that it's wrong. And so by squaring it, it's kind of like taking the absolute value of the error, among other things. All right, so if we look at our example here, what would this be? I want to minimize, want to find a line that minimizes it. So how do I do that? I could easily do it using successive approximation, right? I could choose a line, basically what I am, is I'm choosing a slope, here, right? And, I could, just like Newton Raphson, do successive approximation for awhile, and get the best fit. That's one way to do the optimization. It turns out that for this particular optimization there's something more efficient. You can actually, there is a closed form way of attacking this, and I could explain that, but in fact, I'll explain something even better. It's built into Pylab.

    So Pylab has a function built-in called polyfit. Which, given a set of points, finds the polynomial that gives you the best least squares approximation to those points. It's called polynomial because it isn't necessarily going to be first order, that is to say, a line. It can find polynomials of arbitrary degree. So let's look at the example here, we'll see how it works. So let me uncomment it. So I'm going to get k and b equals Pylab dot polyfit here. What it's going to do is, think about a polynomial. I give you a polynomial of degree one, you have all learned that it's a x plus b, b is the constant, and x is the single variable. And so I multiply a by x and I add b to it, and as I vary x I get new values. And so polyfit, in this case, will take the set of points defined by these two arrays and return me a value for a and a value for b. Now here I've assigned a to k, but don't worry about that. And then, I'm gonna now generate the predictions that I would get from this k and b, and plot those.

    So let's look at it. So here it said the k is 31.475, etc., and it's plotted the line that it's found. Or I've plotted the line. You'll note, a lot of the points don't lie on the line, in fact, most of the points don't lie on the line. But it's asserting that this is the best it can do with the line. And there's some points, for example, up here, that are kind of outliers, that are pretty far from the line. But it has minimized the error, if you will, for all of the points it has. That's quite different from, say, finding the line that touches the most points, right? It's minimizing the sum of the errors. Now, given that I was just looking for a constant to start with, why did I bother even plotting the data? I happen to have known before I did this that polyfit existed, and what I was really looking for was this line. So maybe I should have just done the polyfit and said here's k and I'm done. Would that have been a good idea? Yeah?

    STUDENT: You can't know without seeing the actual data how well it's actually fitting it.

    PROFESSOR: Right. Exactly right. That says, well how would I know that it was fitting it badly or well, and in fact, how would I know that my notion of the model is sound, or that my experiment isn't completely broken? So always, I think, always look at the real data. Don't just, I've seen too many papers where people show me the curve that fits the data, and don't show me the data, and it always makes me very nervous. So always look at the data, as well as however you're choosing to fit it.

    As an example of that, let's look at another set of inputs. This is not a spring. It's the same get data function as before, ignore that thing at the top. I'm going to analyze it and we'll look at it. So here I'm plotting the speed of something over time. So I plotted it, and I've done a least squares fit using polyfit just as before to get a line, and I put the line vs. the data, and here I'm a little suspicious. Right, I fit a line, but when I look at it, I don't think it's a real good fit for the data. Somehow modeling this data as a line is probably not right. A linear model is not good for this data. This data is derived from something, a more complex process. So take a look at it, and tell me what order were calling of polynomial do you think might fit this data? What shape does this look like to you? Pardon?

    STUDENT: Quadratic.

    PROFESSOR: Quadratic, because the shape is a what? It's a parabola. Well, I don't know if I dare try this one all the way to the back. Ooh, at least I didn't hurt anybody. All right, fortunately it's just as easy to fit a ravel parabola as a line. So let's look down here. I've done the same thing, but instead of passing it one, as I did up here as the argument, I'm passing it two. Saying, instead of fitting a polynomial of degree one, fit a polynomial of degree two. And now let's see what it looks like. Well, my eyes tell me this is a much better fit than the line. So again, that's why I wanted to see the scatter plot, so that I could at least look at it with my eyes, and say, yeah, this looks like a better fit. All right, any question about what's going on here?

    What we've been looking at is something called linear regression. It's called linear because the relationship of the dependent variable y to the independent variables is assumed to be a linear function of the parameters. It's not because it has to be a linear function of the value of x, OK? Because as you can see, we're not getting a line, we're getting a parabola. Don't worry about the details, the point I want to make is, people sometimes see the word linear regression and think it can only be used to find lines. It's not so. So when, for example, we did the quadratic, what we had is y equals a x squared plus b x plus c. The graph vs. x will not be a straight line, right, because I'm squaring x. But it is, just about, in this case, the single variable x. Now, when I looked at this, I said, all right, it's clear that the yellow curve is a better fit than the red. It's a red line. But that was a pretty informal statement.

    I can actually look at this much more formally. And we're going to look at something that's the statisticians call r squared. Which in the case of a linear regression is the coefficient of determination. Now, this is a big fancy word for something that's actually pretty simple. So what r squared its going to be, and this is on your handout, is 1 minus e e over d v. So e e is going to be the errors in the estimation. So I've got some estimated values, some predicted values, if you will, given to me by the model, either the line or the parabola in this case. And I've got some real values, corresponding to each of those points, and I can look at the difference between the 2 And that will tell me how much difference there is between the estimated data and the, well, between the predicted data and the measured data, in this case. And then I want to divide that by the variance in the measured data. The data variance. How broadly scattered the measured points are. And I'll do that by comparing the mean of the measured data, to the measured data. So I get the average value of the measured data, and I look at how different the points I measure are.

    So I just want to give to you, informally, because I really don't care if you understand all the math. What I do want you to understand, when someone tells you, here's the r squared value, is, informally what it really is saying. It's attempting to capture the proportion of the response variation explained by the variables in the model. In this case, x. So you'll have some amount of variation that is explained by changing the values of the variables. So if, actually, I'm going to give an example and then come back to it more informally. So if, for example, r squared were to equal 0.9, that would mean that approximately 90 percent of the variation in the variables can be explained by the model. OK, so we have some amount of variation in the measured data, and if r squared is 0.9, it says that 90 percent can be explained by the models, and the other 10 percent cannot. Now, that other 10 percent could be experimental error, or it could be that, in fact, you need more variables in the model. That there are what are called lurking variables. I love this term. A lurking variable is something that actually effects the result, but is not reflected in the model.

    As we'll see a little bit later, this is a very important thing to worry about, when you're looking at experimental data and you're building models. So we see this, for example, in the medical literature, that they will do some experiment, and they'll say that this drug explains x, or has this affect. And the variables they are looking at are, say, the disease the patient has, and the age of the patient. Well, maybe the gender of the patient is also important, but it doesn't happen to be in the model. Now, if when they did a fit, it came out with 0.9, that says at worst case, the variables we didn't consider could cause a 10 percent error. But, that could be big, that could matter a lot. And so as you get farther from 1, you ought to get very worried about whether you actually have all the right variables. Now you might have the right variables, and just experiment was not conducted well, But it's usually the case that the problem is not that, but that there are lurking variables. And we'll see examples of that.

    So, easier to read than the math, at least by me, easier to read than the math, is the implementation of r square. So it's measured and estimated values, I get the diffs, the differences, between the estimated and the measured. These are both arrays, so I subtract 1 array from the other, and then I square it. Remember, this'll do an element-wise subtraction, and then square each element. Then I can get the mean, by dividing the sum of the array measured by the length of it. I can get the variance, which is the measured mean minus the measured value, again squared. And then I'll return 1 minus this. All right? So, just to make sure we sort of understand the code, and the theory here as well, what would we get if we had absolutely perfect prediction? So if every measured point actually fit on the curb predicted by our model, what would r square return? So in this case, measured and estimated would be identical. What gets return by this? Yeah, 1. Exactly right. Because when I compute it, it will turn out that these two numbers will be the, I'll get 0, 1 minus is 0, right? Because the differences will be zero. OK?

    So I can use this, now, to actually get a notion of how good my fit is. So let's look at speed dot pi again here, and now I'm going to uncomment these two things, where I'm going to, after I compute the fit, I'm going to then measure it. And you'll see here that the r squared error for the linear fit is 0.896, and for the quadratic fit is 0.973. So indeed, we get a much better fit here. So not only does our eye tell us we have a better fit, our more formal statistical measure tells us we have a better fit, and it tells us how good it is. It's not a perfect fit, but it's a pretty good fit, for sure.

    Now, interestingly enough, it isn't surprising that the quadratic fit is better than the linear fit. In fact, the mathematics of this should tell us it can never be worse. How do I know it can never be worse? That's just, never is a really strong word. How do I know that? Because, when I do the quadratic fit, if I had perfectly linear data, then this coefficient, whoops, not that coefficient, wrong, this coefficient, could be 0. So if I ask it to do a quadratic fit to linear data, and the a is truly perfectly linear, this coefficient will be 0, and my model will turn out to be the same as the linear model. So I will always get at least as good a fit.

    Now, does this mean that it's always better to use a higher order polynomial? The answer is no, and let's look at why. So here what I've done is, I've taken seven points, and I've generated, if you look at this line here, the y-values, for x in x vals, points dot append x plus some random number. So basically I've got something linear in x, but I'm perturbing, if you will, my data by some random value. Something between and 1 is getting added to things. And I'm doing this so my points won't lie on a perfectly straight line. And then we'll try and fit a line to it. And also, just for fun, we'll try and fit a fifth order polynomial to it. And let's see what we get. Well, there's my line, and there's my fifth order polynomial. Neither is quite perfect, but which do you think looks like a closer fit? With your eye. Well, I would say the red line, the red curve, if you will, is a better fit, and sure enough if we look at the statistics, we'll see it's 0.99, as opposed to 0.978. So it's clearly a closer fit.

    But that raises the very important question: does closer equal better, or tighter, which is another word for closer? And the answer is no. It's a tighter fit, but it's not necessarily better, in the sense of more useful. Because one of the things I want to do when I build a model like this, is have something with predictive power. I don't really necessarily need a model to tell me where the points I've measured lie, because I have them. The whole purpose of the model is to give me some way to predict where unmeasured points would lie, where future points would lie. OK, I understand how the spring works, and I can guess where it would be if things I haven't had the time to measure, or the ability to measure. So let's look at that. Let's see, where'd that figure go. It's lurking somewhere. All right, we'll just kill this for now.

    So let's generate some more points, and I'm going to use exactly the same algorithm. But I'm going to generate twice as many points. But I'm only fitting it to the first half. So if I run this one, figure one is what we looked at before. The red line is fitting them a little better. But here's figure two. What happens when I extrapolate the curve to the new points? Well, you can see, it's a terrible fit. And you would expect that, because my data was basically linear, and I fit in non-linear curve to it. And if you look at it you can see that, OK, look at this, to get from here to here, it thought I had to take off pretty sharply. And so sure enough, as I get new points, the prediction will postulate that it's still going up, much more steeply than it really does. So you can see it's a terrible prediction.

    And that's because what I've done is, I over-fit the data. I've taken a very high degree polynomial, which has given me a good close fit, and I can always get a fit, by the way. If I choose a high enough degree polynomial, I can fit lots and lots of data sets. But I have reason to be very suspicious. The fact that I took a fifth order polynomial to get six points should make me very nervous. And it's a very important moral. Beware of over-fitting. If you have a very complex model, there's a good chance that it's over-fit. The larger moral is, beware of statistics without any theory. You're just cranking away, you get a great r squared, you say it's a beautiful fit. But there was no real theory there. You can always find a fit. As Disraeli is alleged to have said, there are three kinds of lies: lies, damned lies, and statistics. And we'll spend some more time when we get back from Thanksgiving looking at how to lie with statistics. Have a great holiday, everybody.
  • 36
    MIT Lecture 22: Normal, Uniform, and Exponential Distributions
    50:49
    Topics covered: Normal, uniform, and exponential distributions; misuse of statistics

    LECTURE TRANSCRIPT

    PROFESSOR: So, if you remember, just before the break, as long ago as it was, we had looked at the problem of fitting curves to data. And the example we had seen, is that it's often possible, in fact, usually possible, to find a good fit to old values. What we looked at was, we looked at a small number of points, we took a high degree polynomial, sure enough, we got a great fit. The difficulty was, a great fit to old values does not necessarily imply a good fit to new values. And in general, that's somewhat worrisome.

    So now I want to spend a little bit of time I'm looking at some tools, that we can use to better understand the notion of, when we have a bunch of points, what do they look like? How does the variation work? This gets back to a concept that we've used a number of times, which is a notion of a distribution. Remember, the whole logic behind our idea of using simulation, or polling, or any kind of statistical technique, is the assumption that the values we would draw were representative of the values of the larger population. We're sampling some subset of the population, and we're assuming that that sample is representative of the greater population. We talked about several different issues related to that.

    I now want to look at that a little bit more formally. And we'll start with the very old problem of rolling dice. I presume you've all seen what a pair of dice look like, right? They've got the numbers 1 through 6 on them, you roll them and something comes up. If you haven't seen it, if you look at the very back, at the back page of the handout today, you'll see a picture of a very old die. Some time from the fourth to the second century BC. Looks remarkably like a modern dice, except it's not made out of plastic, it's made out of bones. And in fact, if you were interested in the history of gambling, or if you happen to play with dice, people do call them bones. And that just dates back to the fact that the original ones were made that way. And in fact, what we'll see is, that in the history of probability and statistics, an awful lot of the math that we take for granted today, came from people's attempts to understand various games of chance.

    So, let's look at it. So we'll look at this program. You should have this in the front of the handout. So I'm going to start with a fair dice. That is to say, when you roll it, it's equally probable that you get 1, 2, 3, 4, 5, or 6. And I'm going to throw a pair. You can see it's very simple. I'll take d 1, first die is random dot choice from vals 1. d 2 will be random dot choice from vals 2. So I'm going to pass it in two sets of possible values, and randomly choose one or the other, and then return them. And the way I'll conduct a trial is, I'll take some number of throws, and two different kinds of dice. Throws will be the empty set, actually, yeah. And then I'll just do it. For i in range number of throws, d 1, d 2 is equal to throw a pair, and then I'll append it, and then I'll return it. Very simple, right? Could hardly imagine a simpler little program.

    And then, we'll analyze it. And we're going to analyze it. Well, first let's analyze it one way, and then we'll look at something slightly different. I'm going to conduct some number of trials with two fair die. Then I'm going to make a histogram, because I happen to know that there are only 11 possible values, I'll make 11 bins. You may not have seen this locution here, Pylab dot x ticks. That's telling it where to put the markers on the x-axis, and what they should be. In this case 2 through 12, and then I'll label it. So let's run this program. And here we see the distribution of values. So we see that I get more 7s than anything else, and fewer 2s and 12s. Snake eyes and boxcars to you gamblers. And it's a beautiful distribution, in some sense. I ran it enough trials. This kind of distribution is called normal. Also sometimes called Gaussian, after the mathematician Gauss. Sometimes called the bell curve, because in someone's imagination it looks like a bell.

    We see these things all the time. They're called normal, or sometimes even natural, because it's probably the most commonly observed probability distribution in nature. First documented, although I'm sure not first seen, by deMoivre and Laplace in the 1700s. And then in the 1800s, Gauss used it to analyze astronomical data. And it got to be called, in that case, the Gaussian distribution. So where do we see it occur? We see it occurring all over the place. We certainly see it rolling dice. We see it occur in things like the distribution of heights. If we were to take the height of all the students at MIT and plot the distribution, I would be astonished if it didn't look more or less like that. It would be a normal distribution. A lot of things in the same height. Now presumably, we'd have to round off to the nearest millimeter or something. And a few really tall people, and a few really short people.

    It's just astonishing in nature how often we look at these things. The graph looks exactly like that, or similar to that. The shape is roughly that. The normal distribution can be described, interestingly enough, with just two numbers. The mean and the standard deviation. So if I give you those two numbers, you can draw that curve. Now you might not be able to label, you couldn't label the axes, because how would you know how many trials I did, right? Whether I did 100, or 1,000 or a million, but the shape would always be the same. And if I were to, instead of doing, 100,000 throws of the dice, as I did here, I did a million, the label on the y-axis would change, but the shape would be absolutely identical. This is what's called a stable distribution. As you change the scale, the shape doesn't change.

    So the mean tells us where it's centered, and the standard deviation, basically, is a measure of statistical dispersion. It tells us how widely spread the points of the data set are. If many points are going to be very close to the mean, then the standard deviation is what, big or small? Pardon? Small. If they're spread out, and it's kind of a flat bell, then the standard deviation will be large. And I'm sure you've all seen standard deviations. We give exams, and we say here's the mean, here's the standard deviation. And the notion is, that's trying to tell you what the average score was, and how spread out they are. Now as it happens, rarely do we have exams that actually fall on a bell curve. Like this. So in a way, don't be deceived by thinking that we're really giving you a good measure of the dispersion, in the sense, that we would get with the bell curve.

    So the standard deviation does have a formal value, usually written sigma, and it's the estimates of x squared minus the estimates of x, and then I take all of this, the estimates of x, right, squared. So I don't worry much about this, but what I'm basically doing is, x is all of the values I have. And I can square each of the values, and then I subtract from that, the sum of the values, squaring that. What's more important than this formula, for most uses, is what people think of as the -- why didn't I write it down -- this is interesting. There is a some number -- I see why, I did write it down, it just got printed on two-sided. Is the Empirical Rule. And this applies for normal distributions. So anyone know how much of the data that you should expect to fall within one standard deviation of the mean? 68. So 68% within one, 95% of the data falls within two, and almost all of the data within three. These values are approximations, by the way. So, this is, really 95% falls within 1.96 standard deviations, it's not two. But this gives you a sense of how spread out it is. And again, this applies only for a normal distribution. If you compute the standard deviation this way, and apply it to something other than a normal distribution, there's no reason to expect that Empirical Rule will hold. OK people with me on this? It's amazing to me how many people in society talk about standard deviations without actually knowing what they are.

    And there's another way to look at the same data, or almost the same data. Since it's a random experiment, it won't be exactly the same. So as before, we had the distribution, and fortunately it looks pretty much like the last one. We would've expected that. And I've now done something, another way of looking at the same information, really, is, I printed, I plotted, the probabilities of different values. So we can see here that the probability of getting a 7 is about 0.17 or something like that. Now, since I threw 100,000 die, it's not surprising that the probability of 0.17 looks about the same as 17,000 over here. But it's just a different way of looking at things. Right, had I thrown some different number, it might have been harder to visualize what the probability distribution looked like. But we often do talk about that. How probable is a certain value?

    People who design games of chance, by the way, something I've been meaning to say. You'll notice down here there's just, when we want to save these things, there's this little icon that's a floppy disk, to indicate store. And I thought maybe many of you'd never seen a floppy disk, so I decided to bring one in. You've seen the icons. And probably by the time most of you got, any you ever had a machine with a quote floppy drive? Did it actually flop the disk? No, they were pretty rigid, but the old floppy disks were really floppy. Hence they got the name. And, you know it's kind of like a giant size version. And it's amazing how people will probably continue to talk about floppy disks as long as they talk about dialing a telephone. And probably none of you've ever actually dialed a phone, for that matter, just pushed buttons . But they used to have dials that you would twirl. Anyway, I just thought everyone should at least see a floppy disk once. This is, by the way, a very good way to get data security. There's probably no way in the world to read the information on this disk anymore.

    All right, as I said people who design games of chance understand these probabilities very well. So I'm gonna now look at, show how we can understand these things in some other ways of popular example. A game of dice. Has anyone here ever played the game called craps? Did you win or lose money? You won. All right, you beat the odds. Well, it's a very popular game, and I'm going to explain it to you. As you will see, this is not an endorsement of gambling, because one of the things you will notice is, you are likely to lose money if you do this. So I tend not, I don't do it.

    All right, so how does a game of craps work? You start by rolling two dice. If you get a 7 or an 11, the roller, we'll call that the shooter, you win. If you get a 2, 3, or a 12, you lose. I'm assuming here, you're betting what's called the pass line. There are different ways to bet, this is the most common way to bet, we'll just deal with that. If it's not any of these, what you get is, otherwise, the number becomes what's called the point. Once you've got the point, you keep rolling the dice until 1 of o things happens. You get a 7, in which case you lose, or you get the point, in which case you win. So it's a pretty simple game. Very popular game. So I've implemented. So one of the interesting things about this is, if you try and actually figure out what the probabilities are using pencil and paper, you can, but it gets a little bit involved. Gets involved because you have to, all right, what are the odds of winning or losing on the first throw? Well, you can compute those pretty easily, and you can see that you'd actually win more than you lose, on the first throw. But if you look at the distribution of 7s and 11s and 2s, 3s, and 12s, you add them up, you'll see, well, this is more likely than this. But then you say, suppose I don't get those. What's the likelihood of getting each other possible point value, and then given that point value, what's the likelihood of getting that before a 7? And you can do it, but it gets very tedious.

    So those of us who are inclined to think computationally, and I hope by now that's all of you as well as me, say well, instead of doing the probabilities by hand, I'm just going to write a little program. And it's a program that took me maybe 10 minutes to write. You can see it's quite small, I did it yesterday. So the first function here is craps, it returns true if the shooter wins by betting the pass line. And it's just does what I said. Rolls them, if the total is 1 or 11, it returns true, you win. If it's 2, 3, or 12, it returns false, you lose. Otherwise the point becomes the total. And then while true, I'll just keep rolling. Until either, if the total gets the point, I return true. Or if the total's equal 7, I return false. And that's it. So essentially I just took these rules, typed them down, and I had my game.

    And then I'll simulate it will some number of bets. Keeping track of the numbers of wins and losses. Just by incrementing 1 or the other, depending upon whether I return true or false. I'm going to, just to show what we do, print the number of wins and losses. And then compute, how does the house do? Not the gambler, but the person who's running the game, the casino. Or in other circumstances, other places. And then we'll see how that goes. And I'll try it with 100,000 games. Now, this is more than 100,000 rolls of the dice, right? Because I don't get a 7 or 11, I keep rolling. So before I do it, I'll as the easy question first. Who thinks the casino wins more often than the player? Who thinks the player wins more often than the casino? Well, very logical, casinos are not in business of giving away money.

    So now the more interesting question. How steep do you think the odds are in the house's favor? Anyone want to guess? Actually pretty thin. Let's run it and see. So what we see is, the house wins 50, in this case 50.424% of the time. Not a lot. On the other hand, if people bet 100,000, the house wins 848. Now, 100,000 is actually a small number. Let's get rid of these, should have gotten rid of these figures, you don't need to see them every time. We'll keep one figure, just for fun. Let's try it again. Probably get a little different answer. Considerable different, but still, less than 51% of the time in this trial. But you can see that the house is slowly but surely going to get rich playing this game.

    Now let's ask the other interesting question. Just for fun, suppose we want to cheat. Now, I realize none of you would never do that. But let's consider using a pair of loaded dice. So there's a long history, well you can imagine when you looked at that old bone I showed you, that it wasn't exactly fair. That some sides were a little heavier than others, and in fact you didn't get, say, a 5 exactly 1/6 of the time. And therefore, if you were using your own dice, instead of somebody else's, and you knew what was most likely, you might do better. Well, the modern version of that is, people do cheat by putting little weights in dice, to just make tiny changes in the probability of one number or another coming up. So let's do that. And let's first ask the question, well, what would be a nice way to do that? It's very easy here. If we look at it, all I've done is, I've changed the distribution of values, so instead of here being 1, 2, 3, 4, 5, and 6, it's 1, 2, 3, 4, 5, 5, and 6. I snuck in an extra 5 on one of the two dice. So this has changed the odds of rolling a 5 from 1 in 6 to roughly 3 in 12. Now 1/6, which is 2/12, vs. 3/12, it's not a big difference. And you can imagine, if you were sitting there watching it, you wouldn't notice that 5 was coming up a little bit more often than you expected. Normally. Close enough that you wouldn't notice it.

    But let's see if, what difference it makes? What difference do you think it will make here? First of all, is it going to be better or worse for the player? Who thinks better? Who thinks worse? Who thinks they haven't a clue? All right, we have an honest man. Where is Diogenes when we we need him? The reward for honesty. I could reward you and wake him up at the same time. It's good. All right, well, let's see what happens. All right, so suddenly, the odds have swung in favor of the player. This tiny little change has now made it likely that the player win money, instead of the house. So what's the point? The point is not, you should go out and try and cheat casinos, because you'll probably find an unpleasant consequence of that. The point is that, once I've written this simulation, I can play thought experiments in a very easy way. So-called what if games. What if we did this? What if we did that? And it's trivial to do those kinds of things. And that's one of the reasons we typically do try and write these simulations. So that we can experiment with things. Are there any other experiments people would like to perform while we're here? Any other sets of die you might like to try?

    All right, someone give me a suggestion of something that might work in the house's favor. Suppose a casino wanted to cheat. What do you think would help them out? Yeah?

    STUDENT: Increase prevalence of 1, instead of 5?

    PROFESSOR: All right, so let's see if we increase the probability of 1, what it does? Yep, clearly helped the house out, didn't it? So that would be a good thing for the house. Again, you know, three key strokes and we get to try it. It's really a very nice kind of thing to be able to do. OK, this works nicely. We'll get normal distributions. We can look at some things.

    There are two other kinds of distributions I want to talk about. We can get rid of this distraction. As you can imagine, I played a lot with these things, just cause it was fun once I had it. You have these in your handout. So the one on the upper right, is the Gaussian, or normal, distribution we've been talking about. As I said earlier, quite common, we see it a lot. The upper left is what's called a, and these, by the way, all of these distributions are symmetric, just in this particular picture. How do you spell symmetric, one or two m's? I help here. That right? OK, thank you. And they're symmetric in the sense that, if you take the mean, it looks the same on both sides of the mean. Now in general, you can have asymmetric distributions as well. But for simplicity, we'll here look at symmetric ones.

    So we've seen the bell curve, and then on the upper left is what's called the uniform. In a uniform distribution, each value in the range is equally likely. So to characterize it, you only need to give the range of values. I say the values range from to 100, and it tells me everything I know about the uniform distribution. Each value in that will occur the same number of times. Have we seen a uniform distribution? What have we seen that's uniform here? Pardon?

    STUDENT: Playing dice.

    PROFESSOR: Playing dice. Exactly right. Each roll of the die was equally likely. Between 1 and 6. So, we got a normal distribution when I summed them, but if I gave you the distribution of a single die, it would have been uniform, right? So there's an interesting lesson there. One die, the distribution was uniform, but when I summed them, I ended up getting a normal distribution.

    So where else do we see them? In principle, lottery winners are uniformly distributed. Each number is equally likely to come up. To a first approximation, birthdays are uniformly distributed, things like that. But, in fact, they rarely arise in nature. You'll hardly ever run a physics experiment, or a biology experiment, or anything like that, and come up with a uniform distribution. Nor do they arise very often in complex systems. So if you look at what happens in financial markets, none of the interesting distributions are uniform. You know, the prices of stocks, for example, are clearly not uniformly distributed. Up days and down days in the stock market are not uniformly distributed. Winners of football games are not uniformly distributed. People seem to like to use them in games of chance, because they seem fair, but mostly you see them only in invented things, rather than real things.

    The third kind of distribution, the one in the bottom, is the exponential distribution. That's actually quite common in the real world. It's often used, for example, to model arrival times. If you want to model the frequency at which, say, automobiles arrive, get on the Mass Turnpike, you would find that the arrivals are exponential. We see with an exponential is, things fall off much more steeply around the mean than with the normal distribution. All right, that make sense? What else is exponentially distributed? Requests for web pages are often exponentially distributed. The amount of traffic at a website. How frequently they arrive. We'll see much more starting next week, or maybe even starting Thursday, about exponential distributions, as we go on with a final case study that we'll be dealing with in the course.

    You can think of each of these, by the way, as increasing order of predictability. Uniform distribution means the result is most unpredictable, it could be anything. A normal distribution says, well, it's pretty predictable. Again, depending on the standard deviation. If you guess the mean, you're pretty close to right. The exponential is very predictable. Most of the answers are right around the mean. Now there are many other distributions, there are Pareto distributions which have fat tails, there are fractal distributions, there are all sorts of things. We won't go into to those details.

    Now, I hope you didn't find this short excursion into statistics either too boring or too confusing. The point was not to teach you statistics, probability, we have multiple courses to do that. But to give you some tools that would help improve your intuition in thinking about data. In closing, I want to give a few words about the misuse of data. Since I think we misuse data an awful lot. So, point number 0, as in the most important, is beware of people who give you properties of data, but not the data. We see that sort of thing all the time. Where people come in, and they say, OK, here it is, here's the mean value of the quiz, and here's the standard deviation of the quiz, and that just doesn't really tell you where you stand, in some sense. Because it's probably not normally distributed. You want to see the data. At the very least, if you see the data, you can then say, yeah, it is normally distributed, so the standard deviation is meaningful, or not meaningful. So, whenever you can, try and get, at least, to see the data.

    So that's 1, or 0. 1 is, well, all right. I'm going to test your Latin. Cum hoc ergo propter hoc. All right. I need a Latin scholar to translate this. Did not one of you take Latin in high school? We have someone who did. Go ahead.

    STUDENT: I think it means, with this, therefore, because of this.

    PROFESSOR: Exactly right. With this, therefore, because of this. I'm glad that at least one person has a classical education. I don't, by the way. Essentially what this is telling us, is that correlation does not imply causation. So sometimes two things go together. They both go up, they both go down. And people jump to the conclusion that one causes the other. That there's a cause and effect relationship. That is just not true. It's what's called a logical fallacy. So we see some examples of this. And you can get into big trouble.

    So here's a very interesting one. There was a very widely reported epidemiological study, that's a medical study where you get statistics about large populations. And it showed that women, who are taking hormone replacement therapy, were found to have a lower incidence of coronary heart disease than women who didn't. This was a big study of a lot of women. This led doctors to propose that hormone replacement therapy for middle aged women was protective against coronary heart disease. And in fact, in response to this, a large number of medical societies recommended this. And a large number of women were given this therapy. Later, controlled trials showed that in fact, hormone replacement therapy in women caused a small and significant increase in coronary heart disease. So they had taken the fact that these were correlated, said one causes the other, made a prescription, and it turned out to be the wrong one.

    Now, how could this be? How could this be? It turned out that the women in the original study who were taking the hormone replacement therapy, tended to be from a higher socioeconomic group than those who didn't. Because the therapy was not covered by insurance, so the women who took it were wealthy. Turns out wealthy people do a lot of other things that are protective of their hearts. And, therefore, are in general healthier than poor people. This is not a surprise. Rich people are healthier than poor people. And so in fact, it was this third variable that was actually the meaningful one. This is what is called in statistics, a lurking variable.

    Both of the things they were looking at in this study, who took the therapy, and who had a heart disease, each of those was correlated with the lurking variable of socioeconomic position. And so, in effect, there was no cause and effect relationship. And once they did another study, in which the lurking variable was controlled, and they looked at heart disease among rich women separately from poor women, with this therapy, they discovered that therapy was not good. It was, in fact, harmful. So this is a very important moral to remember. When you look at correlations, don't assume cause and effect. And don't assume that there isn't a lurking variable that really is the dominant factor. So that's one statistical, a second statistical worry.

    Number 2 is, beware of what's called, non-response bias. Which is another fancy way of saying, non-representative samples. No one doing a study beyond the trivial can sample everybody or everything. And only mind readers can be sure of what they've missed. Unless, of course, people choose to miss things on purpose. Which you also see. And that brings me to my next anecdote.

    A former professor at the University of Nebraska, who later headed a group called The Family Research Institute, which some of you may have heard about, claimed that gay men have an average life expectancy of 43 years. And they did a study full of statistics showing that this was the case. And the key was, they calculated the figure by checking gay newspapers for obituaries and news about stories of death. So they went through the gay newspapers, took a list of everybody whose obituary appeared, how old they were when they died, took the average, and said it was 43. Then they did a bunch of statistics, with all sorts of tests, showing how, you know, what the curves look like, the distributions, and the significance. All the math was valid. The problem was, it was a very unrepresentative sample. What was the most unrepresentative thing about it? Somebody?

    STUDENT: Not all deaths have obituaries.

    PROFESSOR: Well, that's one thing. That's certainly true. But what else? Well, not all gay people are dead, right? So if you're looking at obituaries, you're in fact only getting -- I'm sure that's what you were planning to say -- sorry. You're only getting the people who are dead, so it's clearly going to make the number look smaller, right? Furthermore, you're only getting the ones that were reported in newspapers, the newspapers are typically urban, rather than out in rural areas, so it turns out, it's also biased against gays who chose not come out of the closet, and therefore didn't appear in these. Lots and lots of things with the problems.

    Believe it or not, this paper was published in a reputable journal. And someone checked all the math, but missed the fact that all of that was irrelevant because the sample was wrong. Data enhancement. It even sounds bad, right? You run an experiment, you get your data, and you enhance it. It's kind of like when you ran those physics experiments in high school, and you've got answers that you knew didn't match the theory, so you fudged the data? I know none of you would have ever done that, but some people been known to do. That's not actually what this means. What this means is, reading more into the data than it actually implies. So well-meaning people are often the guiltiest here.

    So here's another one of my favorites. For example, there are people who try to scare us into driving safely. Driving safely is a good thing. By telling holiday deaths. So you'll read things like, 400 killed on the highways over long weekend. It sounds really bad, until you observe the fact that roughly 400 people are killed on any 3-day period. And in fact, it's no higher on the holiday weekends. I'll bet you all thought more people got killed on holiday weekends. Well, typically not. They just report how many died, but they don't tell you the context, say, oh, by the way, take any 3-day period. So the moral there is, you really want to place the data in context. Data taken out of context without comparison is usually meaningless.

    Another variance of this is extrapolation. A commonly quoted statistic. Most auto accidents happen within 10 miles of home. Anyone here heard that statistic? It's true, but what does it mean? Well, people tend to say it means, it's dangerous to drive when you're near home. But in fact, most driving is done within 10 miles of home. Furthermore, we don't actually know where home is. Home is where the car is supposedly garaged on the state registration forms. So, data enhancements would suggest that I should register my car in Alaska. And then I would never be driving within 10 miles of home, and I would be much safer. Well, it's probably not a fact. So there are all sorts of things on that. Well, I think I will come back to this, because I have a couple more good stories which I hate not to give you. So we'll come back on Thursday and look at a couple of more things that can go wrong with statistics.
  • 37
    MIT Lecture 23: Stock Market Simulation
    51:10
    Topics covered: Stock market simulation

    LECTURE TRANSCRIPT

    PROFESSOR: I want to pick up exactly where I left off last time. When I was talking about various sins one can commit with statistics. And I had been talking about the sin of data enhancement, where the basic idea there is, you take a piece of data, and you read much more into it than it implies. In particular, a very common thing people do with data is they extrapolate. I'd given you a couple of examples. In the real world, it's often not desirable to say that I have a point here, and a point here, therefore the next point will surely be here. And we can just extrapolate in a straight line. We before saw some examples where I had an algorithm to generate points, and we fit a curve to it, used the curve to predict future points, and discovered it was nowhere close.

    Unfortunately, we often see people do this sort of thing. One of my favorite stories is, William Ruckelshaus, who was head of the Environmental Protection Agency in the early 1970s. And he had a press conference, spoke about the increased use of cars, and the decreased amount of carpooling. He was trying to get people to carpool, since at the time carpooling was on the way down, and I now quote, "each car entering the central city, sorry, in 1960," he said, "e