Big Data Analytics with GIS

In this course, we are going to use a variety of data so that we can see how big data analytics can work in different contexts.  Our data will come from municipal government, emergency management, natural resources, and business.  Also, our data will come in many forms: databases, comma separated files, ESRI geodatabases, etc.  

Big data is discussed in many different ways.  We are going to explore the concept of Volume, Variety, and Veracity.  At the end, we'll see how working with the three V's will enable to us add one more V: Value.

While you can use the software and follow along just about anywhere in this course, this lecture will be our first official hands-on exercise by joining tens of thousands of soil boundaries and tens of thousands of building centroids.

Just because a computer lets you push a button to do some work, doesn't mean you should.  Understanding the different types of data is very important if you are going to choose the right tools to analyze big data.  

Sometimes big data is simply too big to comprehend.  The difficulty of comprehending really large data sets require us to classify them in order to show general trends or patterns.  Also, classification allows us to present our data to a broader audience, in a more understandable way.

One of the best tools for working with big data is a robust enterprise class database.  In this lecture, you'll lean about different databases, both large and small.  We'll also focus more on the use of Postgres as a database.  And, knowing how to use Postgres is important because it is powerful, and free.

SQL is the language of databases.  If you want to stick with using wizards that hold your hand through processes, you are never going to progress in your career.  In this lecture, we'll write dozens of SQL statements to interrogate the attributes of traditional databases.  

Adding to our SQL knowledge will be the use of SQL within a spatial context.  Here you'll learn that Spatial is not Special, and that by knowing how to apply SQL techniques within a spatial context, you'll have unlimited power as a data analyst.

All the SQL in the world is not going to do you any good if it takes to long to analyze data.  This lecture goes over the long lost art of using indexes within databases.  For most of us, we never think of using indexes because the data is so small, and computers are so fast. But, if you are going to work with big data, you better get used to working with indexes - they will make your life so much easier.

The Central Limit Theorem is an extremely important statistical concept to consider when working on big data.  And, now that you've learned about indexes, you'll see how we can leverage both of these concepts to provide excellent estimates of data results within seconds.

There are many clever ways to solve problems, and when you understand how indexes work (both spatial and attribute), you can tackle really big problems.  This lecture is going to show an example from my own research where we had to do some creative data preparation to improve the performance.