Parallel processing with desktop computers for GIS

We spoke earlier about your computer being a mini high-performance computing center.  But, the benefits of parallel processing and big data analytics doesn't just happen.  It is important to know how your computer works and some of the considerations you have to employ.  In this lecture we'll briefly talk about how the CPU, RAM, HDD, and other components in your computer effect parallel processing in GIS, and some of the best practices for utilizing these components. 

Although we've talked about the hardware considerations for leveraging parallel processing in on your desktop, there are some additional concepts to consider: processing the data locally vs. distributed processing.  Distributed processing provides almost limitless capabilities for big data analytics.  But the truth is, most of us aren't going to need distributed processing for our day-to-day GIS tasks.  But, we have a great opportunity to utilize the multi-core capabilities of our desktop computers.  This lecture will help you better understand the differences and similarities of local vs. distributed computing when parallelizing GIS tasks. 

This lecture will provide an overview of the many parallel processing functions available with Esri's GeoAnalytics Desktop.  Although similar functionality exists with Postgres/PostGIS, Esri does such a good job of explaining the types of functions that can be performed with parallelization, we'll use their function list and a benchmark for comparing the different approaches. 

We've spent enough time talking - now it's time to do something.  In this hands-on activity, you will learn how to use ArcGIS Pro to perform an overlay of 9 million parking ticket locations and 130+ neighborhoods.  The process is almost impossible to run in ArcMap 10, but with the 64-bit version of ArcGIS Pro, it is more achievable.  Also, once you utilize GeoAnalytics Desktop, you'll see the immediate benefit of parallel processing tools in the ArcGIS Pro ecosystem.  We'll even do some experimentation with performing the task in parallel and also on a single processor.

Not to be outdone, we'll quickly take a look at how Postgres with PostGIS performs the same task of determining the number of parking ticket locations contained in each zone polygon.  This is just a quick demonstration, and the important thing is that my result will likely be faster than your result.  It's not because I'm smarter than you, it's because I've done a little back-end manipulation of the Postgres server to improve the performance of Postgres.  In the next section we'll cover how Postgres performs parallel processing what those manipulations are and how they can benefit your analysis. 

We'll round out our exploration of the different software packages by using Manifold GIS to determine the number of parking tickets in each zone. 

Even though we are performing parallel processing with PostGIS the truth is, PostGIS doesn't implement any parallel code - it creatively leverages parallel processing offered by Postgres.  In this lecture we'll talk about how that is achieved as we consider important Postgres concepts like worker processes, parallel workers, gather operations, and how PostGIS takes advantage of that for certain functions. 

Now that we have a better understanding of important definitions and concepts of parallel processing with Postgres, we'll take a deeper dive into how that is achieved.  One of the most important concepts to understand is the sophisticated capability of Postgres to develop a query plan, giving us insight into how the server intends to solve the problem.  Armed with the knowledge of the Postgres query plan, we'll learn how to make adjustments and modifications to help Postgres perform even better by causing it to reconsider some of its plan. 

This is where the fun really begins!  At this point we know the conceptual ways in which Postgres performs parallel  processing and also know what a query plan is and some of the parameters that can alter and improve the query plan.  Now we are going to revisit our earlier query to determine the number of parking ticket locations in each zone.  Using the EXPLAIN function in Postgres, we'll dive deeper into setting some of the important parameters to improve the query plan, and you'll see some amazing improvements beyond what I showed you earlier.  With just a few slight alterations, we can double our speed in Postgres, and become almost 3 times faster than what ArcGIS GeoAnalytics Desktop can perform.  This lecture is going to help you understand how that works so that you'll have confidence in fine tuning your system when needing to improve performance. 

In our previous lecture we discussed the importance of coordinate system transformation, and the burden it places on our analytical process when we have to perform those transformations on-the-fly.  In this hands-on activity, you'll get to test the differences in processing time using Esri's GeoAnalytics Desktop when you transform the data before running the analysis compared to letting the GIS software perform the transformation on-the-fly.

In our previous lecture we discussed the importance of coordinate system transformation, and the burden it places on our analytical process when we have to perform those transformations on-the-fly.  In this hands-on activity, you'll get to test the differences in processing time using Postgres/PostGIS when you transform the data on-the-fly rather than using the data already in the base coordinate system. 

In our previous lecture we discussed the importance of coordinate system transformation, and the burden it places on our analytical process when we have to perform those transformations on-the-fly.  In this hands-on activity, you'll get to test the differences in processing time using Manifold GIS when you transform the data before running the analysis compared to letting the GIS software perform the transformation on-the-fly.

One method to improve the speed of attribute searches is to create an index of the data.  You'll learn what an attribute index is, and how theoretically it can improve system performance. 

Based on our previous discussion of attribute indexes, you'll get an opportunity to perform an attribute search on millions of records, both with, and without an index using Esri tools. 

Based on our previous discussion of attribute indexes, you'll get an opportunity to perform an attribute search on millions of records, both with, and without an index using Postgres - it is amazing how lighting fast Postgres is with an index. 

This is a short video that I presented many years ago at a conference.  I had a family emergency, and was unable to attend the conference, so I created a short video to go along with the presentation my colleague was giving. 

The problem we ran into was that we were dealing with such large computations that traditional methods for using GIS wasn't working.  So, knowing what I knew about indexes and spatial indexes, I restructured the data into smaller polygons.  It seems counter-intuitive, but by creating more polygons, I was able to do the job so much faster.  Check out the video, and you'll see why that was the case.

The GeoAnalytics Desktop tools aren't some special feature that have to be run under a specific circumstance.  In actuality, Esri has done a great job in integrating GeoAnalytics Desktop within their entire ecosystem.  So, these big data tools are available to you through the GUI, model builder, and even Python.  So, you'll be able to access and deliver big spatial data solutions with GeoAnalytics Desktop just like you would with any other tool that Esri offers. 

In this lecture, you'll see how to use GeoAnalytics Desktop with Model Builder and Python.

You don't have to run everything in the Postgres interface. In fact, a desktop GIS like QGIS can access the data inside of Postgres and manipulate the data externally - and, because it is using the Postgres engine, QGIS can realize the same performance results as Postgres.  Now you have the power of parallel processing in your QGIS desktop application!!