Big Data Analytics with PySpark + Tableau Desktop + MongoDB

Install Apache Spark on Windows by downloading Spark 3.x with Hadoop 2.7, extracting the zip, and configuring spark home and path; install winutils and set Hadoop home and path.

Install and verify Java 8 (JDK 8) on Windows, including downloading from Oracle, accepting the license, and checking with java -version; optional hive cache setup for older Spock versions.

Test Apache Spark installation by launching the Spock shell, converting a sample array to an RDD and a dataframe to confirm the Python API works with PySpark.

Download the MongoDB community server MSI for Windows, choose the latest 2.3 release, install as a complete setup with default data and log directories, then verify using the mongo shell.

Install NoSQL Booster to manage MongoDB, connect locally, and create a quake database with a dummy collection, then query documents using JavaScript via db.dummy.find.

Integrate PySpark with Jupyter notebook by installing the Find Spock library, testing import and initialization, then create a Spark session and a PySpark data frame to run data pipelines.

Import a dataset from GitHub into a Spark dataframe, clean unused columns, derive a year field from the date, and count records by year for MongoDB storage.

Configure the Mongo DV spot connector in a Jupyter notebook, then load cleaned data frames into MongoDB with the Spock connector, overwriting data in the quake and quake 3 collections.

Perform data pre-processing for machine learning by loading earthquake data, creating training and test sets, and cleaning to prepare features. Rename fields, cast to doubles, and drop missing values.

Select latitude, longitude, and depth as features via a vector assembler, train a random forest regression model in a pipeline to predict earthquake magnitude, and evaluate with RMSE below 0.5.

Create the prediction dataset by selecting latitude, longitude, and predicted magnitude, rename the field to predicted magnitudes, add year and RMSE columns for 2017, and load into MongoDB.

Learn how to download and install Visual Studio Code on Windows, set up a project folder, open the editor, and install the Python extension to build data pipeline scripts.

Create a PySpark ETL script to extract, transform, and load earthquake data into MongoDB using a Spark session and data frames.

Create a machine learning script to build and evaluate a random forest model from preprocessed quake data, train on MongoDB data, generate predictions, and store results for Tableau dashboards.

Install the MongoDB ODBC drivers by downloading the BI connector for MongoDB, install the Visual C++ redistributable, and connect Tableau to MongoDB for reports, dashboards, and graphs.

Create a system DSN for MongoDB using the ODBC driver to enable access from all applications. Start the MongoDB BI connector on port 3307, then test the connection.

Create a geo map visualization in Tableau using longitude and latitude, add predicted magnitude and year to tooltips, customize colors, size, opacity, and map background for clear earthquake data insights.

Create a magnitude plot to visualize maximum and average earthquake magnitudes per year with a line graph in Tableau. Add styling, markers, tooltips, and a trend line.

Create a Tableau table plot from the quakes data by dragging the type field to rows and using a count measure to show earthquake types, then rename the sheet.

Create a multi-chart Tableau dashboard combining an earthquake map, magnitude plots by year, and a types table from cleaned MongoDB data via a data pipeline, with interactive location search.