Building Big Data Pipelines with PySpark + MongoDB + Bokeh

Install Python 3.7 on Windows, add to environment variables, verify installation, then install Anaconda to access Jupyter notebooks for big data pipelines.

Open the command prompt as administrator, upgrade pip, and install pandas, numpy, pymongo, and bokeh to build the data pipeline's reporting and database integration.

Install Apache Spark by downloading Spark 3.0 preview with Hadoop 2.7, extract to C drive, and configure SPARK_HOME and PATH. Then install Hadoop winutils and update Hadoop PATH.

Install Java 8 (JDK 8) and verify by running java -version; optional steps include creating a temporary hive folder for older Spock versions before testing installation.

Test Apache Spark installation by launching the Spock shell, converting a sample array to an RDD and a DataFrame, and validating Pi Spall for pipeline development.

Download MongoDB community server for Windows, install with complete setup, configure data and log directories and the service, then verify the installation by launching the Mongo shell and running db.

Install NoSQL Booster for MongoDB, connect to a local instance, and create databases and collections to explore document-based storage in json-like format and basic queries.

Install and configure findspark to connect PySpark with Jupyter notebook, initialize a Spark session, and verify integration by creating and previewing a dataframe.

Extract data from GitHub, load into a PySpark dataframe, and derive a year field. Group by year to count earthquakes, preparing results for MongoDB storage.

Transform data by casting latitude, longitude, depth, and magnitude to double, compute yearly max and average magnitudes, join results, drop nulls, and load into MongoDB for ML-ready pipelines.

Configure the Mongo Spork connector in a Jupyter notebook, load cleaned dataframes, and write them to MongoDB collections like quake and quake 3, then verify the results with NoSQL Booster.

Execute data pre-processing to clean and transform earthquake data, load training from MongoDB and test data from GitHub, align fields, and prepare training and testing sets.

Apply a random forest regression model in PySpark to predict earthquake magnitude using latitude, longitude, and depth, assembled into a feature vector, and evaluate with RMSE on test data.

Extracts latitude, longitude and predicted magnitude from the prediction results, renames to predicted magnitude, adds year 2017 and RMSE, then saves the dataset to MongoDB for reporting and dashboards.

Read data from Mongo DV into pandas data frames, convert collections to data frames, and prepare the 2016 earthquakes data for visualization with Bokeh.

Create an interactive geo map with bokeh to display 2016 earthquakes and 2017 predictions, using murk projection coordinates, custom styling, tooltips, and a base map.

Create a bar chart of earthquakes by year from the quake frequency dataframe, then build a Bokeh figure with labeled axes and tooltips to show year and count.

Create a magnitude plot of maximum and average earthquake magnitudes by year from the quake data frame. Round the average to one decimal and add circles, tooltips, and legends.

Create a grid plot to display crops as a two-row dashboard in a web browser, using geo map, bar plot, and magnitude plot.

Install Visual Studio Code, choose the 64-bit Windows version, then set up a project folder and install the Python extension to begin scripting data pipelines.

Build a PySpark ETL script for the quakes pipeline that extracts data, transforms it (drop unused fields, derive year, compute counts, average magnitude, and maximum magnitude), and loads into MongoDB.

Create a spark-based machine learning pipeline to predict earthquake magnitudes by loading data from csv and MongoDB, training a random forest regressor, and evaluating with root mean squared error.

Create a Flask dashboard server to share a generated dashboard by hosting the HTML file on localhost:5000. Set up a virtual environment, install Flask, and organize templates.