Big Data Analytics with PySpark + Power BI + MongoDB

Install python 3.7 on Windows 64-bit, add to environment variables, verify installation with python --version, then install Anaconda and set up Jupiter notebooks for data analysis.

Install Apache Spark by downloading Spark 3 preview with Hadoop 2.7, unzip to a Spark folder, and configure Spark home and path; install winutils and Hadoop home path.

Install java 8 by downloading the java se development kit 8 from Oracle, accept the license, and verify with java -version; optionally create a hive folder for older spock versions.

Test Apache Spark installation by launching Spock shell, converting a list to an RDD and data frame, then using Pi Spall to interact with Spark via Python.

Download and install the MongoDB community server (Windows, MSI) version 2.3, configure data and log directories, then verify with the MongoDB shell to see the default database.

Install and use no sequel booster to connect to MongoDB locally, create databases and collections, and query documents with JavaScript in the data pipeline.

Install the find Spock library to allow Jupiter notebook to detect the Spock installation and run its execution engine, then initialize a Spock session and preview a simple data frame.

pulls a dataset from GitHub into a Spark dataframe, cleans it by dropping unused columns, adds a year field, and computes yearly earthquake counts for MongoDB using the Spark connector.

Transform data by casting numeric fields to double types, compute yearly max and average magnitudes, and join results, then drop nulls and load into MongoDB for reporting.

Configure the Mongo Spock connector, prepare input and output configurations, and load cleaned data frames into MongoDB, overwriting existing collections as needed.

Pre-process the data by loading training data from MongoDB and test data from GitHub, rename fields to date, latitude, longitude, magnitude, depth, convert to double, and prepare frames.

Build a random forest regression model using a vector assembler to combine latitude, longitude, and depth, then evaluate with RMSE on earthquake magnitude data.

Create a prediction dataset from results by selecting latitude, longitude, and predicted magnitude, renaming field, and adding year 2017 RMSE column; load it into MongoDB collection for reporting and dashboards.

Download and install Visual Studio Code with Windows 64-bit installer, create a project folder in documents, open it in Visual Studio Code, and install the Python extension for debugging datasets.

Create a PySpark ETL script to drop and rebuild a MongoDB database, load and clean the data, derive year and magnitude stats, and write to quakes and quake_freq collections.

Create a PySpark machine learning pipeline that preprocesses data, uses vector assembler features, trains a random forest regressor, and evaluates with rmse; write predictions back to mongodb for dashboards.

Install Power BI Desktop to create interactive dashboards and reports from data stored in MongoDB, then connect the data source to build your first dashboard.

Install the MongoDB ODBC drivers: download the MongoDB API Connector for Windows x64, install the Visual C++ redistributables, then install the MongoDB ODBC driver (v1.3.0) for Windows 64-bit.

Create a system DSN for MongoDB using the BI connector, configure a 64-bit ODBC data source named Mongo API connector on localhost port 3307, and test the connection for Tableau.

Connect to MongoDB from Power BI desktop using the Mongo API connector, load three collections from the quake database, and prepare the data sources to create the dashboard.

Create a geo map visualization in Power BI using latitude and longitude from the prediction data set, sizing markers by predicted magnitude and overlaying reference earthquake layers 2015–2016 and 2017.

Create a table plot in the dashboard displaying prediction data with latitude, longitude, predicted magnitude, and root mean squared era, linked to the map for interactive exploration.

Create an area chart to plot maximum and average magnitude by year, adjust y-axis title to magnitude, and label chart as max and avg magnitude by year, no coding required.

Create a bar chart of earthquakes by year using a stacked column chart, title it 'number of earthquakes by year', and apply the high rise theme.

Create a doughnut chart to count earthquake types in the quakes dataset using Power BI, then finalize the dashboard with axis labels and a renamed page for Mongo DV analytics.