Building Big Data Pipelines with SparkR & PowerBI & MongoDB

Install apache spark by downloading spark 3.4 with hadoop 2.7, unzip the package, and locate the spark folder. Configure SPARK_HOME and PATH, then set Hadoop home with winutils for Windows.

Install Java 8 JDK for Spark by downloading from Oracle, accepting the license, installing on Windows x64, and verifying with java -version; optional Hive folder for older Spark versions.

download and install the mongodb community server for windows, select the latest version, configure data and log directories, and verify by launching the shell to confirm the default database.

Install NoSQL Booster for MongoDB to connect to a local server, create a quick database and a dummy collection, and query json-like documents instead of tables.

Open your studio, create a new project folder, install the SparkR library, and set up a script to start writing code.

Configure sparkR by locating SPORCK installation or SPORCK_HOME, load the SPORCK library, and initialize a session; connect to MongoDB with the SPORCK MongoDB DBI connector and configure input and output.

Extract data from various sources, transform it through cleaning and schema assignment, and load it into a SPORCK data frame for use in external applications.

Transform data by subsetting fields from the data frame and adding a year field from the date. Group by year to count earthquakes and compute maximum and average magnitudes.

Transform earthquake data by grouping by year to compute max and average magnitudes, join these results into a master data frame, and prepare a clean, report-ready dataset for dashboards.

Export cleaned and transformed data to MongoDB by writing two data frames into two collections, overwriting existing data after previewing. Ready for the machine learning stage.

Learn to build a smart data pipeline with extract, transform, load, and predict stages, using data pre-processing, training and testing data, and model validation for earthquake forecasting.

Train a random forest regression model on latitude, longitude, and depth to predict earthquake magnitude, then predict on testing data and compare to 2017 actuals.

Create the 2017 earthquake prediction dataset, rename the prediction field to predicted magnitude, select the fields we want, add the year, and export the results to MongoDB for analytics.

Install Power BI desktop by downloading it from the web or via the Microsoft Store, then install and launch the app after following the on-screen prompts.

Install the MongoDB ODBC drivers, including the MongoDB BI Connector and the ODBC driver version 1.3.0 for Windows 64, with the required Visual C++ redistributables.

Create a system DSN for MongoDB using the ODBC driver, run the BI Connector on localhost:33307, test the connection, and connect to dashboards in Tableau or Power BI.

Connect MongoDB data sources to Power BI using the MongoDB BI Connector, load three tables, and prepare them for building visualizations.

Select data source, latitude and longitude, magnitude sizing, color by magnitude, place tooltips, and a dark base map to reveal intense earthquakes.

Create a donut chart to visualize the different types of earthquakes in the dataset, selecting type for details and values, then customize colors and labels to match the dashboard theme.

Create an area chart to visualize yearly earthquakes from 1965 to 2016 using quake freak data, setting x and y axes and styling a dashboard for clear comparison.

Visualize maximum and average earthquake magnitude by year with a stacked bar chart in power bi, using a mongodb data source and formatting tips for clear dashboards.