Building Big Data Pipelines with R & Sparklyr & Power BI

Install R on Windows, set the path variable and environment variables, verify via the command line, and install RStudio to begin working with R in big data pipelines.

Install Apache Spark on Windows by downloading Spark 3.0 preview with Hadoop 2.7, unzip to spark folder on C drive, and set Spark home, path, and Hadoop home with winutils.

Install Java 8 JDK from Oracle, accept the license, and verify with java -version; for older Spock versions, optionally create tape tmp and hive folders.

Verify the Apache Spark installation by launching SparkR, confirming it works, and running a small code snippet to print a list and perform a basic operation.

Install sparklyr in RStudio, import dplyr, configure a local spark session and driver, then run a test to verify the spark application initializes successfully.

Extract, transform, and load data from a source into a data frame, then preview the first five records to verify successful import and prepare for transformation.

Transform and clean the data by selecting key fields, creating a year field, and computing yearly quake counts, maximum magnitudes, and average magnitudes for dashboards and analytics.

Preview cleaned data frames and export them to csb files in a spark pipeline, saving to an output folder for dashboards and downstream machine learning work.

Load the test data, align it with the training data by selecting relevant fields and renaming columns to match the training schema for machine learning pre-processing.

Train a random forest regressor using latitude, longitude, and depth to predict earthquake magnitude from training data, test with out-of-sample data, and evaluate RMSE below 0.5.

Create the 2017 prediction data set by renaming the prediction field to predicted magnitude, add year and RMSE, and save as a CSV file for visualization on graphs or maps.

Install Power BI Desktop using two straightforward methods: download from the web or install via the Microsoft Store, then launch after installation.

Load csv data sources into Power BI Desktop, rename files to their folder names, and set the first row as headers while converting latitude, longitude, depth, and magnitude to decimals.

Create a geo map in Power BI by mapping latitude and longitude, sizing points by predicted magnitude, and layering 2015–2016 earthquakes with 2017 predictions.

Visualize earthquake types with a donut chart by dragging fields from the quakes data center, resizing visuals, and styling the chart with a red color palette.

Create an area chart to display maximum and average earthquake magnitude per year using the quake freq data center, and adjust colors and the x-axis scale from 1965 to 2016.

Construct a bar chart of earthquakes by year in Power BI, configuring axes, colors, and title, then expand to a dashboard with map, area chart, and end-to-end data pipeline.