AI Powered Customer Spending Forecasting in Insurance

Navigate a data warehouse architecture with data sources, staging, dw, data lake, and data mart to power insurance spending forecasting with ETL, CDC, AI and ML.

Investigates source systems and key tables—dim customer, dim branch, dim account, and fact tables—to design the dm insurance fact data model for predicting insurance policy size.

Create a data warehouse and data mart in MySQL Workbench by establishing DW and datamart schemas, then run SQL scripts to build seven tables including customer data and insurance policies.

Install and run apache niFi on windows, configure java jdk and environment variables, start the niFi server, and access the niFi ui at localhost 8443 to build data flows.

Demonstrates loading multiple CSV files into a MySQL data warehouse using Apache NiFi, including dim branch, dim customer, and dim account, with truncation, transformation to SQL, and validation.

Explore validating dimension and fact tables, create datamarts, and compute yearly 2023 spending—credit card, insurance life, and overdraft—consolidated into the dm insurance forecast with row-count validation.

Configure two process groups in Apache Nifi for truncating the datamart and loading data, then validate results in MySQL.

predict insurance policy case size using a regression model with data cleaning, one-hot encoding, and scaling; train with xgboost and grid search, evaluate via mae and rmse, then deploy.

Visualize data with seaborn pair plots, histograms, kernel density estimates, box plots, and heatmap correlations; clean and encode features, scale with MinMaxScaler, and split training and testing (80/20) before XGBoost.

Understand how XGBoost, or extreme gradient boosting, uses an ensemble of decision trees to sequentially correct errors, starting from an initial prediction and updating with residuals and a learning rate.

Train and evaluate an XGBoost regressor in a Jupyter notebook, using squared error, MAE, RMSE, and feature importance to guide GridSearchCV hyperparameter tuning for insurance spending forecasting.

Understand artificial neural networks, from neurons and layers to weights, biases, activation functions, forward and backpropagation, loss functions, and training with SGD or Adam for insurance forecasting.

Pack and load trained models by using joblib or pickle for XGBoost, and use Keras Model.save and Loadmodel for TensorFlow models, preserving architecture, weights, and training configuration.