Skip to content
Armand Notue

Data Science · Regression · Explainable ML

Medical Insurance Premium Prediction — Choosing the model you can explain

Predicting medical insurance charges with 19 models — and choosing the slightly less accurate one because it can be explained.

Complete

At a glance

Status
Complete
Focus
19 regression models compared · Smoker × BMI interaction as the key engineered feature · Interpretable model chosen for production
Stack
  • Python
  • scikit-learn
  • CatBoost
  • SHAP
  • pandas
Problem
Insurance pricing has to be accurate and explainable: a reviewer or a customer needs to understand why a premium is what it is.
Approach
Engineered 26 features on 1,338 policyholder records, compared 19 regression models, used SHAP to explain predictions, and weighed accuracy against interpretability.

Key results

R², CatBoost (best score)
0.882
R², Lasso (chosen for production)
0.868
models compared
19

01

Overview

An end-to-end regression study on the public medical-insurance dataset (1,338 policyholders): feature engineering, a wide model comparison, SHAP explanations, and a deliberate production choice.

02

Problem & motivation

Leaderboards reward the highest score. Real pricing decisions also need models that can be audited — this project makes that trade-off explicit.

03

My role

  • Independent project — analysis, modeling and write-up.

04

System architecture

  1. Stage 1:Data

    • 1,338 records

      Age, sex, BMI, children, smoker, region, charges

  2. Stage 2:Features

    • 26 engineered features

      Including a smoker × BMI interaction

  3. Stage 3:Models

    • 19 regression models

      From regularized linear models to gradient boosting

  4. Stage 4:Explain

    • SHAP

      Which features drive each prediction

  5. Stage 5:Decide

    • Lasso for production

      Readable coefficients, near-best accuracy

From raw records to an explainable production choice.

05

Technical implementation

  1. Interaction features

    Smoking and BMI matter most together: the engineered smoker × BMI interaction correlates 0.84 with charges.

  2. A wide model comparison

    Nineteen models evaluated, from regularized linear models to gradient boosting; CatBoost scored highest (R² 0.882).

  3. Explanations, not just scores

    SHAP values show which features push each prediction up or down, so the model's behavior can be checked against domain sense.

06

Engineering decisions & challenges

  • Interpretable beats marginally accurate

    Lasso scored R² 0.868 against CatBoost's 0.882 — 1.4 points lower — but its coefficients can be read and audited, which matters for pricing. Inference takes under a millisecond.

07

Evaluation & results

Test-set results from the notebook.

R², CatBoostBest score of 19 models
0.882
R², LassoChosen for production
0.868
smoker × BMI correlation with charges
0.84

Accuracy vs explainability

The production pick gives up 1.4 points of R² for coefficients a pricing reviewer can read.

Test R²Higher is better
  • Lasso (production)0.868
  • CatBoost (best score)0.882
Show data table
ItemValue
Lasso (production)0.868
CatBoost (best score)0.882

Source: Project notebook (Kaggle)

08

Lessons learned

  1. Choose the model for the decision

    The best model is the one that fits how its predictions will be used — here, a priced product that has to be explained.

09

Technologies used

  • Python
  • scikit-learn
  • CatBoost
  • SHAP
  • pandas

Want the deeper walkthrough?

I'm happy to go through the code, the trade-offs and what I'd do next.

Get in touch