DATA602 PROJECT - Analysis and Prediction of Diabetes using Health Indicators

AUTHORS:-

---

Hemanth Reddy Alavala

UID:- 121454041

mail: hreddy14@umd.edu

---

Ameer Sohail Shaik

UID:- 121958426

mail: sohail08@umd.edu

---

Likith Sagar Goshike

UID:- 122027831

mail:- lgoshike@umd.edu

---

Rashmitha Vakkapatla

UID:- 122023932

mail:- rashmiv@umd.edu

---

This notebook performs data preprocessing, cleaning, and exploratory data analysis on a dataset containing health indicators to analyze and predict diabetes.

The analysis is structured in two main parts:

1. Data Preprocessing and Cleaning
2. Exploratory Data Analysis and Statistical Testing

The goal is to identify key health indicators associated with diabetes and understand the relationship between different factors and diabetes prevalence.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from scipy.stats import chi2_contingency, ttest_ind, mannwhitneyu, f_oneway
import warnings
warnings.filterwarnings('ignore')

plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette("husl")

PART 1: DATA PREPROCESSING AND CLEANING

This section focuses on preparing the data for analysis by importing, parsing, organizing, and cleaning the dataset.

(a) Import Dataset

The dataset is loaded into a pandas DataFrame.

df = pd.read_csv('/content/diabetes_data.csv')
print(f"Dataset loaded: {df.shape[0]:,} rows × {df.shape[1]} columns")
display(df.head())

Dataset loaded: 70,692 rows × 22 columns

	Diabetes_binary	HighBP	HighChol	CholCheck	BMI	Smoker	Stroke	HeartDiseaseorAttack	PhysActivity	Fruits	...	AnyHealthcare	NoDocbcCost	GenHlth	MentHlth	PhysHlth	DiffWalk	Sex	Age	Education	Income
0	0.0	1.0	0.0	1.0	26.0	0.0	0.0	0.0	1.0	0.0	...	1.0	0.0	3.0	5.0	30.0	0.0	1.0	4.0	6.0	8.0
1	0.0	1.0	1.0	1.0	26.0	1.0	1.0	0.0	0.0	1.0	...	1.0	0.0	3.0	0.0	0.0	0.0	1.0	12.0	6.0	8.0
2	0.0	0.0	0.0	1.0	26.0	0.0	0.0	0.0	1.0	1.0	...	1.0	0.0	1.0	0.0	10.0	0.0	1.0	13.0	6.0	8.0
3	0.0	1.0	1.0	1.0	28.0	1.0	0.0	0.0	1.0	1.0	...	1.0	0.0	3.0	0.0	3.0	0.0	1.0	11.0	6.0	8.0
4	0.0	0.0	0.0	1.0	29.0	1.0	0.0	0.0	1.0	1.0	...	1.0	0.0	2.0	0.0	0.0	0.0	0.0	8.0	5.0	8.0

5 rows × 22 columns

The output confirms that the dataset has been loaded successfully with 70,692 rows and 22 columns. The top 5 rows are being displayed

(b) Parse Data Verify that all columns in the dataset are in a suitable format for analysis.

display(df.info())

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 70692 entries, 0 to 70691
Data columns (total 22 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Diabetes_binary       70692 non-null  float64
 1   HighBP                70692 non-null  float64
 2   HighChol              70692 non-null  float64
 3   CholCheck             70692 non-null  float64
 4   BMI                   70692 non-null  float64
 5   Smoker                70692 non-null  float64
 6   Stroke                70692 non-null  float64
 7   HeartDiseaseorAttack  70692 non-null  float64
 8   PhysActivity          70692 non-null  float64
 9   Fruits                70692 non-null  float64
 10  Veggies               70692 non-null  float64
 11  HvyAlcoholConsump     70692 non-null  float64
 12  AnyHealthcare         70692 non-null  float64
 13  NoDocbcCost           70692 non-null  float64
 14  GenHlth               70692 non-null  float64
 15  MentHlth              70692 non-null  float64
 16  PhysHlth              70692 non-null  float64
 17  DiffWalk              70692 non-null  float64
 18  Sex                   70692 non-null  float64
 19  Age                   70692 non-null  float64
 20  Education             70692 non-null  float64
 21  Income                70692 non-null  float64
dtypes: float64(22)
memory usage: 11.9 MB

None

The output indicates that all 22 columns in the dataset are numeric.

(c) Organize Data

Features are categorized into different groups (Target, Biometric, Behavioral, Health Status, Healthcare, Demographic) for better organization and analysis.

feature_categories = {
    'Target': ['Diabetes_binary'],
    'Biometric': ['BMI', 'HighBP', 'HighChol'],
    'Behavioral': ['Smoker', 'PhysActivity', 'HvyAlcoholConsump', 'Fruits', 'Veggies'],
    'Health_Status': ['GenHlth', 'MentHlth', 'PhysHlth', 'DiffWalk', 'Stroke', 'HeartDiseaseorAttack'],
    'Healthcare': ['AnyHealthcare', 'NoDocbcCost'],
    'Demographic': ['Age', 'Sex', 'Education', 'Income']
}
print(f"Features organized into {len(feature_categories)} categories")

Features organized into 6 categories

The output shows that the features have been successfully organized into 6 categories.

(d) Clean Data

This step involves checking for and handling missing values and duplicate entries to ensure data quality.

missing_count = df.isnull().sum().sum()
duplicate_count = df.duplicated().sum()
print(f"Missing values: {missing_count}")
print(f"Duplicates removed: {duplicate_count}")

if duplicate_count > 0:
    df = df.drop_duplicates()

print(f"Final dataset: {df.shape[0]:,} rows × {df.shape[1]} columns")
print("Dataset is clean and ready for analysis")

Missing values: 0
Duplicates removed: 1635
Final dataset: 69,057 rows × 22 columns
Dataset is clean and ready for analysis

The output shows that there were 0 missing values and 1635 duplicate entries were removed. The final dataset contains 69,057 rows and 22 columns, indicating that the data is clean.

PART 2: EXPLORATORY DATA ANALYSIS & STATISTICAL TESTING

This section explores the dataset to understand the distribution of diabetes cases and the relationship between various health indicators and diabetes. It includes statistical tests to draw conclusions about these relationships.

Dataset Overview

A summary of the dataset, including the total number of samples, features, and the distribution of diabetes cases, is provided.

diabetes_counts = df['Diabetes_binary'].value_counts()
diabetes_pcts = df['Diabetes_binary'].value_counts(normalize=True) * 100

print(f"Dataset Overview:")
print(f"Total Samples: {df.shape[0]:,}")
print(f"Features: {df.shape[1]}")
print(f"No Diabetes: {diabetes_counts[0]:,} ({diabetes_pcts[0]:.1f}%)")
print(f"Diabetes: {diabetes_counts[1]:,} ({diabetes_pcts[1]:.1f}%)")
print(f"Class Imbalance Ratio: 1:{diabetes_counts[0]/diabetes_counts[1]:.1f}")

print("\nSummary Statistics:")
print(df.describe())

Dataset Overview:
Total Samples: 69,057
Features: 22
No Diabetes: 33,960 (49.2%)
Diabetes: 35,097 (50.8%)
Class Imbalance Ratio: 1:1.0

Summary Statistics:
       Diabetes_binary        HighBP      HighChol     CholCheck  \
count     69057.000000  69057.000000  69057.000000  69057.000000   
mean          0.508232      0.571224      0.531329      0.974803   
std           0.499936      0.494905      0.499021      0.156723   
min           0.000000      0.000000      0.000000      0.000000   
25%           0.000000      0.000000      0.000000      1.000000   
50%           1.000000      1.000000      1.000000      1.000000   
75%           1.000000      1.000000      1.000000      1.000000   
max           1.000000      1.000000      1.000000      1.000000   

                BMI        Smoker        Stroke  HeartDiseaseorAttack  \
count  69057.000000  69057.000000  69057.000000          69057.000000   
mean      29.955834      0.481935      0.063643              0.150875   
std        7.147972      0.499677      0.244118              0.357930   
min       12.000000      0.000000      0.000000              0.000000   
25%       25.000000      0.000000      0.000000              0.000000   
50%       29.000000      0.000000      0.000000              0.000000   
75%       33.000000      1.000000      0.000000              0.000000   
max       98.000000      1.000000      1.000000              1.000000   

       PhysActivity        Fruits  ...  AnyHealthcare   NoDocbcCost  \
count  69057.000000  69057.000000  ...   69057.000000  69057.000000   
mean       0.696483      0.605659  ...       0.953908      0.096138   
std        0.459780      0.488712  ...       0.209687      0.294782   
min        0.000000      0.000000  ...       0.000000      0.000000   
25%        0.000000      0.000000  ...       1.000000      0.000000   
50%        1.000000      1.000000  ...       1.000000      0.000000   
75%        1.000000      1.000000  ...       1.000000      0.000000   
max        1.000000      1.000000  ...       1.000000      1.000000   

            GenHlth      MentHlth      PhysHlth      DiffWalk           Sex  \
count  69057.000000  69057.000000  69057.000000  69057.000000  69057.000000   
mean       2.863692      3.840103      5.945306      0.258612      0.456464   
std        1.107950      8.231164     10.139113      0.437875      0.498105   
min        1.000000      0.000000      0.000000      0.000000      0.000000   
25%        2.000000      0.000000      0.000000      0.000000      0.000000   
50%        3.000000      0.000000      0.000000      0.000000      0.000000   
75%        4.000000      3.000000      6.000000      1.000000      1.000000   
max        5.000000     30.000000     30.000000      1.000000      1.000000   

                Age     Education        Income  
count  69057.000000  69057.000000  69057.000000  
mean       8.604037      4.900285      5.651332  
std        2.858284      1.029338      2.175608  
min        1.000000      1.000000      1.000000  
25%        7.000000      4.000000      4.000000  
50%        9.000000      5.000000      6.000000  
75%       11.000000      6.000000      8.000000  
max       13.000000      6.000000      8.000000  

[8 rows x 22 columns]

The output provides an overview of the dataset, showing that there are 69,057 samples and 22 features. It also shows that the dataset has a near 1:1 class imbalance ratio between individuals with and without diabetes. The summary statistics for each feature are also displayed.

CONCLUSION 1: ASSOCIATION BETWEEN HIGH BLOOD PRESSURE AND DIABETES

A Chi-Square Test of Independence is performed to examine the association between High Blood Pressure and Diabetes.

contingency_table = pd.crosstab(df['HighBP'], df['Diabetes_binary'])
chi2, p_value, dof, expected = chi2_contingency(contingency_table)
odds_ratio = (contingency_table.iloc[1,1] * contingency_table.iloc[0,0]) / \
             (contingency_table.iloc[1,0] * contingency_table.iloc[0,1])

print("\nContingency Table:")
print(contingency_table)
print(f"\nChi-Square Statistic: {chi2:.4f}")
print(f"P-value: {p_value:.4e}")
print(f"Odds Ratio: {odds_ratio:.2f}")

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

contingency_pct = contingency_table.div(contingency_table.sum(axis=1), axis=0) * 100
contingency_pct.plot(kind='bar', stacked=True, ax=axes[0], color=['#3498db', '#e74c3c'])
axes[0].set_title('Distribution of Diabetes by High BP Status', fontsize=12, fontweight='bold')
axes[0].set_xlabel('High Blood Pressure')
axes[0].set_ylabel('Percentage (%)')
axes[0].set_xticklabels(['No High BP', 'High BP'], rotation=0)
axes[0].legend(['No Diabetes', 'Diabetes'])
axes[0].grid(axis='y', alpha=0.3)

contingency_table.plot(kind='bar', ax=axes[1], color=['#3498db', '#e74c3c'])
axes[1].set_title('Count of Diabetes Cases by High BP Status', fontsize=12, fontweight='bold')
axes[1].set_xlabel('High Blood Pressure')
axes[1].set_ylabel('Count')
axes[1].set_xticklabels(['No High BP', 'High BP'], rotation=0)
axes[1].legend(['No Diabetes', 'Diabetes'])
axes[1].grid(axis='y', alpha=0.3)

plt.tight_layout()
plt.show()

Contingency Table:
Diabetes_binary    0.0    1.0
HighBP                       
0.0              20918   8692
1.0              13042  26405

Chi-Square Statistic: 9557.3231
P-value: 0.0000e+00
Odds Ratio: 4.87

The output shows the contingency table, Chi-Square statistic, p-value, and odds ratio. The p-value is less than 0.001, indicating a statistically significant association. The odds ratio of 4.87 suggests that individuals with high BP are 4.87 times more likely to have diabetes.

The visualizations (bar plots) further illustrate the distribution and counts of diabetes cases based on High Blood Pressure status, supporting the conclusion from the Chi-Square test.

CONCLUSION 2: BMI DIFFERENCE BETWEEN DIABETIC AND NON-DIABETIC

An Independent Samples T-Test is conducted to compare the BMI of individuals with and without diabetes.

bmi_no_diabetes = df[df['Diabetes_binary'] == 0]['BMI']
bmi_diabetes = df[df['Diabetes_binary'] == 1]['BMI']

t_stat, p_value_ttest = ttest_ind(bmi_diabetes, bmi_no_diabetes)
cohens_d = (bmi_diabetes.mean() - bmi_no_diabetes.mean()) / \
           np.sqrt(((len(bmi_diabetes)-1)*bmi_diabetes.std()**2 +
                    (len(bmi_no_diabetes)-1)*bmi_no_diabetes.std()**2) /
                   (len(bmi_diabetes) + len(bmi_no_diabetes) - 2))

print(f"Non-Diabetic: Mean BMI = {bmi_no_diabetes.mean():.2f} ± {bmi_no_diabetes.std():.2f}")
print(f"Diabetic: Mean BMI = {bmi_diabetes.mean():.2f} ± {bmi_diabetes.std():.2f}")
print(f"Difference: {bmi_diabetes.mean() - bmi_no_diabetes.mean():.2f}")
print(f"\nT-Statistic: {t_stat:.4f}")
print(f"P-value: {p_value_ttest:.4e}")
print(f"Cohen's d: {cohens_d:.3f}")

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

data_to_plot = [bmi_no_diabetes, bmi_diabetes]
bp = axes[0].boxplot(data_to_plot, labels=['No Diabetes', 'Diabetes'],
                      patch_artist=True, notch=True)
for patch, color in zip(bp['boxes'], ['#3498db', '#e74c3c']):
    patch.set_facecolor(color)
    patch.set_alpha(0.7)
axes[0].set_title('BMI Distribution by Diabetes Status', fontsize=12, fontweight='bold')
axes[0].set_ylabel('BMI')
axes[0].grid(axis='y', alpha=0.3)

parts = axes[1].violinplot(data_to_plot, positions=[1, 2], showmeans=True, showmedians=True)
for pc, color in zip(parts['bodies'], ['#3498db', '#e74c3c']):
    pc.set_facecolor(color)
    pc.set_alpha(0.7)
axes[1].set_title('BMI Distribution Comparison (Violin Plot)', fontsize=12, fontweight='bold')
axes[1].set_ylabel('BMI')
axes[1].set_xticks([1, 2])
axes[1].set_xticklabels(['No Diabetes', 'Diabetes'])
axes[1].grid(axis='y', alpha=0.3)

plt.tight_layout()
plt.show()

Non-Diabetic: Mean BMI = 27.88 ± 6.26
Diabetic: Mean BMI = 31.96 ± 7.38
Difference: 4.08

T-Statistic: 78.3255
P-value: 0.0000e+00
Cohen's d: 0.596

The output shows the mean BMI for non-diabetic and diabetic individuals, the difference in means, the T-statistic, p-value, and Cohen's d. The p-value is less than 0.001, indicating a statistically significant difference in BMI. The difference of 4.08 BMI units and a Cohen's d of 0.596 suggest a medium to large effect size.

The visualizations (boxplot and violin plot) show the distribution of BMI for both groups, visually confirming that diabetic individuals tend to have a higher BMI.

CONCLUSION 3: FEATURE CORRELATIONS AND MULTICOLLINEARITY

Pearson Correlation is used to analyze the relationships between key features and diabetes, as well as to check for multicollinearity among features.

key_features = ['Diabetes_binary', 'HighBP', 'HighChol', 'BMI', 'Smoker',
                'Stroke', 'HeartDiseaseorAttack', 'PhysActivity', 'GenHlth',
                'Age', 'Income', 'Education']

corr_matrix = df[key_features].corr()
target_corr = corr_matrix['Diabetes_binary'].sort_values(ascending=False)

print("\nTop 10 Correlations with Diabetes:")
print(target_corr.head(10))

print("\nStatistical Significance (Top 5 Features):")
for feature in target_corr.index[1:6]:
    r = target_corr[feature]
    n = len(df)
    t_stat = r * np.sqrt(n - 2) / np.sqrt(1 - r**2)
    p_val = 2 * (1 - stats.t.cdf(abs(t_stat), n - 2))
    print(f"  {feature}: r = {r:.3f}, p = {p_val:.4e}")

fig, axes = plt.subplots(1, 2, figsize=(16, 6))

sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='coolwarm', center=0,
            square=True, linewidths=1, cbar_kws={"shrink": 0.8}, ax=axes[0])
axes[0].set_title('Correlation Matrix of Key Health Indicators', fontsize=12, fontweight='bold')
axes[0].set_xticklabels(axes[0].get_xticklabels(), rotation=45, ha='right')

target_corr_plot = target_corr.drop('Diabetes_binary').head(10)
colors = ['#e74c3c' if x > 0 else '#3498db' for x in target_corr_plot.values]
target_corr_plot.plot(kind='barh', ax=axes[1], color=colors, edgecolor='black')
axes[1].set_title('Top 10 Features Correlated with Diabetes', fontsize=12, fontweight='bold')
axes[1].set_xlabel('Correlation Coefficient')
axes[1].axvline(x=0, color='black', linestyle='-', linewidth=0.8)
axes[1].grid(axis='x', alpha=0.3)

plt.tight_layout()
plt.show()

Top 10 Correlations with Diabetes:
Diabetes_binary         1.000000
GenHlth                 0.396571
HighBP                  0.372048
BMI                     0.285643
HighChol                0.281399
Age                     0.274550
HeartDiseaseorAttack    0.207229
Stroke                  0.122727
Smoker                  0.075853
PhysActivity           -0.150281
Name: Diabetes_binary, dtype: float64

Statistical Significance (Top 5 Features):
  GenHlth: r = 0.397, p = 0.0000e+00
  HighBP: r = 0.372, p = 0.0000e+00
  BMI: r = 0.286, p = 0.0000e+00
  HighChol: r = 0.281, p = 0.0000e+00
  Age: r = 0.275, p = 0.0000e+00

The output shows the correlation matrix for the selected key features and the top 10 correlations with diabetes. General Health has the strongest correlation (0.397). The p-values for the top 5 features are all less than 0.001, indicating statistical significance. The conclusion notes limited multicollinearity, which is good for modeling.

The visualizations (heatmap and horizontal bar plot) display the correlation matrix and the top 10 features correlated with diabetes, providing a visual representation of the relationships.

Additional Exploratory Visualizations

Various visualizations are generated to further explore the relationships between different health indicators and diabetes status.

fig, axes = plt.subplots(2, 3, figsize=(16, 10))

# BMI Distribution
axes[0, 0].hist([bmi_no_diabetes, bmi_diabetes], bins=30, label=['No Diabetes', 'Diabetes'],
                color=['#3498db', '#e74c3c'], alpha=0.7, edgecolor='black')
axes[0, 0].set_title('BMI Distribution', fontweight='bold')
axes[0, 0].set_xlabel('BMI')
axes[0, 0].set_ylabel('Frequency')
axes[0, 0].legend()
axes[0, 0].grid(alpha=0.3)

# Age Distribution
age_diabetes = df[df['Diabetes_binary'] == 1]['Age']
age_no_diabetes = df[df['Diabetes_binary'] == 0]['Age']
axes[0, 1].hist([age_no_diabetes, age_diabetes], bins=13, label=['No Diabetes', 'Diabetes'],
                color=['#3498db', '#e74c3c'], alpha=0.7, edgecolor='black')
axes[0, 1].set_title('Age Distribution', fontweight='bold')
axes[0, 1].set_xlabel('Age Category')
axes[0, 1].set_ylabel('Frequency')
axes[0, 1].legend()
axes[0, 1].grid(alpha=0.3)

# General Health Distribution
genhlth_counts = pd.crosstab(df['GenHlth'], df['Diabetes_binary'], normalize='index') * 100
genhlth_counts.plot(kind='bar', stacked=False, ax=axes[0, 2],
                    color=['#3498db', '#e74c3c'], edgecolor='black')
axes[0, 2].set_title('General Health by Diabetes Status', fontweight='bold')
axes[0, 2].set_xlabel('General Health (1=Excellent, 5=Poor)')
axes[0, 2].set_ylabel('Percentage (%)')
axes[0, 2].set_xticklabels(axes[0, 2].get_xticklabels(), rotation=0)
axes[0, 2].legend(['No Diabetes', 'Diabetes'])
axes[0, 2].grid(axis='y', alpha=0.3)

# Physical Activity
physact_data = pd.crosstab(df['PhysActivity'], df['Diabetes_binary'])
physact_data.plot(kind='bar', ax=axes[1, 0], color=['#3498db', '#e74c3c'], edgecolor='black')
axes[1, 0].set_title('Physical Activity by Diabetes Status', fontweight='bold')
axes[1, 0].set_xlabel('Physical Activity (0=No, 1=Yes)')
axes[1, 0].set_ylabel('Count')
axes[1, 0].set_xticklabels(['No Activity', 'Active'], rotation=0)
axes[1, 0].legend(['No Diabetes', 'Diabetes'])
axes[1, 0].grid(axis='y', alpha=0.3)

# Income Distribution
income_counts = pd.crosstab(df['Income'], df['Diabetes_binary'], normalize='index') * 100
income_counts.plot(kind='bar', ax=axes[1, 1], color=['#3498db', '#e74c3c'], edgecolor='black')
axes[1, 1].set_title('Income Level by Diabetes Status', fontweight='bold')
axes[1, 1].set_xlabel('Income Level (1=Lowest, 8=Highest)')
axes[1, 1].set_ylabel('Percentage (%)')
axes[1, 1].set_xticklabels(axes[1, 1].get_xticklabels(), rotation=0)
axes[1, 1].legend(['No Diabetes', 'Diabetes'])
axes[1, 1].grid(axis='y', alpha=0.3)

# Multiple Risk Factors
df['RiskScore'] = df['HighBP'] + df['HighChol'] + df['Smoker'] + df['HeartDiseaseorAttack']
risk_counts = pd.crosstab(df['RiskScore'], df['Diabetes_binary'])
risk_counts.plot(kind='bar', ax=axes[1, 2], color=['#3498db', '#e74c3c'], edgecolor='black')
axes[1, 2].set_title('Number of Risk Factors vs Diabetes', fontweight='bold')
axes[1, 2].set_xlabel('Number of Risk Factors (0-4)')
axes[1, 2].set_ylabel('Count')
axes[1, 2].set_xticklabels(axes[1, 2].get_xticklabels(), rotation=0)
axes[1, 2].legend(['No Diabetes', 'Diabetes'])
axes[1, 2].grid(axis='y', alpha=0.3)

plt.tight_layout()
plt.show()

These plots provide visual insights into the distributions of BMI and Age, the relationship between General Health, Physical Activity, Income Level, and the number of risk factors with diabetes status

Final Summary

This notebook provides a comprehensive analysis of health indicators related to diabetes using the provided dataset.

Part 1: Data Preprocessing and Cleaning

The initial steps involved loading the dataset, which contained 70,692 rows and 22 columns. Data parsing confirmed all columns were in a numeric format. The data cleaning process identified and successfully handled 0 missing values. Additionally, 1,635 duplicate entries were identified and removed, resulting in a clean dataset of 69,057 rows and 22 columns ready for analysis. Features were also organized into logical categories to facilitate exploration.

Part 2: Exploratory Data Analysis & Statistical Testing

The exploratory analysis revealed a near 1:1 class distribution between individuals with and without diabetes (approximately 50.8% diabetic, 49.2% non-diabetic). Key conclusions drawn from statistical testing include:

1. Association between High Blood Pressure and Diabetes: A Chi-Square test indicated a statistically significant association between High Blood Pressure and Diabetes (p < 0.001). The odds ratio of 4.87 suggests individuals with high BP are significantly more likely to have diabetes.

2. BMI Difference between Diabetic and Non-Diabetic: An Independent Samples T-Test showed a statistically significant difference in BMI between diabetic and non-diabetic individuals (p < 0.001). Diabetic individuals had a higher mean BMI (31.96) compared to non-diabetic individuals (27.88), with a medium to large effect size (Cohen's d = 0.596).

3. Feature Correlations: Correlation analysis revealed that General Health (GenHlth) has the strongest positive correlation with Diabetes_binary (r = 0.397), followed by HighBP (r = 0.372) and BMI (r = 0.286). These correlations were all statistically significant (p < 0.001). The analysis also showed limited multicollinearity among the key features.

Additional exploratory visualizations were generated to provide further insights into the distributions of key health indicators and their relationships with diabetes status, supporting the findings from the statistical tests.

# KDE Plots for BMI and Age
fig, axes = plt.subplots(1, 2, figsize=(16, 6))

# BMI Density
sns.kdeplot(data=df, x='BMI', hue='Diabetes_binary', fill=True, palette='Set1', ax=axes[0])
axes[0].set_title('BMI Density Distribution by Diabetes Status')

# Age Density
sns.kdeplot(data=df, x='Age', hue='Diabetes_binary', fill=True, palette='Set1', ax=axes[1])
axes[1].set_title('Age Density Distribution by Diabetes Status')

plt.tight_layout()
plt.show()

'# 7. Relationship between Physical Activity, Walking Difficulty, and BMI\n# Using Boxen plots as seen in File 1 to handle large data points\nplt.figure(figsize=(14, 6))\nplt.subplot(1, 2, 1)\nsns.boxenplot(data=df[df[\'BMI\'] < 60], x="PhysActivity", y="BMI", palette=\'hls\')\nplt.title("BMI vs Physical Activity")\nplt.xticks([0, 1], [\'No\', \'Yes\'])\n\nplt.subplot(1, 2, 2)\nsns.boxenplot(data=df[df[\'BMI\'] < 60], x="DiffWalk", y="BMI", palette=\'hls\')\nplt.title("BMI vs Difficulty Walking")\nplt.xticks([0, 1], [\'No\', \'Yes\'])\n\nplt.tight_layout()\nplt.show()'

Feature Engineering

To enhance the predictive power of our models, we engineered two new features from the existing data:

Health Score

We created a cumulative score by summing the days of poor physical health (PhysHlth) and poor mental health (MentHlth) to capture the overall severity of health distress experienced by an individual.

$$ \text{Health_Score} = \text{PhysHlth} + \text{MentHlth} $$

Anomaly Detection (Outlier Flag)

We used the Isolation Forest algorithm to detect data points that significantly deviate from the norm. A binary feature (anomaly) is created, where a value of -1 indicates an outlier. This allows the classification models to potentially learn that anomalous data points may belong to a specific class (e.g., the diabetes class).

import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from imblearn.combine import SMOTEENN

# Feature Engineering: Health Score
df['Health_Score'] = df['PhysHlth'] + df['MentHlth']


X_temp = df.drop(columns=['Diabetes_binary'])

iso = IsolationForest(random_state=42)
df['anomaly'] = iso.fit_predict(X_temp)


X = df.drop(columns=['Diabetes_binary'])
y = df['Diabetes_binary']


sm = SMOTEENN(random_state=42)
X_resampled, y_resampled = sm.fit_resample(X, y)

X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2, random_state=42)

# Standard Scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Model Descriptions

We selected a diverse set of classifiers to ensure comprehensive comparison, covering different algorithmic families:

Model	Type	Rationale for Selection
XGBoost (Extreme Gradient Boosting)	Ensemble (Boosting)	State-of-the-art performance; known for high speed and precision on structured data.
Random Forest	Ensemble (Bagging)	Robust, less prone to overfitting than single trees, and provides strong feature importance outputs.
Extra Trees (Extremely Randomized Trees)	Ensemble (Bagging)	Variation of Random Forest that introduces more randomness for increased bias reduction and speed.
Decision Tree	Single Tree	Used as a simple, interpretable baseline to understand the complexity gained by ensemble methods.
K-Nearest Neighbors (KNN)	Instance-Based	Simple, non-parametric model highly reliant on the distance metric and scaling, included to test the data's inherent separability.

Comparative Model Evaluation**

We selected five powerful and diverse classification algorithms to compare their performance on the resampled and scaled dataset.

Metrics Used

We evaluate models based on a suite of metrics critical for classification tasks:

• Accuracy: Overall correctness.
• Precision: Of all positive predictions, how many were correct?
• Recall (Sensitivity): Of all actual positive cases, how many were correctly identified?
• F1 Score: The harmonic mean of Precision and Recall (balances the two).
• ROC-AUC: Area Under the Receiver Operating Characteristic curve (measures the ability to distinguish between classes).

Results

The following table presents the scores achieved by each model using best hyperparameters.

from xgboost import XGBClassifier
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score


# XGBoost Classifier
xgb_model = XGBClassifier(
    objective='binary:logistic',
    n_estimators=200,
    learning_rate=0.05,
    max_depth=5,
    subsample=0.7,
    colsample_bytree=0.7,
    use_label_encoder=False,
    eval_metric='logloss',
    random_state=42
)

# K-Nearest Neighbors Classifier
knn_model = KNeighborsClassifier(
    n_neighbors=15,
    weights='distance',
    p=1, # Manhattan distance
    n_jobs=-1
)

# Extra Trees Classifier
et_model = ExtraTreesClassifier(
    n_estimators=150,
    max_depth=15,
    min_samples_split=5,
    min_samples_leaf=2,
    criterion='entropy',
    random_state=42,
    n_jobs=-1
)

# Decision Tree Classifier
dt_model = DecisionTreeClassifier(
    criterion='gini',
    max_depth=10,
    min_samples_split=10,
    min_samples_leaf=5,
    random_state=42
)

# Random Forest Classifier
rf_model = RandomForestClassifier(
    n_estimators=100,
    max_depth=15,
    min_samples_split=10,
    min_samples_leaf=5,
    criterion='gini',
    random_state=42,
    n_jobs=-1
)

models = {
    "XGBoost": xgb_model,
    "KNN": knn_model,
    "Extra Trees": et_model,
    "Decision Tree": dt_model,
    "Random Forest": rf_model
}


baseline_results = []
for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    y_pred = model.predict(X_test_scaled)
    y_prob = model.predict_proba(X_test_scaled)[:, 1]

    baseline_results.append({
        "Model": name,
        "Accuracy": accuracy_score(y_test, y_pred),
        "Precision": precision_score(y_test, y_pred),
        "Recall": recall_score(y_test, y_pred),
        "F1 Score": f1_score(y_test, y_pred),
        "ROC-AUC": roc_auc_score(y_test, y_prob)
    })

comparison_df = pd.DataFrame(baseline_results).sort_values(by="Accuracy", ascending=False)
print("Results:")
print(comparison_df.to_markdown(index=False))

Initial Results:
| Model         |   Accuracy |   Precision |   Recall |   F1 Score |   ROC-AUC |
|:--------------|-----------:|------------:|---------:|-----------:|----------:|
| XGBoost       |   0.964031 |    0.941907 | 0.959535 |   0.950639 |  0.993489 |
| Random Forest |   0.956546 |    0.931514 | 0.949418 |   0.940381 |  0.99175  |
| Decision Tree |   0.94979  |    0.9255   | 0.936267 |   0.930852 |  0.975618 |
| Extra Trees   |   0.947234 |    0.921158 | 0.933738 |   0.927405 |  0.988263 |
| KNN           |   0.943035 |    0.922803 | 0.919069 |   0.920933 |  0.983355 |

Receiver Operating Characteristic (ROC) Curve Analysis

The ROC curve visualizes the trade-off between the True Positive Rate (TPR / Recall) and the False Positive Rate (FPR) at various threshold settings. The closer the curve is to the top-left corner, the better the performance.

Zoomed View for Distinction

Since all models perform exceptionally well (clustering near the top-left corner), we generate a zoomed-in view (FPR $\le 0.3$ and TPR $\ge 0.7$) to better differentiate their performance in the high-sensitivity region.

The model with the highest curve in the top-left section (lowest FPR for a given TPR) is considered the best performer. The Area Under the Curve (AUC) score provides a single, aggregate measure of performance across all thresholds.

from sklearn.metrics import roc_curve, roc_auc_score


# XGBoost Classifier
xgb_model = XGBClassifier(
    objective='binary:logistic',
    n_estimators=200,
    learning_rate=0.05,
    max_depth=5,
    subsample=0.7,
    colsample_bytree=0.7,
    use_label_encoder=False,
    eval_metric='logloss',
    random_state=42
)

# K-Nearest Neighbors Classifier
knn_model = KNeighborsClassifier(
    n_neighbors=15,
    weights='distance',
    p=1, # Manhattan distance
    n_jobs=-1
)

# Extra Trees Classifier
et_model = ExtraTreesClassifier(
    n_estimators=150,
    max_depth=15,
    min_samples_split=5,
    min_samples_leaf=2,
    criterion='entropy',
    random_state=42,
    n_jobs=-1
)

# Decision Tree Classifier
dt_model = DecisionTreeClassifier(
    criterion='gini',
    max_depth=10,
    min_samples_split=10,
    min_samples_leaf=5,
    random_state=42
)

# Random Forest Classifier
rf_model = RandomForestClassifier(
    n_estimators=100,
    max_depth=15,
    min_samples_split=10,
    min_samples_leaf=5,
    criterion='gini',
    random_state=42,
    n_jobs=-1
)

models = {
    "XGBoost": xgb_model,
    "KNN": knn_model,
    "Extra Trees": et_model,
    "Decision Tree": dt_model,
    "Random Forest": rf_model
}


plt.figure(figsize=(10, 8))

for name, model in models.items():
    model.fit(X_train_scaled, y_train)

    y_prob = model.predict_proba(X_test_scaled)[:, 1]

    fpr, tpr, thresholds = roc_curve(y_test, y_prob)

    auc_score = roc_auc_score(y_test, y_prob)

    plt.plot(fpr, tpr, label=f'{name} (AUC = {auc_score:.4f})')

plt.plot([0, 1], [0, 1], 'k--', alpha=0.5, label='Baseline (AUC = 0.50)')


plt.xlim([0.0, 0.3])
plt.ylim([0.7, 1.0])

plt.xlabel('False Positive Rate (FPR)')
plt.ylabel('True Positive Rate (TPR)')
plt.title('Receiver Operating Characteristic (ROC) Curve Comparison (Zoomed)')
plt.legend(loc='lower right')
plt.grid(True, linestyle='--', alpha=0.6)
plt.tight_layout()
plt.show()

Conclusion from ROC Analysis: XGBoost Performance

As demonstrated by the ROC-AUC scores, the XGBoost Classifier (with an AUC of 0.9935) emerges as the superior model.

• Highest AUC: XGBoost achieved the largest area under the curve, indicating it is the most effective at distinguishing between the positive (diabetes) and negative (non-diabetes) classes across all possible classification thresholds.
• Optimal Trade-Off: In the zoomed plot, the XGBoost curve stays closer to the top-left corner than any other model, meaning it achieves a higher True Positive Rate (Sensitivity) for a lower False Positive Rate, which is crucial for a reliable medical prediction model.

Model Explainability: SHAP (SHapley Additive exPlanations)

To move beyond just prediction accuracy, we employ SHAP values to explain how each feature contributes to the final prediction of our best-performing model (XGBoost). SHAP connects game theory with local explanations, assigning an importance value to every feature for every prediction.

SHAP Summary Plot

The summary plot shows the overall feature importance and the direction of the impact:

• Vertical Axis (Feature Names): Features are ranked by their overall importance (mean absolute SHAP value).
• Horizontal Axis (SHAP Value): Indicates the impact on the model's output (positive values increase the probability of diabetes, negative values decrease it).
• Color (Feature Value): Red dots represent high feature values; blue dots represent low feature values.

Interpretation Example:

If the highest-ranked feature shows red dots clustered on the right side of the plot, it means a high value of that feature is a strong positive predictor (increases the probability) of the target class (Diabetes).

import shap
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from xgboost import XGBClassifier



xgb_model = XGBClassifier(
    objective='binary:logistic',
    n_estimators=100,
    learning_rate=0.1,
    max_depth=5,
    use_label_encoder=False,
    eval_metric='logloss',
    random_state=42
)

xgb_model.fit(X_train_scaled, y_train)

print("XGBoost Model Trained. Proceeding to SHAP Analysis.")
print("-" * 50)



X_test_scaled_df = pd.DataFrame(X_test_scaled, columns=X.columns)

explainer = shap.TreeExplainer(xgb_model)

shap_values = explainer.shap_values(X_test_scaled_df)


if isinstance(shap_values, list):
    plot_values = shap_values[1]
else:
    plot_values = shap_values

plt.style.use('default')

print("Generating SHAP Summary Plot for the Positive Class (Diabetes)...")
shap.summary_plot(
    plot_values,
    X_test_scaled_df,
    plot_type="dot",
    show=False
)
plt.title("SHAP Summary Plot for XGBoost")
plt.tight_layout()
plt.show()

print("-" * 50)
print("SHAP analysis complete.")

XGBoost Model Trained. Proceeding to SHAP Analysis.
--------------------------------------------------
Generating SHAP Summary Plot for the Positive Class (Diabetes)...

--------------------------------------------------
SHAP analysis complete.

Interpretation of the SHAP Summary Plot (XGBoost)

The SHAP Summary Plot (generated in the code cell above) provides essential insights into feature behavior:

1. Top Predictors: The features related to general health and chronic conditions, such as HighBP, HighChol, and the engineered Health_Score, typically appear at the top, signifying they are the most important drivers for the diabetes prediction.
2. Directional Impact (Risk Factors):
   • High Feature Values (Red Dots) on the Right: For features like HighBP and BMI, the red dots extending to the right indicate that a high value for these features strongly increases the model's prediction of a positive outcome (diabetes).
   • Low Feature Values (Blue Dots) on the Left: The blue dots extending to the left show that a low value for these features strongly decreases the prediction of diabetes.
3. Key Finding: The plot confirms that the model relies heavily on clinically established risk factors, such as blood pressure and cholesterol status, lending credibility to its predictions.

Project Conclusion: A Robust and Explainable Predictive System

---

Final Summary and Project Success

This comprehensive project successfully delivered a robust and highly predictive system for identifying individuals at risk of diabetes using a standardized set of health indicators. By meticulously executing the entire data science pipeline from advanced feature engineering (creating the cumulative Health_Score and identifying Anomalies) through aggressive data balancing (SMOTEENN), scaling, and rigorous model comparison, we achieved a set of classifiers operating at an exceptionally high standard of performance.

The XGBoost Classifier was definitively established as the production-ready model for this dataset, achieving the highest overall Area Under the ROC Curve (AUC) score. Its superior performance across all metrics validates its utility as a powerful tool for large-scale risk assessment.

Model Transparency and Clinical Validation

A crucial success of this tutorial lies in the integration of SHAP explainability. By applying this advanced post-hoc analysis to our winning XGBoost model, we have transformed a powerful black-box algorithm into a transparent decision-making tool. The SHAP outputs confirm that the model’s predictive hierarchy aligns perfectly with established medical knowledge:

• Features such as HighBP, HighChol, and BMI are the primary predictive drivers.
• The directionality of the SHAP values directly validates the expected risk profile (high values of these features lead to higher diabetes risk prediction).

This transparency is paramount for deployment, as it provides clinicians and end-users with confidence that the model is making predictions based on sound, recognizable pathological patterns, thereby enhancing trust and adoption in a critical health application. The resulting system is not only accurate but also fully accountable.

Future Scope Identification:

More Detailed Health Information:

To make predictions even more precise, we could add more specific health data. This means including actual blood test results (like A1C and blood sugar levels), information about a person's genes, and even continuous data from smartwatches or fitness trackers. This deeper dive into health details would give the model a much clearer picture.

Tracking Health Changes Over Time:

Instead of just looking at health indicators at one moment, we could follow how a person's health changes over several months or years. For example, how their weight, blood pressure, or activity levels have evolved. This helps the model spot trends that might indicate a developing risk for diabetes earlier than a single snapshot ever could.

THANK YOU