XGBoost Classification of the Titanic Dataset – Adversarial Machine Learning

Import Modules

import pandas as pd
import seaborn as sns
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.preprocessing import OrdinalEncoder
from sklearn.preprocessing import StandardScaler

Load the Titanic Dataset

# Load the dataset from Seaborn datasets
df = sns.load_dataset('titanic')

# Display the first 10 rows
df.head(10)

	survived	pclass	sex	age	sibsp	parch	fare	embarked	class	who	adult_male	deck	embark_town	alive	alone
0	0	3	male	22.0	1	0	7.2500	S	Third	man	True	NaN	Southampton	no	False
1	1	1	female	38.0	1	0	71.2833	C	First	woman	False	C	Cherbourg	yes	False
2	1	3	female	26.0	0	0	7.9250	S	Third	woman	False	NaN	Southampton	yes	True
3	1	1	female	35.0	1	0	53.1000	S	First	woman	False	C	Southampton	yes	False
4	0	3	male	35.0	0	0	8.0500	S	Third	man	True	NaN	Southampton	no	True
5	0	3	male	NaN	0	0	8.4583	Q	Third	man	True	NaN	Queenstown	no	True
6	0	1	male	54.0	0	0	51.8625	S	First	man	True	E	Southampton	no	True
7	0	3	male	2.0	3	1	21.0750	S	Third	child	False	NaN	Southampton	no	False
8	1	3	female	27.0	0	2	11.1333	S	Third	woman	False	NaN	Southampton	yes	False
9	1	2	female	14.0	1	0	30.0708	C	Second	child	False	NaN	Cherbourg	yes	False

# Print data shape: 891 samples(passengers), 15 features
df.shape

(891, 15)

# Show the features and other basic information
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 15 columns):
 #   Column       Non-Null Count  Dtype   
---  ------       --------------  -----   
 0   survived     891 non-null    int64   
 1   pclass       891 non-null    int64   
 2   sex          891 non-null    object  
 3   age          714 non-null    float64 
 4   sibsp        891 non-null    int64   
 5   parch        891 non-null    int64   
 6   fare         891 non-null    float64 
 7   embarked     889 non-null    object  
 8   class        891 non-null    category
 9   who          891 non-null    object  
 10  adult_male   891 non-null    bool    
 11  deck         203 non-null    category
 12  embark_town  889 non-null    object  
 13  alive        891 non-null    object  
 14  alone        891 non-null    bool    
dtypes: bool(2), category(2), float64(2), int64(4), object(5)
memory usage: 80.7+ KB

Data Preparation and Preprocessing

# Remove columns that have identical information
df.drop(['alive'], axis=1, inplace=True) # Identical to 'survived'
df.drop(['class'], axis=1, inplace=True) # Identical to 'pclass'
df.drop(['embarked'], axis=1, inplace=True) # Identical to 'embark_town'

# Check if there are columns with missing values
df.isnull().sum()

	0
survived	0
pclass	0
sex	0
age	177
sibsp	0
parch	0
fare	0
who	0
adult_male	0
deck	688
embark_town	2
alone	0

dtype: int64

# Remove rows with missing information for 'age' and 'embark_town'
df.dropna(subset=['embark_town', 'age'], axis=0, inplace=True)

# Remove the column 'deck'
df.drop(['deck'], axis=1, inplace=True)

# Check again the information about the dataset
df.info()

<class 'pandas.core.frame.DataFrame'>
Index: 712 entries, 0 to 890
Data columns (total 11 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   survived     712 non-null    int64  
 1   pclass       712 non-null    int64  
 2   sex          712 non-null    object 
 3   age          712 non-null    float64
 4   sibsp        712 non-null    int64  
 5   parch        712 non-null    int64  
 6   fare         712 non-null    float64
 7   who          712 non-null    object 
 8   adult_male   712 non-null    bool   
 9   embark_town  712 non-null    object 
 10  alone        712 non-null    bool   
dtypes: bool(2), float64(2), int64(4), object(3)
memory usage: 57.0+ KB

# Convert categorical variables to numerical
# Select the categorical columns
categs_feats = df[['sex', 'who', 'adult_male', 'embark_town', 'alone']]

# Initialize the Ordinal Encoder
encoder = OrdinalEncoder()

# Fit the encoder and transform the values in categorical columns into numerical values
categs_encoded = encoder.fit_transform(categs_feats)

# Replace the values in the categorical columns in the Data Frame with numerical values
df[['sex', 'who', 'adult_male', 'embark_town', 'alone']] = pd.DataFrame(categs_encoded, columns=categs_feats.columns, index=categs_feats.index)

# Display again the first 10 rows
df.head(10)

	survived	pclass	sex	age	sibsp	parch	fare	who	adult_male	embark_town	alone
0	0	3	1.0	22.0	1	0	7.2500	1.0	1.0	2.0	0.0
1	1	1	0.0	38.0	1	0	71.2833	2.0	0.0	0.0	0.0
2	1	3	0.0	26.0	0	0	7.9250	2.0	0.0	2.0	1.0
3	1	1	0.0	35.0	1	0	53.1000	2.0	0.0	2.0	0.0
4	0	3	1.0	35.0	0	0	8.0500	1.0	1.0	2.0	1.0
6	0	1	1.0	54.0	0	0	51.8625	1.0	1.0	2.0	1.0
7	0	3	1.0	2.0	3	1	21.0750	0.0	0.0	2.0	0.0
8	1	3	0.0	27.0	0	2	11.1333	2.0	0.0	2.0	0.0
9	1	2	0.0	14.0	1	0	30.0708	0.0	0.0	0.0	0.0
10	1	3	0.0	4.0	1	1	16.7000	0.0	0.0	2.0	0.0

# Check the distribution of the target values
df['survived'].value_counts()
# 424 passengers didn't survive, 288 passengers survived

	count
survived
0	424
1	288

dtype: int64

Prepare Training and Testing Datasets

# Define features (X) and targets (y) variables
X = df.drop('survived', axis=1)
y = df['survived']

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=123, stratify=y)

# Print the shapes of data and label sets
print('Training data features', X_train.shape)
print('Training labels', y_train.shape)
print('Testing data features', X_test.shape)
print('Testing labels', y_test.shape)

Training data features (534, 10)
Training labels (534,)
Testing data features (178, 10)
Testing labels (178,)

# Apply StandardScaler to re-scale each column to have zero mean and unit standard deviation
scaler = StandardScaler()

scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

Train the XGBoost Model

# Create an instance of an XGBoost model
model = XGBClassifier(random_state=42)

# Fit the model to the training dataset
model.fit(X_train_scaled, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, device=None, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              feature_weights=None, gamma=None, grow_policy=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=None, max_bin=None, max_cat_threshold=None,
              max_cat_to_onehot=None, max_delta_step=None, max_depth=None,
              max_leaves=None, min_child_weight=None, missing=nan,
              monotone_constraints=None, multi_strategy=None, n_estimators=None,
              n_jobs=None, num_parallel_tree=None, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Evaluate the Model

# Predict on test dataset
model_predictions = model.predict(X_test_scaled)

# Calculate and print the accuracy
accuracy = accuracy_score(y_test, model_predictions)
print(f"Accuracy on test dataset: {accuracy*100:5.2f}%")

Accuracy on test dataset: 80.90%

# Print confusion matrix
print("Confusion Matrix:")
print(confusion_matrix(y_test, model_predictions))

Confusion Matrix:
[[90 16]
 [18 54]]

The confusion matrix shows: - True Negatives (top-left): Correctly predicted non-survival cases - False Positives (top-right): Non-survival cases incorrectly predicted as survival - False Negatives (bottom-left): Survival cases incorrectly predicted as non-survival - True Positives (bottom-right): Correctly predicted survival cases

# Print classification report
print("Classification report:")
print(classification_report(y_test, model_predictions))

Classification report:
              precision    recall  f1-score   support

           0       0.83      0.85      0.84       106
           1       0.77      0.75      0.76        72

    accuracy                           0.81       178
   macro avg       0.80      0.80      0.80       178
weighted avg       0.81      0.81      0.81       178

The report provides detailed metrics: - Precision: Of all predicted positives, how many were correct? - Recall: Of all actual positives, how many were detected? - F1-Score: Harmonic mean of precision and recall.