Gabriel Rocha
Projects

Customer Churn Analysis

A study of bank customer churn, from exploratory analysis to modeling and an interactive churn predictor.

2024Repository

A study of bank customer churn, done as a pair with my friend Juan Braz on Kaggle’s public “Bank Customer Churn” dataset (10,000 customers and 18 columns). The work became a series of three articles on Medium, in English: I took the exploratory analysis (Part 1, on my profile skicklel) and Juan took the modeling and validation (Part 2) and the interactive predictor (Part 3). The goal was to understand which characteristics separate those who leave from those who stay and, afterwards, to predict churn.

What was investigated

In the exploratory part I compared each variable against the target Exited (whether the customer left or stayed). The dataset had no null or duplicate values, so cleaning meant checking types and moving on to the visualizations.

The clearest patterns:

In the correlation matrix, the variables with the strongest correlation with Exited were Age, Balance and IsActiveMember. That matrix used only numeric columns, which is why Complain, the dataset’s big leak, did not show up there.

To read each category, I plotted the count split by who left and who stayed and annotated the percentage of each group above the bar:

for coluna in ['Card Type', 'HasCrCard']:
    plt.figure(figsize=(10, 6))
    ax = sns.countplot(x=df_bank[coluna], hue=df_bank['Exited'],
                       palette="viridis", dodge=True)
    for container in ax.containers:
        total = sum(p.get_height() for p in container)
        for p in container:
            height = p.get_height()
            percentage = f'{100 * height / total:.1f}%' if total > 0 else '0.0%'
            ax.annotate(percentage, (p.get_x() + p.get_width() / 2., height),
                        ha='center', va='bottom', fontsize=10, color='black')
    plt.title(f'Distribuição de Churn por {coluna}')

Modeling and validation

In Part 2 Juan went with the classic pipeline: one-hot encoding on the categoricals, StandardScaler on the numerics, an 80/20 stratified split and three models (Logistic Regression, Random Forest and XGBoost), compared by accuracy, AUC, precision, recall and F1.

The split and the encoding from the notebook:

X = df_bank.drop('Exited', axis=1)   # features
y = df_bank['Exited']                # alvo

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

X_train_encoded = pd.get_dummies(X_train)
X_test_encoded = pd.get_dummies(X_test)
X_train_encoded, X_test_encoded = X_train_encoded.align(
    X_test_encoded, join='left', axis=1, fill_value=0)

All three models came out with accuracy above 99.7% on the test set, AUC between 0.997 and 0.9996 and F1 above 0.99, and the 5-fold cross-validation repeated the pattern (always near 0.998). On a churn dataset with only ~20% churn, such a perfect result should have been the first sign that something was wrong with the data, but at the time we treated it as “well prepared” data and even concluded there was no leakage. Part 3 closes with a script that collects a customer’s data via input() and returns each model’s churn probability.

Particularities and points I would revisit

  1. Information leakage built into the dataset. The Complain and Satisfaction Score columns are recorded after the complaint and the service interaction, so they do not exist before the customer decides to leave, and the first is practically the answer key for the target: Complain has a mean of 0.2044 while Exited has 0.2038, nearly the same count. With that, the models “learn” to answer from the leaked column and hit ~99.8%. Today I would revisit the articles’ conclusion: it was the Kaggle data itself leaking, not merit of the pipeline.

  2. The interactive predictor inherits the leakage. The Part 3 tool asks for Complain, Satisfaction Score and Points Earned as input. In a real scenario that information only exists after the event one wants to predict, so the “real-time predictor” replicates the same problem as the dataset.

  3. Two preprocessing pipelines in the notebook. One cell builds a df_encoded (one-hot with drop_first=True + StandardScaler) and announces that the data is ready for modeling, but the modeling right after ignores that dataframe, redoing the encoding on X_train/X_test with align. It left dead code behind and two different ways of preparing the same data.

  4. Identifier columns enter as features. X is built with only df_bank.drop('Exited'), so RowNumber, CustomerId and Surname remain in the model. Worse: Surname is object, and pd.get_dummies(X_train) without filtering turns that into thousands of useless dummy columns.

  5. “External validation” on random synthetic data. The external dataset was generated with np.random.choice, drawing Exited and Complain independently; in a dataset like that the features have no relation to the target, so reporting an AUC near 1 on it is implausible. I would revalidate that step, because the result does not match what a purely random dataset should produce.

  6. Heavy repository with no .gitignore. The 837 KB CSV and the main.ipynb with all outputs and base64 images (about 1.4 MB) were committed; a clean notebook would be much smaller and easier to review.

Points for improvement