Let the machine find the rules

Churn prediction · private banking · synthetic data

1 · The data

fictitious clients with the same signals as the game. We know who left, so the machine can learn from the past.

Add bad data quality to the history the model learns from
Data quality problems in the learning data

Applied to the clients only. The test clients stay clean, so step 5 shows the true impact on the model.

clients
to learn from ( outcomes)
kept aside to test (, never seen)
of clients left

What the data cannot see: competitor offers, house purchases, family events… exactly like in real life.

2 · Your rules

Before the machine speaks: write down the rules the room used in the game. They are scored later, on the same unseen clients as the machine.

The room's rules

Signals in the data

No score yet: the room's rules are revealed next to the machine's at step 4.

3 · Train a decision tree

The decision tree looks for the question that best separates leavers from loyal clients, then repeats inside each group.

Decision tree. The same rules are listed as text in step 4.
Share of clients who left:lowhigh

Why a minimum group size? A rule that covers 2 clients is an anecdote, not a pattern. Anomaly detection (e.g. Isolation Forest) does the opposite: it keeps splitting until one client is isolated, and the fastest isolated are the most unusual.

How is the best level found?

Score = how well the model ranks leavers above loyal clients (50% = coin flip).

Clients it learned from Hidden clients
Score by number of questions, on learned clients and on hidden clients.

What it predicts for new clients

Three new clients the model has never seen (their outcome is revealed only in aggregate, at step 5):

4 · The rules the machine found

Each branch of the tree is a rule a business person can read. Now compare with the rules the room wrote at step 2.

The room's rules

    Same signals? Same thresholds? Edit the room's rules

    5 · How good is it on new clients?

    Tested on the clients of the model has never seen.

    Confusion matrix: what the model said × what really happened.

    Really left
    Really stayed
    Model says
    call
    Model says call, client really left:
    Caught
    leaver flagged in time
    Model says call, client really stayed:
    False alarm
    one call for nothing
    Model says
    fine
    Model says fine, client really left:
    Missed
    leaver not flagged
    Model says fine, client really stayed:
    Left alone
    loyal, not bothered
    Sensitivity
    of leavers are caught
    Specificity
    of loyal clients are left alone

    How is the threshold chosen?

    This ROC curve shows each possible threshold as a dot. Top-left would be perfect; the diagonal is a coin flip.

    ROC curve: share of leavers caught against false-alarm rate, one point per threshold.

    Calls to make
    Accuracy
    Business view (adjust live)
    Net value of the campaign:

    6 · Back to our 8 clients

    What does the model say about Marc, Sophie, Jean…? Same threshold as step 5 ().

    7 · Monitor new clients

    The model is live (inference). new clients arrive in , and whether they leave will only be known months later. Until then, we watch the data going in.

    Break the incoming data

    Data quality problems in production

    The model does not change. Only the data it receives does.

    Data drift monitor

    What the model does

    Known today

    Known only months later, when outcomes arrive