Building Machine Learning Models with Scikit-learn

Train and evaluate a classical ML model with scikit-learn: split data, fit a pipeline, and measure performance honestly.

What you will learn

Train/test splits and why leakage kills demos
Preprocessing with pipelines
Fit a model and evaluate with the right metric
Cross-validation and persist a model for later reuse

Prerequisites

Python 3.11+
Comfortable with pandas basics and NumPy arrays

Install

pip install scikit-learn pandas numpy joblib

User guide: scikit-learn user guide and Getting Started.

Why pipelines

Fit scalers and encoders only on training data, then apply the same transform to the test set. A Pipeline packages those steps so you cannot accidentally leak test information during preprocessing.

Pipeline sketch

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
import joblib

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
print("CV f1:", scores.mean(), scores.std())

alt = Pipeline([
    ("scale", StandardScaler()),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42)),
])
alt.fit(X_train, y_train)
joblib.dump(model, "breast_cancer_logreg.joblib")

Docs: Pipelines and composite estimators.

Choosing a metric

Balanced classification demo: accuracy can be OK
Rare positive class: prefer precision, recall, or F1
Ranking / probability quality: ROC AUC

Write the metric down before you tune hyperparameters.

Practice checklist

Train logistic regression and a random forest on the same split
Compare hold-out metrics with 5-fold CV on the training set only
Persist the winner with joblib and reload it in a second script
List three features that would cause leakage if included (IDs, future timestamps, target proxies)

Common pitfalls

Scaling the full dataset before the split
Tuning hyperparameters on the test set
Reporting training accuracy as if it were generalization

Sources and further reading

Who this guide is for

Python developers building a first supervised-learning workflow
Analysts moving from prediction experiments to evaluation
Teams reviewing whether a model is safe to pilot

Definition of done

A leakage-resistant scikit-learn Pipeline with a documented baseline, cross-validation, appropriate metrics, error analysis, reproducible artifact, and explicit non-goals.

Finishing the commands is not the same as finishing the guide. Keep the result only when another person can reproduce it, inspect the evidence, and understand the remaining risks.

Professional workflow

Phase 1 — Frame the decision

Goal: Connect prediction to a real action and cost.

Define target, prediction time, unit of observation, and intended user.
List false-positive and false-negative costs.
Choose a simple non-ML baseline.

Verification: The evaluation metric follows the decision cost rather than convenience.

Phase 2 — Build a leakage-safe experiment

Goal: Keep preprocessing inside the validation boundary.

Split data according to time, groups, or random sampling as the domain requires.
Put imputation, encoding, scaling, and model inside one Pipeline.
Tune only on training folds.

Verification: No feature uses information unavailable at the declared prediction time.

Phase 3 — Evaluate and hand off

Goal: Understand failure patterns before discussing deployment.

Compare baseline and candidate across multiple folds.
Inspect errors by meaningful cohorts.
Record data, code, parameters, and environment identity.

Verification: The model card states performance range, known weak cohorts, monitoring plan, and rejection criteria.

Review questions before you continue

What action follows the prediction?
Could any feature leak future or target information?
Does the split reflect real deployment?
Which cohort failure would make the model unacceptable?

If an answer is unknown, record it as an explicit follow-up instead of hiding the uncertainty behind a successful demo.

Troubleshooting decision guide

Validation is implausibly high: audit leakage, duplicates, split logic, and target proxies.
Scores vary widely: inspect sample size, stratification, groups, and unstable features.
Production features differ: package transformations with the model and validate schema.
A metric looks good but outcomes do not: revisit threshold and business cost.

Change one variable at a time, preserve the first useful error, and write down the command or condition that fixed it. That turns a private debugging session into team knowledge.

Production-readiness checklist

Document training-data provenance and consent constraints.
Version model, features, code, and environment.
Monitor input drift, output distribution, latency, and outcome quality.
Provide fallback and rollback behavior.
Require human and domain review for consequential decisions.

These checks are intentionally stricter than a beginner demo. Use them to decide whether to continue learning, run a controlled pilot, or request specialist review.

Your next 30 minutes

Write the non-ML baseline and decision cost.
Move all preprocessing into a Pipeline.
Add cohort-level error analysis.
Create a model card before any live pilot.

CTA: Pick the first unchecked step, complete it now, and save the evidence in your project README or decision log. Then open the official documentation linked below before adopting the result in production.