Classification Metrics

#1: Accuracy

📖 Definition

Accuracy measures the proportion of correctly predicted instances (both positive and negative) among the total number of instances evaluated. It’s the simplest and most intuitive metric for classification.

🧮 Mathematical Equation & Components

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

• TP = True Positives
• TN = True Negatives
• FP = False Positives
• FN = False Negatives

🧰 Typical Use Cases

• Balanced binary or multi-class classification problems
• Quick benchmarking for initial model comparison
• Tasks where false positives and false negatives are equally important

⭐ Strength Points

• Easy to compute and interpret
• Works well when classes are balanced
• Suitable for high-level overviews or dashboards

⚠️ Weakness Points

• Misleading in imbalanced datasets. For example, 95% accuracy may mean the model always predicts the majority class
• Doesn’t distinguish between types of errors (FP vs FN)
• Can mask poor model performance on minority classes

🎯 Best Practice Recommendation

• Use Accuracy only when your dataset is balanced
• Always pair with other metrics like Precision, Recall, and F1 Score—especially in sensitive domains like healthcare or fraud detection
• For multi-class settings, consider macro-averaged or weighted accuracy variants

#2: Precision

📖 Definition

Precision (also called Positive Predictive Value) measures the proportion of correctly predicted positive instances out of all instances that were predicted as positive. It reflects how precise the model is when it claims something is positive.

🧮 Mathematical Equation & Components

$$\text{Precision} = \frac{TP}{TP + FP}$$

• TP = True Positives
• FP = False Positives

🧰 Typical Use Cases

• Spam detection – better to only flag actual spam (high precision), even if you miss a few
• Information retrieval – e.g., search engines prioritizing relevance
• Medical diagnosis – useful when minimizing false alarms (false positives) is key

⭐ Strength Points

• Effective when the cost of false positives is high
• Ideal when accuracy on the positive class is critical
• Great for retrieval tasks and classification under class imbalance

⚠️ Weakness Points

• Ignores false negatives – misleading if many positives are missed
• Alone, doesn’t offer a full picture of model performance
• High precision may reduce recall – fewer true positives detected to avoid false positives

🎯 Best Practice Recommendation

• Use Precision alongside Recall to balance insights
• In high-stakes domains (like fraud detection or cancer screening), evaluate trade-offs using F1 Score or PR Curves
• For multi-class, analyze class-specific precision for granular evaluation

#3: Recall

(Also known as Sensitivity or True Positive Rate)

📖 Definition

Recall measures the proportion of actual positive cases that were correctly identified by the model. It answers the question: "Out of all actual positives, how many did the model catch?"

🧮 Mathematical Equation & Components

$$\text{Recall} = \frac{TP}{TP + FN}$$

• TP = True Positives
• FN = False Negatives

🧰 Typical Use Cases

• Medical diagnostics – catching all possible disease cases is crucial
• Search engines and recommendation systems – showing all relevant items
• Intrusion detection or fraud detection – better to catch all real threats, even with some false alarms

⭐ Strength Points

• Critical when missing positives is costly
• Ensures safety and comprehensiveness (e.g., catching all cancers or intrusions)
• Can be tuned with recall-oriented thresholds for sensitivity

⚠️ Weakness Points

• Ignores false positives – may lead to many incorrect positive predictions
• High recall may reduce precision
• Alone, can be misleading in imbalanced datasets

🎯 Best Practice Recommendation

• Always evaluate Recall alongside Precision for balance
• For recall-critical domains, tune thresholds or use cost-sensitive learning
• Pair with Precision-Recall Curves or F1 Score for robust evaluation

#4: F1 Score

📖 Definition

F1 Score is the harmonic mean of Precision and Recall. It combines both into a single number that balances the trade-off between false positives and false negatives, especially useful when you need both precision and recall to be reasonably high.

🧮 Mathematical Equation & Components

$$\text{F1 Score} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

Substituting from earlier:

$$= \frac{2TP}{2TP + FP + FN}$$

• TP = True Positives
• FP = False Positives
• FN = False Negatives
• Precision = \( \frac{TP}{TP + FP} \)
• Recall = \( \frac{TP}{TP + FN} \)

🧰 Typical Use Cases

• Imbalanced classification problems – such as fraud detection or rare disease diagnosis
• NLP tasks – like Named Entity Recognition or text classification
• Any scenario requiring a balance between missing positives and making false alarms

⭐ Strength Points

• Balances the trade-off between Precision and Recall
• Especially effective in imbalanced datasets
• Can be macro/micro/weighted averaged in multi-class settings

⚠️ Weakness Points

• Doesn’t distinguish the importance between Precision and Recall (assumes equal weight)
• Can obscure detail when one metric is significantly higher than the other
• Not intuitive to interpret directly in isolation

🎯 Best Practice Recommendation

• Use F1 Score when both Precision and Recall matter, especially in imbalanced datasets
• For custom importance, use the Fβ Score to weigh Recall (β > 1) or Precision (β < 1) differently
• In multi-class problems, select from macro-F1 (unweighted mean), micro-F1 (global), or weighted-F1 (support-aware)

#5: Specificity

(Also known as True Negative Rate)

📖 Definition

Specificity measures the proportion of actual negative cases that were correctly identified as negative by the model. It answers: "Out of all actual negatives, how many did we correctly mark as negative?"

🧮 Mathematical Equation & Components

$$\text{Specificity} = \frac{TN}{TN + FP}$$

• TN = True Negatives
• FP = False Positives

🧰 Typical Use Cases

• Medical testing – ensuring healthy people are not wrongly diagnosed
• Binary classifiers where false positives must be minimized (e.g., spam filters avoiding valid emails)
• Anomaly detection – avoiding false alarms on normal behavior

⭐ Strength Points

• Complements Recall by evaluating performance on the negative class
• Valuable when false positives have serious consequences
• Can be combined with Recall to compute Balanced Accuracy

⚠️ Weakness Points

• Doesn’t consider false negatives – can lead to overconfidence in results
• May appear high in imbalanced datasets due to the prevalence of negative cases
• Requires pairing with Recall or Balanced Accuracy for a full view

🎯 Best Practice Recommendation

• Use together with Recall in binary classification tasks that require careful risk trade-offs
• Consider the trio: Specificity, Sensitivity (Recall), and Balanced Accuracy for fairness
• Highly relevant in epidemiology, diagnostics, cybersecurity, and quality control systems

#6: ROC Curve (Receiver Operating Characteristic)

📖 Definition

The ROC Curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. It plots the True Positive Rate (Recall) against the False Positive Rate across various threshold levels.

🧮 Mathematical Concepts & Components

• True Positive Rate (TPR) = \( \frac{TP}{TP + FN} \) → Same as Recall
• False Positive Rate (FPR) = \( \frac{FP}{FP + TN} \)

The ROC Curve is constructed by plotting TPR vs. FPR as the classification threshold changes.

🧰 Typical Use Cases

• Binary classification problems with an emphasis on threshold-independent performance
• Medical diagnostics – visualizing sensitivity vs. false alarm trade-offs
• Model comparison – selecting classifiers based on performance across all thresholds

⭐ Strength Points

• Threshold-independent: summarizes model performance across all decision boundaries
• Effective with imbalanced classes
• Visualizes the trade-off between sensitivity and specificity

⚠️ Weakness Points

• Less informative when the positive class is rare
• Can be misleading if the costs of FP and FN are asymmetric
• In highly imbalanced data, Precision-Recall Curves may be more insightful

🎯 Best Practice Recommendation

• Use ROC for initial evaluation when classes are relatively balanced
• Include AUC (Area Under Curve) for a scalar performance summary
• Complement with PR Curves in imbalanced settings for clearer insight

#7: AUC (Area Under Curve)

📖 Definition

AUC stands for Area Under the ROC Curve. It represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance by the classifier. It provides a scalar summary of ROC Curve performance.

🧮 Mathematical Concept & Components

There is no simple closed-form expression. AUC is typically calculated using numerical integration (e.g., the trapezoidal rule) over the ROC curve:

$$\text{AUC} = \int_{0}^{1} TPR(FPR) \, dFPR$$

• AUC ranges from 0 to 1
• AUC = 0.5 → Random classifier
• AUC = 1.0 → Perfect classifier
• Higher AUC implies better model at ranking positives above negatives

🧰 Typical Use Cases

• Binary classification with probabilistic model outputs
• Medical diagnostics – balancing sensitivity and specificity in risk scoring
• Model benchmarking – comparing classifiers without fixing a threshold

⭐ Strength Points

• Threshold-independent assessment
• Robust to class imbalance versus accuracy
• Acts as a ranking metric – intuitive interpretation for decision prioritization

⚠️ Weakness Points

• Doesn’t indicate how well-calibrated predictions are
• Can be overly optimistic in highly imbalanced datasets
• May misalign with real-world objectives in cost-sensitive problems

🎯 Best Practice Recommendation

• Use AUC when evaluating or comparing classifiers across thresholds
• In imbalanced datasets, pair AUC with Precision-Recall AUC or F1 Score
• Avoid relying on AUC alone—especially when decision thresholds affect business risk

#8: Confusion Matrix

📖 Definition

A Confusion Matrix is a summary table used to evaluate the performance of a classification model. It displays the number of correct and incorrect predictions, broken down by each actual class vs. predicted class.

🧮 Structure & Components

For binary classification, the confusion matrix is:

From this, you can derive:

• Accuracy = \( \frac{TP + TN}{TP + TN + FP + FN} \)
• Precision = \( \frac{TP}{TP + FP} \)
• Recall = \( \frac{TP}{TP + FN} \)
• Specificity = \( \frac{TN}{TN + FP} \)

It naturally extends to multi-class classification with rows and columns for each class label.

🧰 Typical Use Cases

• Applicable to all classification tasks — binary and multi-class
• Debugging model predictions by exposing common misclassifications
• Understanding per-class performance and class confusion patterns

⭐ Strength Points

• Highly interpretable: visually intuitive for most audiences
• Serves as a foundation for all other classification metrics
• Reveals confused classes in multi-class setups

⚠️ Weakness Points

• Not a standalone metric: requires further interpretation or summarization
• Can become large and complex for high-class-count problems
• Raw counts may be misleading without normalization (e.g., support-based)

🎯 Best Practice Recommendation

• Start with the confusion matrix in every classification evaluation pipeline
• Use normalized or percentage values for interpretability
• Visualize with heatmaps to spot systematic errors and misclassifications

#9: Logarithmic Loss (Log Loss)

(Also known as Logistic Loss or Cross-Entropy Loss in deep learning)

📖 Definition

Log Loss evaluates the uncertainty of predictions by measuring how far predicted probabilities deviate from actual class labels. It heavily penalizes confident but incorrect predictions and rewards models that are well-calibrated in their probability estimates.

🧮 Mathematical Equation & Components

For binary classification:

$$\text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left[y_i \log(p_i) + (1 - y_i) \log(1 - p_i)\right]$$

• N = total number of samples
• y_i = actual label (0 or 1)
• p_i = predicted probability for class 1

For multi-class classification:

$$\text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{M} y_{ij} \log(p_{ij})$$

• M = number of classes
• y_ij = indicator (1 if class j is correct for sample i)
• p_ij = predicted probability for class j on sample i

🧰 Typical Use Cases

• Probabilistic classification where prediction confidence matters
• Model evaluation in competitions like Kaggle
• Assessing calibrated classifiers such as logistic regression or softmax-based neural networks

⭐ Strength Points

• Confidence-sensitive: rewards accurate probability estimates
• Applicable to binary and multi-class problems
• Threshold-independent: no fixed decision boundary needed

⚠️ Weakness Points

• Heavily penalizes confident but incorrect predictions
• Not intuitive without contextual benchmarks
• Sensitive to outliers and mislabeled data

🎯 Best Practice Recommendation

• Use when model confidence calibration is important
• Combine with Accuracy, AUC, or F1 Score for full evaluation
• Ideal for model selection and hyperparameter tuning in probabilistic learning

#10: Matthews Correlation Coefficient (MCC)

📖 Definition

MCC measures the quality of binary classifications by considering all four categories of the confusion matrix: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). The result ranges from –1 to +1, where:

• +1 = perfect prediction
• 0 = random prediction
• –1 = total disagreement between predicted and actual labels

It is often described as the correlation coefficient between observed and predicted classifications.

🧮 Mathematical Equation & Components

$$\text{MCC} = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$

• TP = True Positives
• TN = True Negatives
• FP = False Positives
• FN = False Negatives

🧰 Typical Use Cases

• Binary classification under imbalanced class distributions
• Bioinformatics, medical diagnostics, and risk scoring domains
• Fair model comparison when class prevalence differs significantly

⭐ Strength Points

• Considers all confusion matrix values for holistic evaluation
• Resilient to class imbalance
• Balanced performance metric even with skewed class distributions

⚠️ Weakness Points

• Not intuitive for business or non-technical audiences
• May be undefined if any denominator term equals zero
• Less commonly supported in some ML tooling/libraries

🎯 Best Practice Recommendation

• Use MCC when facing imbalanced binary classification problems
• Ideal when both false positives and false negatives matter equally
• Consider MCC as a scientific-grade alternative to the F1 Score

#11: Cohen’s Kappa

📖 Definition

Cohen’s Kappa measures the agreement between two raters or classifiers, adjusting for agreement that could occur by chance. It is widely used in tasks involving inter-annotator reliability, such as human labeling of text, images, or audio.

🧮 Mathematical Equation & Components

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

• p_o = Observed agreement = \( \frac{TP + TN}{N} \)
• p_e = Expected agreement by chance = \( (P_1 \cdot P_2) + (N_1 \cdot N_2) \), where:
  • P₁, P₂ = probability both raters choose positive
  • N₁, N₂ = probability both raters choose negative

Value Range:

• +1: Perfect agreement
• 0: Agreement expected by chance
• –1: Complete disagreement

🧰 Typical Use Cases

• Evaluating human label consistency — e.g., annotators tagging sentiment or entities
• Classifier vs. classifier agreement for model comparison
• Medical diagnostics — comparing model vs. expert assessment

⭐ Strength Points

• Corrects for chance agreement, unlike raw accuracy
• Supports binary and multi-class classification tasks
• Well-suited for qualitative tasks with subjective labeling

⚠️ Weakness Points

• Assumes independent raters, which may not hold in collaborative environments
• Can be misleading with imbalanced datasets
• Hard to interpret for non-technical audiences

🎯 Best Practice Recommendation

• Use when measuring agreement beyond chance, especially with human-annotated data
• Combine with accuracy, precision, recall, or F1 for broader evaluation
• For >2 raters, explore extensions like Fleiss’ Kappa

#12: Balanced Accuracy

📖 Definition

Balanced Accuracy is the average of sensitivity (recall for positives) and specificity (recall for negatives). It is designed to handle imbalanced class distributions, where standard accuracy can give misleadingly high results by favoring the majority class.

🧮 Mathematical Equation & Components

$$\text{Balanced Accuracy} = \frac{1}{2} \left( \frac{TP}{TP + FN} + \frac{TN}{TN + FP} \right)$$

Which simplifies to:

$$\text{Balanced Accuracy} = \frac{\text{Sensitivity} + \text{Specificity}}{2}$$

• TP = True Positives
• TN = True Negatives
• FP = False Positives
• FN = False Negatives

🧰 Typical Use Cases

• Binary classification with class imbalance
• Rare event detection — e.g., fraud, disease prediction, equipment failure
• Regulatory domains where performance on both classes is crucial

⭐ Strength Points

• Equally fair to both classes, even in imbalanced scenarios
• Improves on raw accuracy by correcting class imbalance bias
• Simple to interpret and implement

⚠️ Weakness Points

• Can be misleading in extremely skewed data without supporting metrics
• Less informative in multi-class classification unless macro-averaged
• Doesn’t reflect error cost or contextual risk

🎯 Best Practice Recommendation

• Use as a baseline performance metric in imbalanced classification problems
• Pair with F1 Score, AUC, or Precision/Recall for a balanced view
• Helpful in benchmarking across varied datasets

#13: Top-K Accuracy

📖 Definition

Top-K Accuracy measures whether the true label appears among the top K predicted probabilities. It’s especially useful in multi-class classification problems where being approximately right is still valuable—such as ranking, retrieval, or large output spaces.

🧮 Mathematical Concept & Components

$$\text{Top-K Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}(y_i \in \text{Top-K}(\hat{y}_i))$$

• N = total number of samples
• y_i = true label for the i^th sample
• 𝑇𝑜𝑝−𝐾(ŷ_i) = set of top K predicted classes for that sample
• 𝟙() = indicator function: returns 1 if the condition is true, 0 otherwise

🧰 Typical Use Cases

• Image classification – such as ImageNet where Top-5 is a standard benchmark
• Recommendation systems – ensuring correct items appear in the suggested list
• NLP tasks – next-word prediction, response generation, intent detection

⭐ Strength Points

• Realistic in user-facing applications where multiple options are shown
• Less rigid than Top-1 Accuracy – gives credit for near-miss predictions
• Useful in fine-grained classification problems with many similar categories

⚠️ Weakness Points

• May overestimate model usefulness – correct answer in Top-K isn’t always actionable
• No insight into rank position within the Top-K (top-2 is treated same as top-K)
• Not relevant for binary classification or when only one option is allowed

🎯 Best Practice Recommendation

• Use for multi-class settings with high cardinality and fuzzy category boundaries
• Report both Top-1 and Top-5 Accuracy to give a full picture
• Complement with confusion matrix or Precision@K for deeper diagnostics

#14: Brier Score

📖 Definition

The Brier Score measures the mean squared difference between predicted probabilities and actual binary outcomes. It evaluates both accuracy and calibration of predictions—where lower values indicate better performance.

🧮 Mathematical Equation & Components

$$\text{Brier Score} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2$$

• N = number of instances
• p_i = predicted probability of the positive class for instance i
• y_i = actual class label (0 or 1)

For multi-class classification:

$$\text{Brier Score}_{\text{multi}} = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{M} (p_{ij} - y_{ij})^2$$

🧰 Typical Use Cases

• Binary classification with probabilistic outputs (e.g., logistic regression)
• Evaluating model calibration (e.g., forecast confidence)
• Applied in weather forecasting, medical risk scoring, credit prediction

⭐ Strength Points

• Measures both calibration and accuracy of predicted probabilities
• Works with probabilistic models across domains
• Differentiable – usable as a training objective in some models

⚠️ Weakness Points

• Harsh on overconfident mistakes (high confidence, wrong prediction)
• Less intuitive than threshold-based metrics like accuracy or F1
• Can be skewed by class imbalance unless adjusted

🎯 Best Practice Recommendation

• Use when probability quality and calibration are more important than classification
• Combine with calibration plots or reliability diagrams for interpretability
• For skewed class distributions, consider class-weighted Brier or alternative scoring rules

🔁 Key Trade-offs Between Classification Metrics

📊 Comparative Tables

🎯 Table 1: Recommended Usage by Scenario

⚖️ Table 2: Sensitivity to Class Imbalance

💬 Table 3: Interpretability and Practical Use

🧠 1. Threshold Optimization Techniques

🎯 Why Threshold Optimization?

Most classifiers output probabilities. To convert them into final class predictions, a decision threshold is applied—typically 0.5. But 0.5 isn’t always optimal, especially in:

• Imbalanced datasets
• Risk-sensitive domains
• Multi-objective classification (e.g., maximizing both precision and recall)

📈 Tools for Visualizing Threshold Effects

• ROC Curve – trade-off between True Positive Rate (Recall) and False Positive Rate
• Precision-Recall Curve – more informative under class imbalance
• Threshold-Performance Curve – plot F1, precision, or recall directly vs. threshold

🧮 Key Threshold Optimization Methods

✅ 1. Youden’s J Statistic

$$J = \text{Sensitivity} + \text{Specificity} - 1$$

• Maximize J to find optimal threshold
• Best when FP and FN are equally costly
• Common in biometrics and medical diagnostics

✅ 2. F1 Score Maximization

• Compute F1 score across many thresholds
• Select threshold that maximizes F1
• Excellent for imbalanced classification
• Supported by precision_recall_curve in scikit-learn

✅ 3. Cost-sensitive Thresholding

• Define a custom cost matrix (e.g., FN is 5× more costly than FP)
• Pick threshold that minimizes expected cost
• Used in finance, fraud detection, and medicine

🧪 How to Apply in Code (Scikit-learn Example)

from sklearn.metrics import f1_score
import numpy as np

thresholds = np.arange(0.0, 1.0, 0.01)
f1_scores = [f1_score(y_true, y_prob >= t) for t in thresholds]
best_thresh = thresholds[np.argmax(f1_scores)]
  

🧠 Pro Tips

• Always tune thresholds on a validation set
• Re-tune thresholds after retraining or deploying to a new domain
• Calibrate probabilities (e.g., Platt scaling, isotonic regression) before threshold tuning

🧪 2. Cross-Validation with Metrics

📖 Why Use Cross-Validation?

• Provides unbiased estimates of model performance
• Prevents overfitting to a single train/test split
• Reveals performance variance across different data splits

🔁 How It Works

Cross-validation divides your dataset into K equal-sized folds. The model is trained and evaluated K times, each time holding out one fold as the validation set while training on the remaining K–1.

🔧 Common Techniques

🧮 Scikit-learn Example – Basic Accuracy CV

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

scores = cross_val_score(RandomForestClassifier(), X, y, cv=5, scoring='accuracy')
print("Accuracy (CV):", scores.mean())
  

🧪 Evaluating Multiple Metrics Together

from sklearn.model_selection import cross_validate
from sklearn.metrics import make_scorer, f1_score, recall_score

scoring = {
    'accuracy': 'accuracy',
    'f1': make_scorer(f1_score, average='macro'),
    'recall': make_scorer(recall_score, average='macro')
}

results = cross_validate(RandomForestClassifier(), X, y, cv=5, scoring=scoring)
print("Average F1:", results['test_f1'].mean())
  

📌 Best Practices

• Use StratifiedKFold for classification to maintain class proportions
• Choose metrics aligned with your domain objectives
• For temporal or sequential data, use TimeSeriesSplit instead of regular K-Fold

🎨 3. Visual Tools for Classification Metrics

Visualizations make classification performance intuitive, actionable, and presentable. Here are essential plots every ML practitioner should know:

🔲 1. Confusion Matrix Heatmaps

What It Shows: Frequency of actual vs. predicted class labels as a matrix with heatmap intensity.

Use Case: Understand where your model makes mistakes, especially in multi-class classification.

from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

cm = confusion_matrix(y_true, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel("Predicted"); plt.ylabel("Actual")
plt.title("Confusion Matrix")
plt.show()
  

📈 2. ROC and Precision-Recall Curves

What They Show:

• ROC Curve: True Positive Rate (Recall) vs. False Positive Rate
• Precision-Recall Curve: Precision vs. Recall

Use Case: Evaluate threshold behavior and model comparison—especially under class imbalance (PR).

from sklearn.metrics import roc_curve, precision_recall_curve, RocCurveDisplay, PrecisionRecallDisplay

fpr, tpr, _ = roc_curve(y_true, y_scores)
RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

precision, recall, _ = precision_recall_curve(y_true, y_scores)
PrecisionRecallDisplay(precision=precision, recall=recall).plot()
plt.show()
  

📉 3. Calibration Plots (Reliability Diagrams)

What It Shows: Whether predicted probabilities match actual event frequencies—used to check calibration.

Use Case: Use when evaluating Log Loss or Brier Score, especially in probabilistic models.

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=10)
plt.plot(prob_pred, prob_true, marker='o', label='Model')
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfectly Calibrated')
plt.xlabel("Mean Predicted Probability")
plt.ylabel("Fraction of Positives")
plt.title("Calibration Curve")
plt.legend()
plt.show()
  

🎯 Best Practice Recommendation

• Include visual tools during both development and presentation
• Great for communicating insights with non-technical stakeholders
• Use alongside metrics for a holistic evaluation

🎯 4. Metric Selection Checklist

Use this structured checklist to identify the most appropriate classification evaluation metrics based on your dataset, goals, and application context.

This checklist helps you align metric choices with your task's structure, constraints, and stakeholder needs.