Loss Function

The loss function is the signal that tells a machine learning model how badly it is performing. During training, the model makes predictions; the loss function computes a scalar value measuring the discrepancy between predictions and ground truth labels. The optimizer (gradient descent) then adjusts model parameters to minimize this loss value. In this sense, the loss function is literally what the model is trying to optimize — it defines the learning objective.

Different tasks require different loss functions. For regression (predicting continuous values), Mean Squared Error (MSE) is standard: it penalizes large errors heavily, as the square amplifies discrepancies. Mean Absolute Error (MAE) is more robust to outliers. For binary classification, Binary Cross-Entropy measures how well the predicted probability matches the true label — rewarding confident correct predictions and heavily penalizing confident wrong ones. For multi-class classification, Categorical Cross-Entropy generalizes this. For object detection, specialized losses like Focal Loss address class imbalance by down-weighting easy examples.

Choosing the right loss function is a critical design decision. A loss that penalizes the wrong things will train a model that optimizes the wrong objective — even if the model achieves low loss, it may not perform well on the task that matters. For example, accuracy is a poor loss function for imbalanced datasets (a model that always predicts 'no fraud' achieves 99.9% accuracy on data where fraud is rare, but is useless). Cross-entropy and F1-score-based losses are better aligned with the true objective in such cases.