Week 1

Introduction to ML Strategy

• A good ML Strategy can save months of work

Orthogonalization

• Each control changes one thing and one thing only.

• E.g. In a car, the steering wheel only controls the wheel angles, the accelerator pedal controls speed, the brake pedal controls brake.

• We want orthogonalization in our controls while training a model too

  

  Image 1: Chain of assumptions and the respective knobs/controls we can tweak to improve each step of the chain

Setting up the goal

Single number evaluation metric

• A single real number evaluation metric is a good starting point for a ML project
• E.g. F1 Score (Harmonic mean of Precision and Recall) is a better metric than using two metrics Precision and Recall to evaluate models. F1 Score = $\frac{2}{1/P + 1/R}$. This allows the iterative process of a building a classification task faster
• Taking average of all metrics is another option to obtain a single evaluation metric

Optimizing and Satisficing metrics

• Optimizing metric: We want to be as good as possible on this metric
• Satisficing metric: We want to satisfy a threshold on this metric, but beyond that, we do not care much
• If we have N metrics, 1 should be the optimizing metric, the remaining N-1 metrics can be the satisficing metrics
• E.g. Accuracy (%) and Inference Time (ms) for a classification model

Train/Dev/Test distributions

• Dev and Test should always be from the same distribution. These sets should contain data that we expect to get in the future and consider important to do better on