Machine Learning with Python

Supervised learning maps inputs X to a known target y. Classification predicts a discrete class (spam or ham, digit 0 through 9, iris species). Regression predicts a continuous value (house price, exam score, temperature). The training data carries the right answer for every example, the model learns the mapping, and at inference time it produces predictions on unseen X.

Unsupervised learning has no y. The model finds structure in X alone. Clustering groups similar examples (KMeans, DBSCAN, hierarchical). Dimensionality reduction projects high-dimensional X to a smaller space while preserving variance (PCA, t-SNE, UMAP). Anomaly detection flags points that look unlike the rest of the data (IsolationForest, OneClassSVM). The use case decides the family, not the algorithm.

Scikit-learn unifies both families behind one estimator interface. Every estimator implements `fit(X, y)` for supervised or `fit(X)` for unsupervised. Supervised estimators add `predict(X)` and `score(X, y)`. Clustering estimators add `labels_` after fitting. Transformers (like StandardScaler, OneHotEncoder) implement `fit` plus `transform`. This shared shape is what makes Pipeline work: you can swap a LogisticRegression for a RandomForest without changing any other code.

Reporting accuracy on the training set tells you nothing about generalization. A model with enough parameters memorizes the training data and scores 100% there while failing on new examples. The fix: split the labeled data into a training set (the model learns from this), a validation set (you compare candidate models and tune hyperparameters here), and a held-out test set (touched exactly once, at the end, to report the honest score).

The common ratio is 70 / 15 / 15 or 60 / 20 / 20. When data is scarce, k-fold cross-validation replaces the validation set: split the training data into k folds, train on k-1 folds, validate on the remaining fold, rotate through all k folds, average the scores. `cross_val_score(estimator, X, y, cv=5)` runs the whole loop in one call. Use 5-fold or 10-fold by default. Stratified k-fold preserves class proportions per fold and matters for imbalanced classification.

Random splits leak information when the data has temporal or grouped structure. For time series, the test set must come after the training set in time. `TimeSeriesSplit` produces forward-rolling folds. For grouped data (multiple rows per patient, per user, per document), use `GroupKFold` to keep all rows from one group on the same side of the split. Mixing rows from the same patient across train and test inflates the score and the resulting model fails on truly new patients.

Real datasets need preprocessing before they reach the estimator. Numeric columns get scaled. Categorical columns get one-hot encoded. Missing values get imputed. The naive approach runs each step manually on the full dataset and then splits into train and test. This leaks test-set statistics into the training pipeline because the scaler saw the test means.

Pipeline solves this by chaining transformers and a final estimator into one object. `Pipeline([("scaler", StandardScaler()), ("clf", LogisticRegression())])` exposes a single `fit` method that fits the scaler on training data only, transforms both train and test with those frozen statistics, then fits the classifier on the transformed training data. Cross-validation calls fit on each fold's training portion, so the imputer mean and scaler std come from inside the fold only. No leakage.

ColumnTransformer handles mixed-type tables. Pass a list of (name, transformer, columns) tuples, and each transformer applies only to the named columns. Numeric columns route to a StandardScaler. Categorical columns route to a OneHotEncoder. The output is one concatenated matrix that the downstream estimator consumes. Wrapping a ColumnTransformer plus an estimator in a Pipeline produces an end-to-end model with one saved artifact through `joblib.dump`.

Accuracy is the fraction of correct predictions. It misleads when one class dominates. On a fraud detection dataset where 1% of transactions are fraud, a model that predicts "not fraud" every time scores 99% accuracy and catches zero fraud. The right metric depends on what mistakes cost.

Precision answers "of the items I flagged as positive, how many were actually positive". Recall answers "of the actual positives, how many did I catch". The two trade off: pushing the decision threshold lower catches more positives (higher recall) at the cost of more false alarms (lower precision). The F1 score is the harmonic mean of precision and recall and balances both. ROC-AUC summarizes how well the model ranks positives above negatives across every threshold and ignores the threshold choice entirely, which is useful when downstream consumers pick their own cutoff.

For regression, the choice is between mean squared error (MSE, penalizes large errors quadratically), mean absolute error (MAE, treats every error linearly and resists outliers), and R squared (fraction of variance explained, dimensionless, useful for comparing models on the same target). For multi-class classification, pass `average="macro"` to compute the metric per class and average, or `average="weighted"` to weight by class size. Choose macro when each class matters equally, weighted when frequent classes matter more.

The iris dataset is the canonical first ML assignment. 150 flowers, 4 numeric features (sepal length, sepal width, petal length, petal width), 3 species labels. The task: predict species from features. The full pipeline runs in under 40 lines and demonstrates every step a CS229 or 10-601 lab brief checks.

The order: load data, split into train and test with stratification, fit two candidate models (LogisticRegression as a linear baseline, RandomForest as a non-linear comparison), compare via 5-fold cross-validation on the training set, pick the winner, evaluate on the test set exactly once, print a confusion matrix and per-class metrics. Set `random_state=42` everywhere a stochastic process appears so the grader reproduces your numbers.

The output is interpretable. The confusion matrix is a 3-by-3 grid where row i column j counts how many true-class-i examples were predicted as class j. The diagonal is correct predictions. Off-diagonal entries are mistakes, and the largest off-diagonal cell tells you which two classes the model confuses most. For iris that is usually versicolor with virginica, because their petal measurements overlap. `classification_report` prints precision, recall, F1 per class plus a weighted average.

LinearRegression fits the best straight-line relationship between X and y, minimizing squared error. It is fast, interpretable (every feature has one coefficient), and a strong baseline. The weakness: it overfits when features are correlated, and the coefficients explode when the design matrix is near-singular. Multi-collinearity inflates standard errors and produces unstable predictions.

Ridge regression adds an L2 penalty on the coefficients. The optimization minimizes `MSE + alpha * sum(coef ** 2)`, which shrinks coefficients toward zero and stabilizes them under collinearity. Larger alpha means stronger shrinkage. The hyperparameter is tuned with cross-validation, typically across a grid like `[0.01, 0.1, 1, 10, 100]`. Lasso (`Lasso`, L1 penalty) goes further and pushes some coefficients to exact zero, which doubles as feature selection.

RandomForestRegressor fits hundreds of decision trees on bootstrap samples of the data and averages their predictions. It captures non-linear relationships and interactions without manual feature engineering. The cost: longer training time, larger memory footprint, less interpretable than a single linear model. The 80/20 rule: try LinearRegression first, switch to Ridge if features are correlated, switch to RandomForest if the linear residuals show clear non-linear structure on a residual plot.