First, what kind of problem is it?
- Regression — predict a number (price, demand, time)
- Classification — predict a category (spam/not spam, churn/stay)
- Clustering — group similar items without labels (customer segments)
- Dimensionality reduction — compress many features into fewer, for visualisation or speed
Linear and logistic regression
Linear regression fits a straight-line relationship between features and a numeric target. Logistic regression adapts the idea to predict the probability of a class. Both are fast, easy to explain and excellent baselines — always try them first.
- Strengths: simple, interpretable coefficients
- Watch out for: non-linear relationships and outliers; scale features for regularised versions
Decision trees
A decision tree splits data with a series of yes/no questions (for example, “tenure < 6 months?”) until it reaches a prediction. Easy to visualise and explain to non-technical people.
- Strengths: interpretable, handles mixed data types
- Watch out for: overfitting — limit depth or use ensembles
Random forests and gradient boosting
Ensembles combine many trees. A random forest averages many trees trained on random samples, which reduces overfitting. Gradient boosting (XGBoost, LightGBM) builds trees one after another, each correcting the previous one's errors — often the top performer on tabular data.
- Strengths: high accuracy on structured data, feature importance
- Watch out for: less interpretable; boosting needs careful tuning
K-nearest neighbours (KNN)
KNN predicts using the most similar examples in the training data. Simple and intuitive, but slow on large datasets and sensitive to feature scaling.
Support vector machines (SVM)
SVMs find the boundary that separates classes with the widest margin, and kernels let them draw curved boundaries. Strong on small-to-medium datasets with many features; slower to train at large scale.
Naive Bayes
A probabilistic classifier that assumes features are independent. Very fast and surprisingly effective for text problems such as spam detection and sentiment analysis.
K-means clustering and PCA
K-means groups data into k clusters around centre points — useful for customer segmentation. PCA (principal component analysis) compresses many correlated features into a few components that keep most of the information, which helps visualisation and speeds up other models.
Train and compare models with scikit-learn
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
models = {
"logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"random_forest": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in models.items():
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
print(name, round(scores.mean(), 3))How to choose — a practical rule of thumb
- Start with a simple baseline (linear/logistic regression)
- For tabular data, try a random forest and gradient boosting next
- Need to explain every decision? Prefer linear models or shallow trees
- Text data? Try Naive Bayes or logistic regression on TF-IDF features, then modern embeddings
- Always compare on the same validation set with a metric that fits the problem