ML Algorithms: Linear, Logistic, Decision Tree, KNN

Linear regression predicts a number from a straight-line relationship. Logistic regression predicts a probability and a class (yes or no) from the same kind of weighted sum. A decision tree predicts by asking a series of if-then questions about the features. K-nearest neighbors (KNN) predicts by looking at the most similar examples it has seen. Use linear and logistic regression as fast, explainable baselines, trees for rules and mixed data, and KNN for small datasets where similarity is meaningful.

This guide explains how each one works, shows the key formula, compares them side by side, lists the common mistakes, and ends with a short code example. For the bigger picture, start with machine learning explained.

#Key takeaways

• Linear regression predicts a continuous number (price, demand).
• Logistic regression is a classifier despite the name. It outputs a probability between 0 and 1.
• Decision trees are easy to read, handle mixed data, and overfit if left unchecked.
• KNN has no training step but needs scaled features and gets slow on large data.
• Always start with a simple baseline and measure before using something fancier.

#Side-by-side comparison

#Linear regression

Linear regression models the target as a weighted sum of the features plus an intercept:

ŷ = b0 + b1·x1 + b2·x2 + … + bn·xn

It chooses the weights that minimise the mean squared error between predictions and actual values (ordinary least squares). Each weight says how much the prediction changes when that feature goes up by one unit, holding the others fixed, which makes it easy to explain.

Use it for: predicting prices, demand, time or any continuous value where the relationship is roughly linear.

Watch out for: outliers that drag the line, highly correlated features that make weights unstable, and non-linear patterns that a straight line cannot capture.

#Logistic regression

Logistic regression takes the same weighted sum, z = b0 + b1·x1 + …, and squeezes it through the sigmoid function:

p = 1 / (1 + e^(−z))

The result p is a probability between 0 and 1. A common rule is to predict class 1 when p is at least 0.5, but you can move that threshold to trade precision against recall. The weights are fitted by minimising log loss.

Use it for: spam or not, churn or not, fraud or not, and any yes-or-no question where you want a probability and an explanation.

Watch out for: treating it as a regression model (it predicts classes), class imbalance, and non-linear boundaries it cannot draw without extra features.

#Decision trees

A decision tree splits the data again and again with a yes-or-no question about one feature, such as "age under 30?", until the groups are mostly one class. At each step it picks the split that makes the resulting groups purest. A common purity measure for classification is Gini impurity:

Gini = 1 − Σ pᵢ²

where pᵢ is the share of class i in the group. A group that is all one class scores 0.

Use it for: problems where you need readable rules, data with a mix of numeric and categorical features, and as the building block of random forests and gradient boosting.

Watch out for: overfitting. A deep tree can memorise the training set. Limit max_depth, require a minimum samples per leaf, or prune. Small changes in the data can also produce a very different tree. See overfitting and bias-variance.

#K-nearest neighbors (KNN)

KNN stores the training data. To predict for a new point, it finds the k closest training points, usually by Euclidean distance, then takes a majority vote (classification) or an average (regression).

Use it for: small to medium datasets where similar inputs really do have similar outcomes, and as a quick sanity-check baseline.

Watch out for:

• Scale your features. A feature measured in thousands will swamp one measured in single digits.
• Choose k carefully. A tiny k is noisy and a huge k blurs everything. Pick it with cross-validation.
• Many features hurt. In high dimensions distances become less meaningful.
• Predictions are slow on big datasets because each one compares against stored points.

#Which one should you use?

A good habit: fit logistic or linear regression first, then see whether a tree or KNN beats it on held-out data. If a complex model barely improves on the baseline, keep the baseline.

#A short scikit-learn example

pythonfrom sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

models = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "tree": DecisionTreeClassifier(max_depth=3, random_state=42),
    "knn": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    print(name, round(model.score(X_test, y_test), 3))

Notes: the scaler is inside the pipeline so it is fitted on training data only, which avoids leaking test data. The tree does not need scaling. For linear regression use LinearRegression on a numeric target in the same way.

#Common mistakes

1. Evaluating on the training data. Always hold out test data or use cross-validation. See model evaluation metrics.
2. Using accuracy on imbalanced classes. Check precision, recall and F1 as well.
3. Forgetting to scale for KNN and, usually, for regression with regularisation.
4. Leaking information from the test set into preprocessing.
5. Skipping feature work. Good inputs beat clever algorithms. See feature engineering.

#Need a model built for your data?

We build and deploy machine learning systems, starting with a measured baseline. Get a scope estimate for your data project, or read how data science differs from analytics and ML.

#FAQ

#What is the difference between linear and logistic regression?

Linear regression predicts a continuous number. Logistic regression predicts the probability of a class using the sigmoid function, so it is used for classification.

#Is logistic regression a classification or regression algorithm?

Classification. The name comes from the underlying regression on the log-odds.

#When should I use a decision tree instead of logistic regression?

Use a tree when you need readable rules, when relationships are non-linear, or when features are a mix of numbers and categories. Use logistic regression when you want a stable, well-calibrated probability and a simple baseline.

#Why does KNN need scaled features?

KNN compares distances, so a feature with large numeric values dominates the distance. Scaling puts features on comparable ranges.

#How do I choose k in KNN?

Try several values with cross-validation and keep the one with the best held-out score. Odd values avoid ties in two-class problems.

#Do decision trees overfit?

Yes, if grown without limits. Restrict depth, set a minimum leaf size or prune, and compare against a held-out set.

#Which algorithm is best?

None is best everywhere. Start with a simple baseline, measure on held-out data and move to more complex models only if they clearly help.