Feature Engineering: The Skill That Makes or Breaks ML Models

Ask experienced data scientists what improves model performance most, and many will say the same thing: better features, not fancier algorithms. Feature engineering is turning raw data into inputs that make patterns easy for a model to learn. It's where domain knowledge meets data, and where many subtle bugs, especially leakage, hide.

#Key takeaways

• Good features encode domain knowledge in a form models can use.
• For tabular data, aggregations over time windows (counts, sums, recency, trends) are often the most predictive features.
• Prevent leakage: every feature must use only information available at prediction time.
• Put transformations in pipelines so training and serving compute features identically.
• Validate features with proper cross-validation, and drop features that don't help.

#Why features matter so much

Models learn relationships between inputs and targets. If the relationship is buried (hidden in a timestamp, spread across many transaction rows, encoded in free text), even powerful models struggle. A well-designed feature like "days since last purchase" or "support tickets in the last 30 days" hands the model the signal directly.

#Numeric features

#Scaling

Distance-based and gradient-based models (k-NN, SVMs, neural networks, regularized linear models) are sensitive to feature scales. Tree-based models generally aren't.

• Standardization: (x − mean) / std.
• Min-max scaling to [0, 1].
• Robust scaling (median and IQR) when outliers are present.

#Transformations

• Log transform for skewed values like revenue, counts or prices: log1p(x).
• Binning to capture non-linear effects for linear models (age groups), though trees handle this naturally.
• Clipping/winsorizing extreme outliers when they're errors or disproportionately influential.

#Ratios and interactions

Domain-driven combinations often beat raw values:

• revenue_per_visit = revenue / visits
• utilization = used_seats / purchased_seats
• price_vs_category_median = price / median_price_in_category

#Categorical features

Target encoding warning: computing the target mean on the full training data leaks the label into the feature. Always compute it out-of-fold (within cross-validation) and add smoothing for rare categories.

Handle unseen categories at serving time (an "unknown" bucket) and group rare categories into "other".

#Date and time features

Timestamps hide a lot of signal:

pythondf["hour"] = df.ts.dt.hour
df["day_of_week"] = df.ts.dt.dayofweek
df["is_weekend"] = df.day_of_week >= 5
df["month"] = df.ts.dt.month
df["days_since_signup"] = (df.ts - df.signup_at).dt.days

• Cyclical encoding for periodic features (hour 23 is close to hour 0): use sine and cosine transforms.
• Holidays and business calendars for demand and behavior models.
• Recency: time since the last event is frequently among the strongest predictors.

#Aggregation features (the powerhouse for tabular data)

Most business entities (customers, accounts, devices) have event histories. Aggregate them over windows relative to the prediction time:

pythondef customer_features(events, as_of):
    window = events[(events.ts < as_of) & (events.ts >= as_of - pd.Timedelta(days=90))]
    g = window.groupby("customer_id")
    return pd.DataFrame({
        "orders_90d": g.size(),
        "revenue_90d": g.amount.sum(),
        "avg_order_value_90d": g.amount.mean(),
        "days_since_last_order": (as_of - g.ts.max()).dt.days,
        "distinct_categories_90d": g.category.nunique(),
    })

Useful patterns:

• Counts, sums, means, max/min over several windows (7, 30, 90 days).
• Trends: last 30 days vs the previous 30 days.
• RFM: recency, frequency, monetary value, a classic for customer models.
• Behavioral ratios: support tickets per active user, failed payments per attempt.

#Text features

• Simple statistics: length, word count, presence of keywords.
• TF-IDF vectors for classic models.
• Embeddings from pre-trained models capture meaning and often work very well with simple classifiers on top (text embeddings explained).
• LLM-extracted features: sentiment, intent or topic labels generated by an LLM, validated on a labeled sample.

#Missing values

Missingness can itself be informative (no phone number provided).

• Tree models can often handle missing values natively.
• For other models, impute (median, mode, or a model-based approach) and add a missing indicator column.
• Never impute using statistics computed on the full dataset including the test set. Fit imputers on training data only.

#Data leakage: the silent killer

Leakage happens when a feature contains information that wouldn't be available when making real predictions. Classic examples:

• A churn model using "cancellation_reason" (only exists after churn).
• Aggregates computed over the full history, including the future relative to the prediction date.
• Target encoding or scaling fit on the whole dataset before splitting.
• Random train/test splits on time-series data, so the model trains on the future to predict the past.

Prevention:

1. Define an explicit prediction time for each example, and compute features using only data before it ("point-in-time correct" features).
2. Use time-based splits for temporal problems (time series forecasting).
3. Be suspicious of too-good-to-be-true validation scores and inspect top feature importances.

#Pipelines: consistency between training and serving

Feature logic implemented twice (once in a notebook, once in production code) will diverge. That's training/serving skew. Use pipelines:

pythonfrom sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", min_frequency=20), categorical_cols),
])
model = Pipeline([("prep", preprocess), ("clf", LogisticRegression(max_iter=1000))])
model.fit(X_train, y_train)   # the fitted pipeline is what you deploy

At larger scale, feature stores manage feature definitions, point-in-time correct training data and low-latency serving (MLOps explained).

#Feature selection

More features aren't always better: they add noise, overfitting risk and maintenance cost (overfitting and bias-variance).

• Remove near-constant and highly duplicated features.
• Use permutation importance on validation data to find features that genuinely help.
• Try ablation: remove a feature group and measure the change in cross-validated performance.
• Prefer fewer, stable, explainable features for production systems.

#FAQ

#Is feature engineering still needed with deep learning?

Less for unstructured data like images and text, where networks learn representations. For tabular business data it remains crucial, and even deep learning projects benefit from domain-informed inputs and careful leakage prevention.

#How do I know if a feature is useful?

Measure it. Compare cross-validated performance with and without the feature or feature group, and check permutation importance on held-out data. Intuition generates candidates; validation decides.

#What is a feature store?

A system that manages feature definitions, computes and stores features, provides point-in-time correct training datasets, and serves features with low latency in production, helping keep training and serving consistent.

#Can LLMs help with feature engineering?

Yes. LLMs can extract structured signals from text (intent, sentiment, entities), suggest candidate features from domain descriptions, and help write transformation code. Validate LLM-derived features like any other: with held-out evaluation and leakage checks.