Linear Regression: From Fitting a Line to Regularized Models

Linear regression from a best-fit line to regularized models

This lesson begins a broader progression through machine learning: from supervised to unsupervised learning, from linear to nonlinear models, and from classical statistical methods to deep learning. Linear regression is a useful starting point because its prediction rule, loss function, assumptions, and limitations can all be inspected directly.

This lesson focuses on the practical workflow: how linear regression predicts, how loss guides fitting, how the model extends to multiple features, where ordinary linear regression becomes unreliable, and how regularization and cross-validation address specific weaknesses.

Learning Objectives

By the end of this lesson, you should be able to:

1. Explain what simple and multiple linear regression are trying to predict.
2. Interpret the intercept, coefficients, predictions, residuals, and squared-error loss.
3. Compare candidate regression lines using mean squared error.
4. Train simple and multiple linear regression models with scikit-learn.
5. Explain the main practical weaknesses of ordinary linear regression.
6. Distinguish Lasso regression from Ridge regression.
7. Explain why the regularization strength is a hyperparameter.
8. Select a Lasso or Ridge penalty with leakage-safe cross-validation.

---

1. The Prediction Problem

Suppose a factory wants to predict the daily energy consumption of a machine before the day is complete.

For the first model, the factory records only one feature:

• operating_hours: how many hours the machine runs during the day

The target is:

• energy_kwh: total energy consumed during that day

A small teaching dataset is:

The task is a regression problem because the model predicts a numeric quantity rather than a class label.

The points show a clear pattern: days with more operating hours usually consume more energy, but the observations do not fall perfectly on one line.

Machine operating hours plotted against daily energy use

The model is not trying to memorize every point. It is trying to learn a compact rule that captures the main relationship and can be applied to a future machine-day.

---

2. The Basic Idea of Simple Linear Regression

Simple linear regression uses one feature and predicts with a straight line:

$$\hat y = \beta_0 + \beta_1 x$$

where:

• $x$ is the input feature, machine operating hours
• $\hat y$ is the predicted daily energy use
• $\beta_0$ is the intercept
• $\beta_1$ is the slope or weight

In plain language:

Start with a baseline energy value, then add a fixed amount for every additional operating hour.

For example, consider:

$$\hat y = 20 + 6.2x$$

For a machine operating for 10 hours:

$$\hat y = 20 + 6.2(10) = 82 \text{ kWh}$$

The coefficient $6.2$ says that the model associates one additional operating hour with an increase of about $6.2$ kWh.

The intercept $20$ is the model's predicted energy use when $x=0$. It is mathematically necessary for positioning the line, but it is only physically meaningful when zero operating hours is a realistic value within the relevant data range.

Why a line?

A line is the simplest rule that can represent a constant rate of change. It is easy to fit, fast to evaluate, and directly interpretable.

Its limitation is equally important: one straight line cannot represent a relationship whose slope changes substantially across the input range.

---

3. Prediction Error and Loss

For observation $i$, the model makes prediction:

$$\hat y_i = \beta_0 + \beta_1x_i$$

The residual is:

$$e_i = y_i - \hat y_i$$

where:

• $y_i$ is the observed energy use
• $\hat y_i$ is the predicted energy use
• $e_i$ is the signed prediction error

A positive residual means the model predicted too low. A negative residual means it predicted too high.

For the 10-hour observation:

$$y=82, \qquad \hat y=82$$

so:

$$e = 82-82=0$$

For the 14-hour observation:

$$y=104, \qquad \hat y=106.8$$

so:

$$e = 104-106.8=-2.8$$

To evaluate an entire line, ordinary linear regression squares the residuals and adds them:

$$\operatorname{SSE}(\beta_0,\beta_1) = \sum_{i=1}^{n} \left(y_i-(\beta_0+\beta_1x_i)\right)^2$$

The mean squared error is:

$$\operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n}(y_i-\hat y_i)^2$$

Squaring has three effects:

1. Positive and negative residuals cannot cancel.
2. Larger mistakes receive more weight.
3. The objective becomes smooth and mathematically convenient.

The main limitation is that a very large residual receives a very large squared penalty, so outliers can strongly influence the fitted line.

---

4. Trying Different Weights and Comparing Loss

A linear model is defined by its intercept and slope. Different parameter values create different lines.

For the six machine-days, compare three candidates:

Three candidate regression lines with their mean squared errors

Line B has the smallest MSE among these three candidates, so it fits the observations best among them.

The actual least-squares solution for this dataset is approximately:

$$\hat y = 22.14 + 5.93x$$

Its parameters are not chosen by visual judgment. They are the values that minimize the sum of squared residuals over all possible lines.

What is the model learning?

The learning process is parameter estimation:

$$(\hat\beta_0,\hat\beta_1) = \arg\min_{\beta_0,\beta_1} \sum_{i=1}^{n} \left(y_i-(\beta_0+\beta_1x_i)\right)^2$$

The notation $\arg\min$ means “the parameter values that make the expression as small as possible.”

The fitted coefficients are written with hats because they are estimates learned from a sample, not immutable laws of the physical system.

---

5. A Complete Numerical Prediction

Using the least-squares line:

$$\hat y = 22.14 + 5.93x$$

suppose a machine will operate for 11 hours.

Substitute $x=11$:

$$\hat y = 22.14 + 5.93(11)$$ $$\hat y = 22.14 + 65.23$$ $$\hat y \approx 87.37 \text{ kWh}$$

This is a point prediction. It does not say that the machine must consume exactly 87.37 kWh. Measurement noise, load, ambient temperature, and omitted operating conditions can move the observed value above or below the prediction.

---

6. Simple Linear Regression with scikit-learn

The essential workflow is:

1. Place the feature in a two-dimensional table.
2. Split the data into training and test sets.
3. Fit the model on training data.
4. Evaluate predictions on data not used for fitting.
5. Predict a new observation.

import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import train_test_split


RANDOM_STATE = 42
FEATURE_COLUMNS = ["operating_hours"]
TARGET_COLUMN = "energy_kwh"


data = pd.DataFrame(
    {
        "operating_hours": [4, 6, 8, 10, 12, 14, 16, 18],
        "energy_kwh": [46, 58, 68, 82, 95, 104, 121, 132],
    }
)

X = data[FEATURE_COLUMNS]
y = data[TARGET_COLUMN]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=RANDOM_STATE,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, predictions))
print("RMSE:", root_mean_squared_error(y_test, predictions))
print("R²:", r2_score(y_test, predictions))
print("Intercept:", model.intercept_)
print("Slope:", model.coef_[0])

new_day = pd.DataFrame({"operating_hours": [11.0]})
print("Predicted kWh:", model.predict(new_day)[0])

What the outputs mean

• model.intercept_ estimates $\hat\beta_0$.
• model.coef_[0] estimates $\hat\beta_1$.
• model.predict() returns numeric predictions.
• MAE is the average absolute prediction error.
• RMSE gives extra weight to large errors and has the same unit as the target.
• $R^2$ measures improvement relative to predicting the target mean on the evaluated data.

A high $R^2$ on a tiny teaching dataset is not evidence that a production model will generalize well.

---

7. Upgrading to Multiple Linear Regression

Operating hours are not the only factor affecting energy use. Two machine-days with the same runtime may differ because one day has a heavier load or a higher ambient temperature.

Add two features:

• $x_1$: operating hours
• $x_2$: average load percentage
• $x_3$: ambient temperature in degrees Celsius

Multiple linear regression predicts:

$$\hat y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_3x_3$$

The same idea extends to $p$ features:

$$\hat y = \beta_0 + \sum_{j=1}^{p}\beta_jx_j$$

Multiple linear regression combines several weighted features

Suppose the fitted model is:

$$\hat y = 18 +6.4x_1 +0.92x_2 +1.25x_3$$

For a machine-day with:

$$x_1=14,\qquad x_2=72,\qquad x_3=27$$

we obtain:

$$\hat y = 18+6.4(14)+0.92(72)+1.25(27)$$ $$\hat y = 18+89.6+66.24+33.75$$ $$\hat y=207.59\text{ kWh}$$

How to interpret a coefficient

The coefficient $\beta_1=6.4$ means:

Among observations with the same included load percentage and ambient temperature, increasing operating time by one hour is associated with an expected increase of 6.4 kWh.

The phrase “holding the other included variables fixed” is essential.

A coefficient is not automatically causal. If important variables are omitted, measurements are biased, or the data were generated by a selection process, the coefficient may describe association rather than intervention.

---

8. Multiple Linear Regression Code

import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import train_test_split


RANDOM_STATE = 42
FEATURE_COLUMNS = [
    "operating_hours",
    "average_load_pct",
    "ambient_temperature_c",
]
TARGET_COLUMN = "energy_kwh"


data = pd.read_csv("machine_energy.csv")

X = data[FEATURE_COLUMNS]
y = data[TARGET_COLUMN]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=RANDOM_STATE,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, predictions))
print("RMSE:", root_mean_squared_error(y_test, predictions))
print("R²:", r2_score(y_test, predictions))

coefficient_table = pd.DataFrame(
    {
        "feature": FEATURE_COLUMNS,
        "coefficient": model.coef_,
    }
)
print(coefficient_table)

new_machine_day = pd.DataFrame(
    [
        {
            "operating_hours": 14.0,
            "average_load_pct": 72.0,
            "ambient_temperature_c": 27.0,
        }
    ]
)

predicted_energy = model.predict(new_machine_day)[0]
print(f"Predicted energy: {predicted_energy:.2f} kWh")

The full runnable example in source_code/example.py creates a reproducible dataset, trains simple and multiple regression models, evaluates them, and adds Lasso and Ridge cross-validation.

---

9. Main Weaknesses of Ordinary Linear Regression

Ordinary least squares is a strong baseline, but it is not automatically appropriate for every dataset.

Four common weaknesses of ordinary linear regression

9.1 A linear function may be too simple

The model assumes that the expected target is a linear combination of the supplied features.

If energy use rises slowly at first and then sharply near maximum machine load, a single constant slope will leave a systematic curved pattern in the residuals. The model is then underfitting the relationship, even when its training error appears reasonable.

9.2 Squared loss is sensitive to outliers

Because the loss squares every residual, one extreme observation can contribute more to the objective than many ordinary observations combined.

A faulty meter reading or an unusual shutdown day can pull the fitted line toward itself and distort predictions for typical machine-days.

9.3 Correlated predictors make coefficients unstable

operating_hours and runtime_minutes contain almost the same information. When both are included, many coefficient combinations can produce nearly identical predictions.

Predictions may remain acceptable while individual coefficients change sharply across samples. This makes coefficient interpretation unreliable and is a common symptom of multicollinearity.

9.4 Too many predictors can increase variance

When the number of candidate features is large relative to the amount of training data, ordinary least squares can assign coefficients to accidental noise.

Training error may continue to decrease while test error increases. The fitted model becomes too dependent on the particular sample used for training.

9.5 Extrapolation can be unreliable

A line fitted on machines operating between 4 and 20 hours can still return a number for 30 hours, but that number is outside the region supported by the data.

The physical process may change beyond the observed range, so the model's numerical output should not be mistaken for evidence.

9.6 Predictive association is not causation

A coefficient describes the relationship found after conditioning on the included features. It does not prove that deliberately changing a feature will change the target by that amount.

Omitted variables, selection effects, and measurement bias can create associations that are not causal.

9.7 Statistical conclusions require additional assumptions

The model can always calculate coefficients when the numerical problem is solvable. However, confidence intervals, hypothesis tests, and standard errors require assumptions about sampling, dependence, variance, and model specification.

A useful prediction equation and a valid inferential analysis are related but different goals.

---

10. How to Address These Weaknesses

The appropriate response depends on the failure mode. Regularization is important, but it solves only a subset of linear regression's weaknesses.

The next two methods focus specifically on unstable or overly flexible coefficient estimates:

• Lasso adds an absolute-value penalty and can remove some features by setting their coefficients to zero.
• Ridge adds a squared-coefficient penalty and shrinks all coefficients smoothly, which is especially useful for correlated predictors.

Neither method automatically fixes nonlinear structure, corrupted observations, causal bias, unsupported extrapolation, or distribution shift.

---

11. Lasso Regression: Shrink and Select

Lasso regression adds an $L_1$ penalty:

$$\hat\beta^{\text{lasso}} = \arg\min_{\beta_0,\beta} \left[ \sum_{i=1}^{n} \left(y_i-\beta_0-x_i^T\beta\right)^2 + \alpha\sum_{j=1}^{p}|\beta_j| \right]$$

where:

• $x_i$ is the feature vector for observation $i$
• $\beta$ contains the feature coefficients
• $\alpha\geq0$ controls penalty strength
• the intercept is usually not penalized

The first term rewards accurate predictions. The second makes large absolute coefficient values expensive.

The absolute-value penalty has a sharp corner at zero. Because of this geometry, the fitted solution can place some coefficients exactly at zero.

Lasso regression can produce exactly zero coefficients

Why Lasso helps

Lasso is useful when:

• many candidate features may be irrelevant
• a sparse model is valuable for deployment or communication
• automatic feature selection is part of the modeling goal

For the machine-energy example, a weak sensor or a redundant derived variable may receive a coefficient of exactly zero.

What Lasso does not guarantee

When several predictors are strongly correlated, Lasso may select one and suppress another somewhat arbitrarily. The selected feature set can change across training samples.

A zero coefficient means the feature was not selected by this fitted model under this penalty strength. It does not prove that the feature has no physical relationship with energy use.

---

12. Ridge Regression: Shrink All Coefficients

Ridge regression adds an $L_2$ penalty:

$$\hat\beta^{\text{ridge}} = \arg\min_{\beta_0,\beta} \left[ \sum_{i=1}^{n} \left(y_i-\beta_0-x_i^T\beta\right)^2 + \alpha\sum_{j=1}^{p}\beta_j^2 \right]$$

The first term asks the model to fit the observations. The second asks it to keep coefficients small.

Ridge regression shrinks standardized coefficients

Why Ridge helps

When correlated predictors compete to explain the same signal, ordinary least squares may assign one a large positive coefficient and another a large negative coefficient. Ridge makes such extreme combinations expensive.

This introduces some bias but can substantially reduce variance. The resulting model may make more stable predictions on new observations.

What Ridge does not do

Ridge usually does not remove features. It shrinks coefficients toward zero but typically leaves them nonzero.

Ridge also does not identify which correlated measurement is the uniquely correct explanation. It mainly stabilizes the fitted prediction rule.

---

13. Lasso vs. Ridge

Neither method is universally better. The choice should reflect the modeling goal and be evaluated with the same validation procedure.

Elastic Net combines $L_1$ and $L_2$ penalties. It is often useful when sparsity is desirable but the predictors also contain strongly correlated groups.

---

14. What Is the Hyperparameter?

The central Lasso and Ridge hyperparameter is the penalty strength, usually written as $\alpha$ or $\lambda$.

Parameter vs. hyperparameter

A parameter is learned during model fitting:

• intercept $\beta_0$
• coefficients $\beta_1,\ldots,\beta_p$

A hyperparameter controls the fitting procedure and must be selected outside the final coefficient optimization:

• Lasso or Ridge penalty strength $\alpha$

Effect of Alpha

• $\alpha=0$: no regularization; the objective reduces to ordinary least squares
• small $\alpha$: weak shrinkage
• large $\alpha$: strong shrinkage and greater risk of underfitting

For Lasso, software generally expects a strictly positive value rather than using alpha=0 as an ordinary least-squares solver.

Why scaling matters

Suppose operating hours range from 4 to 20 while runtime minutes range from 240 to 1200. A penalty applied directly to raw coefficients would depend heavily on the measurement units.

Standardization transforms each feature using training-set statistics:

$$z_j = \frac{x_j-\bar x_j}{s_j}$$

where $\bar x_j$ is the training mean and $s_j$ is the training standard deviation.

After standardization, the penalty compares coefficients attached to similarly scaled features.

The scaler must be fitted only with training data. During cross-validation, it must be fitted separately inside each fold. A scikit-learn Pipeline enforces this workflow.

---

15. Selecting Alpha with Cross-Validation

Choosing $\alpha$ from training error would favor little or no regularization. The penalty must instead be judged by performance on observations not used to estimate the coefficients.

K-fold cross-validation works as follows:

1. Split the training data into $K$ folds.
2. Hold out one fold for validation.
3. Fit the complete preprocessing-and-model pipeline on the other $K-1$ folds.
4. Evaluate the held-out fold.
5. Rotate the held-out fold until every fold has been used once.
6. Average the validation metric.
7. Repeat for every candidate $\alpha$.
8. Select the best $\alpha$, refit on all training data, and evaluate the test set once.

Cross-validation workflow for selecting regularization strength

Leakage-safe Lasso and Ridge code

import numpy as np

from sklearn.linear_model import Lasso, Ridge
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


ALPHA_GRID = np.logspace(-3, 3, 49)
CV_FOLDS = 5
RANDOM_STATE = 42


def build_regularized_search(model_name: str) -> GridSearchCV:
    if model_name == "lasso":
        regressor = Lasso(
            max_iter=50_000,
            tol=1e-5,
            random_state=RANDOM_STATE,
        )
    elif model_name == "ridge":
        regressor = Ridge()
    else:
        raise ValueError("model_name must be 'lasso' or 'ridge'.")

    pipeline = Pipeline(
        steps=[
            ("scaler", StandardScaler()),
            ("regressor", regressor),
        ]
    )

    return GridSearchCV(
        estimator=pipeline,
        param_grid={"regressor__alpha": ALPHA_GRID},
        scoring="neg_root_mean_squared_error",
        cv=CV_FOLDS,
        n_jobs=-1,
        refit=True,
    )

Training Lasso:

lasso_search = build_regularized_search("lasso")
lasso_search.fit(X_train, y_train)

best_lasso = lasso_search.best_estimator_
best_lasso_alpha = lasso_search.best_params_["regressor__alpha"]
test_predictions = best_lasso.predict(X_test)

Training Ridge:

ridge_search = build_regularized_search("ridge")
ridge_search.fit(X_train, y_train)

best_ridge = ridge_search.best_estimator_
best_ridge_alpha = ridge_search.best_params_["regressor__alpha"]
test_predictions = best_ridge.predict(X_test)

scikit-learn uses a negative RMSE score because its model-selection interface maximizes scores. The candidate with the value closest to zero has the lowest RMSE.

What should remain untouched?

The test set must not influence:

• feature scaling
• feature selection
• alpha selection
• model comparison
• stopping decisions

Use it once at the end to estimate final generalization performance.

---

16. Practical Model-Selection Guidance

A disciplined progression is:

1. Fit ordinary linear regression as a baseline.
2. Inspect residuals and test performance.
3. Add scientifically justified features or transformations.
4. Use Lasso when a sparse feature set is valuable.
5. Use Ridge when coefficient instability or broad shrinkage is the main concern.
6. Consider Elastic Net when correlated feature groups and sparsity are both important.
7. Select hyperparameters with cross-validation.
8. Compare all candidates on the same untouched test set.
9. Report uncertainty and operational limitations, not only a single metric.

A regularized model is not automatically better. When the dataset is large, the feature set is small, and multicollinearity is mild, ordinary least squares may perform just as well and remain easier to interpret.

---

17. Complete Practical Implementation

The accompanying program demonstrates:

• reproducible machine-energy data creation
• a train/test split
• simple linear regression
• multiple linear regression
• Lasso and Ridge pipelines
• standardization inside cross-validation
• alpha selection with GridSearchCV
• MAE, RMSE, and $R^2$
• coefficient reporting
• one new prediction

Run it from the package directory:

python source_code/example.py

The program intentionally includes redundant and weak predictors. This makes Lasso sparsity and Ridge shrinkage visible in a realistic modeling workflow.

---

18. Connecting Code to the Main Ideas

All three regressors return numeric predictions. Their outputs are not probabilities, confidence intervals, or causal effects.

---

19. Key Takeaways

Prediction rules

Simple linear regression:

$$\hat y=\beta_0+\beta_1x$$

Multiple linear regression:

$$\hat y=\beta_0+x^T\beta$$

Learning objective

Ordinary least squares selects coefficients that minimize squared residuals:

$$\sum_i(y_i-\hat y_i)^2$$

Main weaknesses

Linear regression can underfit nonlinear relationships, react strongly to outliers, produce unstable coefficients with correlated predictors, and overfit when many weak features are available.

Lasso

$$\operatorname{SSE}+\alpha\sum_j|\beta_j|$$

Lasso shrinks coefficients and can set some exactly to zero, producing a sparse model.

Ridge

$$\operatorname{SSE}+\alpha\sum_j\beta_j^2$$

Ridge shrinks coefficients smoothly and is especially useful when predictors are correlated or coefficient estimates have high variance.

Hyperparameter selection

The penalty strength $\alpha$ controls the trade-off between fitting the training data and shrinking the coefficients. Select it through cross-validation with preprocessing inside a pipeline.

Final warning

A fitted equation is not automatically a trustworthy model. Validation, residual inspection, data quality, deployment range, and the distinction between prediction and causation still matter.