Introduction to Machine Learning

Machine learning concepts organized before individual algorithms

Machine learning is often introduced through a list of algorithms. That approach can make the field feel like a collection of unrelated recipes: one method for classification, another for clustering, another for images, and another for language.

A better starting point is to understand the common problem behind the algorithms.

Machine learning uses data to construct a rule, representation, or model that remains useful beyond the examples used to build it.

This chapter establishes that foundation before the series moves through:

• supervised learning and unsupervised learning,
• linear models and nonlinear models,
• statistical models and deep learning models.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain what machine learning is trying to accomplish at a high level.
2. Distinguish training from prediction or discovery.
3. Explain supervised learning using features, targets, and labeled examples.
4. Explain unsupervised learning using unlabeled observations and structural objectives.
5. Compare supervised and unsupervised learning without treating one as a weaker version of the other.
6. Describe generalization and why training performance alone is insufficient.
7. Explain the roles of inductive bias, validation data, and test data.
8. Understand why unsupervised learning may have several reasonable answers.

---

1. What Is Machine Learning Trying to Do?

Suppose a factory records two measurements from each machine every hour:

• temperature deviation from normal,
• vibration deviation from normal.

A small dataset might look like this:

A human engineer could try to write a rule manually:

Trigger an alert when temperature is high and vibration is high.

That may work for a few simple cases, but real relationships are often too complicated, noisy, or changeable to express with a fixed hand-written rule.

Machine learning takes a different approach:

1. Show the computer historical observations.
2. Define what kind of result would be useful.
3. Let a learning algorithm search for a pattern that serves that result.
4. Evaluate whether the learned pattern works on new observations.

The learned object is called a model. Depending on the task, the model may produce:

• a predicted class,
• a predicted number,
• a ranking,
• a grouping of similar observations,
• a compact representation,
• an estimate of how unusual an observation is,
• or a generated sample resembling the data.

The central idea is not simply “find a pattern.” Almost any sufficiently flexible method can find patterns in a finite dataset, including meaningless ones. The goal is to find a pattern that is useful for the intended task and reliable on future data.

1.1 Learning Is Not the Same as Memorizing

A lookup table can memorize every training example:

• Observation 1 → fault
• Observation 2 → fault
• Observation 3 → no fault

But the lookup table says nothing about a new machine state that has never appeared before.

Machine learning is useful when the model can transfer what it learned from past examples to a new case. This ability is called generalization.

A model that performs perfectly on its training data but poorly on new data has not learned the intended relationship. It has overfit the sample.

1.2 Machine Learning Is a Goal-Directed Process

The same data can support different questions.

Using the factory sensor data, we might ask:

• Will this machine fail within 24 hours?
• How much remaining operating time does it have?
• Which machine states look similar?
• Does this observation look unusual?
• Can the two sensor readings be compressed into one health indicator?

These questions require different outputs and different criteria for success. There is no universally best model independent of the task.

A machine learning workflow from question definition to monitoring

A complete machine learning workflow therefore includes more than fitting an algorithm. It includes:

• defining the question,
• collecting and representing data,
• choosing an objective,
• fitting a model,
• evaluating it on unseen data,
• deploying or using the result,
• and monitoring whether the data or environment changes.

---

2. The Basic Vocabulary: Examples, Features, Targets, and Models

Before comparing supervised and unsupervised learning, we need a few common terms.

2.1 Observation or Example

One row of a dataset is an observation, example, sample, or data point.

For the machine-monitoring problem, one observation could be the sensor state of one machine at one hour.

2.2 Features

The information provided to a model is represented by features.

For one machine state, we can write:

$$x= \begin{bmatrix} \text{temperature deviation}\\ \text{vibration deviation} \end{bmatrix}$$

The symbol $x$ represents the input feature vector.

Features are not automatically useful just because they are available. A model cannot reliably predict a fault if the inputs contain no information related to faults. Feature choice and data quality are therefore part of the learning problem.

2.3 Target or Label

In some tasks, each training example includes the answer we want the model to learn to predict.

Examples include:

• fault or no fault,
• the amount of energy used tomorrow,
• the relevance score of a search result.

This answer is called the target, label, or response and is often written as $y$.

2.4 Model

A model is a parameterized rule that maps an input to an output or representation.

For a predictive task, we can write this abstractly as:

$$\hat y=f(x)$$

where:

• $x$ is the input,
• $f$ is the learned model,
• $\hat y$ is the model's prediction.

The hat in $\hat y$ means “estimated” or “predicted.” It distinguishes the model's output from the true target $y$.

2.5 Learning Algorithm

The model is the learned rule. The learning algorithm is the procedure used to select that rule from data.

This distinction matters:

• A model makes predictions or produces representations.
• A learning algorithm adjusts or selects the model during training.

---

3. Supervised Learning

3.1 The Main Idea

In supervised learning, the training data contains both:

• the input features $x$,
• the desired target $y$.

The training dataset therefore consists of input–target pairs:

$$(x_1,y_1), (x_2,y_2), \ldots, (x_n,y_n)$$

The word supervised does not mean that a person watches the algorithm train. It means the learning process receives an answer signal for each training example.

Using the machine-monitoring example:

The model sees both the measurements and the historical outcome. It uses the mismatch between its current prediction and the known target to improve.

3.2 The Question Supervised Learning Asks

Supervised learning asks:

Given examples with known answers, can we learn a rule that predicts the answer for a new input?

For a new machine state:

$$x_{\text{new}}= \begin{bmatrix} 1.6\\ 1.2 \end{bmatrix}$$

we want the model to produce a prediction such as:

$$\hat y_{\text{new}}=\text{fault}$$

The target for the new example may not be known at prediction time. The model must use the relationship learned from historical examples.

3.3 Common Supervised Tasks

Classification

The target is a category.

Examples:

• fault or no fault,
• fraudulent or legitimate transaction,
• defective or acceptable product.

Numeric prediction

The target is a numerical value.

Examples:

• remaining operating hours,
• tomorrow's energy demand,
• delivery time.

Ranking

The target concerns relative order or relevance.

Examples:

• which maintenance alerts should be inspected first,
• which search results are most relevant,
• which products are most likely to be useful to a user.

These tasks use different objective functions and evaluation metrics, but they share the same supervised structure: training examples include a target signal.

3.4 What the Answers Actually Supervise

Targets guide the model toward one particular use of the data.

Suppose two machines have similar temperature and vibration readings. A supervised model does not group them merely because they look similar. It asks whether treating them similarly helps predict the target.

This is important because a feature may be useful for one target and irrelevant for another.

For example:

• vibration may be highly useful for predicting mechanical faults,
• temperature may be more useful for predicting cooling-system load,
• neither feature may be sufficient for predicting operator error.

Supervision determines what “useful similarity” means.

---

4. Unsupervised Learning

4.1 The Main Idea

In unsupervised learning, the training data contains inputs but no answer label for the intended task:

$$x_1,x_2,\ldots,x_n$$

The algorithm must organize, summarize, represent, or model the observations using structure in the inputs themselves.

For example, the machine data may contain only sensor readings:

No column tells us which examples are faulty.

An unsupervised method might discover:

• two common operating regions,
• a small number of representative machine states,
• one unusual observation far from the common patterns,
• a compact health representation combining the two sensors.

4.2 The Question Unsupervised Learning Asks

Unsupervised learning asks a broader question:

What structure in the observations is useful for the objective we have chosen?

That wording is deliberate. Without labels, the data does not announce one uniquely correct structure.

The same observations could be grouped by:

• operating mode,
• machine type,
• maintenance condition,
• production shift,
• environmental temperature.

Different representations and objectives can reveal different structures.

4.3 Common Unsupervised Tasks

Clustering

Group observations so that examples in the same group are similar according to a chosen representation and similarity measure.

In machine monitoring, clusters might correspond to idle, normal-load, and heavy-load operation.

Dimensionality reduction and representation learning

Construct a smaller set of variables that preserves important information.

Dozens of sensor measurements might be summarized by a few latent health factors.

Density estimation

Model which regions of the input space are common and which are rare.

This can support simulation, uncertainty estimation, or anomaly detection.

Anomaly detection

Assign a high anomaly score to observations that do not resemble typical data.

The sensor reading $(0.3,2.8)$ might be unusual because high vibration occurs without the temperature pattern seen in common operating states.

Generative modeling

Learn enough about the data distribution to produce new samples with similar structure.

The generated samples are not copied rows. They are new observations produced from the learned distribution or representation.

4.4 Unsupervised Does Not Mean “No Objective”

A common misunderstanding is that unsupervised learning simply lets an algorithm freely inspect data until it finds something interesting.

In practice, every learning method still needs an objective or criterion. Examples include:

• make points within a cluster close together,
• reconstruct an input after compressing it,
• assign high probability to observed data,
• distinguish typical observations from rare ones.

The absence of labels changes where the training signal comes from. It does not eliminate the need for a training signal.

---

5. Supervised and Unsupervised Learning Side by Side

Supervised and unsupervised learning applied to the same sensor data

5.1 Neither Category Is Inherently More Advanced

Unsupervised learning is not supervised learning with missing labels. It is a different problem definition.

Supervised learning is often easier to evaluate because a known target provides a direct reference. Unsupervised learning may require indirect evaluation because several structures can be reasonable.

However, supervised learning can be difficult when:

• labels are expensive or unreliable,
• the future differs from the training period,
• the target is only a rough proxy for the real decision,
• rare cases are underrepresented.

Unsupervised learning can be valuable when labels are unavailable, when the goal is exploration or representation, or when the structure itself is the object of interest.

5.2 The Boundary Is Useful, Not Absolute

Real systems often combine learning settings.

For example, a factory might:

1. learn a representation from millions of unlabeled sensor readings,
2. use a smaller labeled dataset to predict faults,
3. continue monitoring new unlabeled data for distribution changes.

The supervised–unsupervised distinction describes where the learning signal comes from. It does not require every project to fit permanently into only one box.

---

6. Training, Evaluation, and Generalization

6.1 Training Data

The training set is used to fit the model.

During training, the learning algorithm repeatedly uses the training examples to choose model parameters or structure.

6.2 Validation Data

The validation set is used to make development decisions, such as:

• choosing among model types,
• selecting hyperparameters,
• deciding when to stop training,
• choosing a decision threshold.

Because validation performance influences these decisions, the validation set is part of the model-development process.

6.3 Test Data

The test set is reserved for a final, relatively unbiased estimate of performance on unseen data.

If the test results repeatedly influence model changes, the test set gradually becomes another validation set. Its estimate is no longer clean.

6.4 Why Unseen Data Matters

The model is trained on a finite sample. We ultimately care about its behavior on future observations drawn from the environment in which it will be used.

A strong training score can result from:

• learning a real relationship,
• memorizing noise,
• exploiting data leakage,
• benefiting from duplicated examples,
• fitting quirks that will not repeat.

Evaluation on unseen data helps distinguish these possibilities.

6.5 Data Leakage

Data leakage occurs when information unavailable at real prediction time enters model training or evaluation.

For example, a feature called “repair completed within 48 hours” would make historical fault prediction easy, but the feature is only known after the event. A model using it would appear excellent in testing yet fail in real operation.

The split between training and evaluation data must reproduce the information and timing constraints of the real task.

---

7. What Makes a Machine Learning Problem Learnable?

A learning algorithm cannot create information that is absent from the data. Successful machine learning usually depends on several conditions.

7.1 The Inputs Must Contain Useful Signal

If future faults are unrelated to temperature and vibration, no algorithm can reliably predict them from those features.

A more complicated model cannot solve a fundamentally uninformative problem.

7.2 The Training Data Must Represent the Intended Use

A model trained only on one machine type may not work on another. A model trained in winter may behave differently in summer. A model trained before a manufacturing process changes may become outdated.

The relationship between training data and future data is as important as the amount of data.

7.3 The Objective Must Match the Real Goal

Suppose a fault-detection model achieves high overall accuracy by ignoring rare but dangerous failures. The training metric may look good while the operational result is unacceptable.

The objective and evaluation metrics should reflect the costs of different mistakes.

7.4 The Model Needs an Appropriate Inductive Bias

Many patterns can fit a finite dataset. A learning method needs preferences that guide it toward some patterns rather than others.

These preferences are called inductive bias.

Examples include preferences for:

• simpler rules,
• smooth relationships,
• nearby examples behaving similarly,
• sparse explanations,
• hierarchical representations,
• invariance to certain transformations.

Without inductive bias, generalization would be impossible: the data alone cannot specify how the model should behave at every unseen point.

7.5 Evaluation Must Reflect Reality

Randomly splitting rows is not always appropriate.

For time-dependent machine data, a realistic evaluation may train on earlier periods and test on later periods. If repeated measurements come from the same machine, all rows from one machine may need to remain in the same split.

The evaluation design should answer the real question: “How will this system perform when used?”

---

8. Common Misunderstandings

8.1 “Machine Learning Discovers the True Rule”

A model estimates a useful relationship from finite, noisy observations. The learned rule depends on:

• the available data,
• the features,
• the objective,
• the model family,
• the optimization procedure,
• and the evaluation design.

It may approximate an underlying mechanism, but it is not automatically the true physical or causal explanation.

8.2 “More Data Always Solves the Problem”

More representative data often helps, but it does not fix:

• incorrect labels,
• missing relevant features,
• a badly defined target,
• systematic bias,
• data leakage,
• distribution shift,
• an inappropriate model or objective.

Data quantity and data quality are different properties.

8.3 “A High Accuracy Means the Model Is Good”

Accuracy is only one metric. Its usefulness depends on class balance, error costs, threshold choice, and the deployment context.

For unsupervised learning, there may not even be a natural accuracy measure because no target labels were provided.

8.4 “Unsupervised Learning Finds the Natural Clusters”

Data can support several valid groupings. A clustering result depends on:

• the feature representation,
• the distance or similarity measure,
• the number of groups,
• the objective,
• and the scale of the features.

Clusters are model-dependent summaries, not automatically objective categories existing independently of all assumptions.

8.5 “The Model's Score Is Automatically a Probability”

Many models produce raw scores, distances, or anomaly measures. These values should not be interpreted as probabilities unless the model and output are specifically defined and, when necessary, calibrated for that interpretation.

---