Anomaly detection has a deceptively simple surface: observe a stream of data and identify when something unusual happens. In industrial IoT, that stream may represent thousands of observed signals produced by a physical process. Detecting unusual behavior in those signals—and doing so early enough for users to respond—is very much a challenging machine learning and system engineering problem.

Compared with some of the most popular problems in machine learning community, such as ads/feeds/search ranking in social media platforms, recommendation in marketplace or streaming service setting, real-time anomaly detection in industrial IoT is very much less studied and less understood.

With anomaly detection slowly climbs into the mainstream, a majority of available materials on internet still only focus on particular machine learning algorithms. However, just like those popular problems, solving real-time anomaly detection requires a rigorously engineered machine learning system. And, the problem remains one of the most challenging problems in machine leaning community till this day.

Over the past year or so, I have spent a lot of time thinking about products and developing machine learning solutions in this space. I have not found many articles that follow the entire journey, so I would like to work through it as a series. This first post defines the problem and develops a concrete theoretical model that we can use as an anchor. The second will walk through a product I have worked on and examine how the theoretical difficulties appear in practice. The third will move into machine learning system design and explore how data, models, evaluation, feedback, and serving can be assembled around the problem.

A Theoretical Model of Real-time Anomaly Detection

Anomaly detection can take several forms as a machine learning problem. When historical anomalies are labeled, a supervised model may learn to recognize them. When labels are sparse, incomplete, or unavailable, we may instead learn a representation of normal behavior and score observations by how much they deviate from it.

In practice, these cases often overlap: some failures are known, many events are poorly labeled, anomalous events we care about are rare, and future failures may look different from the ones already observed. This makes the target unusually elusive.

This becomes especially interesting in an industrial setting. A production process might generate thousands of telemetry signals: temperature, pressure, flow rate, machine speed, valve positions, and many others. These signals are noisy, dependent on one another, and shaped by a physical process that we often can only partially observe. On top of that, we would often like to detect a problem while it is happening, when the evidence is still incomplete.

Just like any problem which we employ machine learning system to solve, anomaly detection in real-time requires a rigorous design process. One thing I have found useful is to slow down before jumping into the actual shape of the system, and to understand things like: what exactly are we trying to detect? What does “real-time” mean? What information is available when a decision has to be made? And, if the detector makes a mistake, how should we think about its cost?

This post is the first in a series about real-time anomaly detection for industrial telemetry. I want to give this problem a chance to surface as a concrete shape because I haven’t really seen much about it in the community. In particular, I want to build a theoretical model, starting from a time series model and gradually arriving at a cost-sensitive real-time anomaly detection problem.

To avoid being too abstract, the theoretical model will be simplified, and several parts of it will remain hypothetical. The goal is to put the problem on paper so we can build a foundation before reasoning how to engineer a solution to it.

How to Model a Time Series

Let’s start with the observations.

Suppose we observe a multivariate time series over a discrete time domain. Let

be a random variable at time $t$, and let $x_t \in \mathbb R^d$ be its realization. The observed time series over a horizon of length $T$ is

We may also have some known external inputs or controls. Let $U_t \in \mathbb R^k$ be the random variable representing those inputs and $u_t$ be its realization. In an industrial process, $u_t$ might contain a machine setpoint, a valve command, a production recipe, or some other signal that influences the process and is available to us.

At time $t$, the observable history is

This history is useful because an individual observation rarely tells the entire story. A flow rate of 20 liters per minute might be ordinary after a machine starts and surprising after it has reached a steady operating state.

Having the observations and their history still does not fully describe the stochastic process that generated them. One convenient way to reason about that process is through a state-space model.¹ State-space model provides a unified view of time series.

Let $S_t \in \mathbb R^m$ represent the latent state of the system at time $t$. A probabilistic state-space model can be written as

Equation (3) gives us an initial distribution over the system state. Equation (4) describes how the state evolves, conditional on its previous state and the known input. Equation (5) describes how an observation is generated from the current state and input.

We can express the same idea using functions and explicit noise terms:

where $\eta_t$ represents process noise, such as a normal temperature fluctuation, and $\epsilon_t$ represents observation noise, such as a sensor noise. The distinction is conceptually useful. Process noise changes how the underlying system evolves; observation noise changes what a sensor reports about that system. In reality, separating them can be difficult.

Many time-series methods can be related to some part of this view. ARIMA models can be represented in state-space form, and exponential smoothing has a natural state-space formulation. A transformer-based model such as TimesFM² and a state-space-model-based architecture such as Time-SSM³ also learn predictive structure from sequences, although they do so through different architectures.

For anomaly detection, a common use of a forecasting model is straightforward. We predict the distribution or expected value of $X_t$ from the available history, observe $x_t$, and then measure how unlikely the observation is under that prediction. For example, a large residual or a low predictive likelihood can become evidence that something unusual has happened.

This already exposes our first difficulty. The true functions $\mathcal F$ and $\mathcal G$ are generally unknown. We learn some parameterized approximations from the data, and the quality of those approximations depends on what the data contains, what the model can represent, and whether the process continues to behave as it did during training.

Let’s keep this challenge in mind and move one step closer to anomaly detection.

From a Time Series to an Anomaly Score

One convenient way to formulate anomaly detection is to assign an anomaly score to an observation and then convert that score into a decision.

Suppose we want to assess the observation that occurred at time $t$. Let $\tau \geq t$ be the time at which we make that assessment. The information available to the detector is $h_\tau$, and we can define the score as

where $\mathcal R_\sigma$ is a scoring function parameterized by $\sigma$, and $y_{t,\tau} \in \mathbb R$ measures how anomalous the observation at time $t$ appears given the information available at time $\tau$.

The argument $t$ tells the function which observation in the history is being assessed. Writing the score this way also allows the detector to use a window, a learned state, including the known inputs and a short amount of information that arrives after $x_t$.

We can turn the score into a binary decision using a threshold:

where $a_{t,\tau}=1$ means the detector flags the observation at time $t$ as anomalous. The threshold $\beta_{t,\tau}$ may be fixed, or it may change with time, operating condition, or the amount of history available.

This definition is broad enough to cover several familiar approaches:

• A forecasting model can use $h_{t-1}$ to predict $X_t$ and derive $y_{t,t}$ from a residual or negative log-likelihood after $x_t$ arrives.
• A robust statistical method such as median absolute deviation (MAD) can score the distance between $x_t$ and a robust estimate of the center.
• Isolation Forest can score how readily an observation or a feature vector is isolated by randomized partitions.
• An autoencoder can use its reconstruction error as the anomaly score. If it receives a window of observations, the reconstruction can also reflect temporal structure within that window.

These methods do not make the same assumptions. A forecast-based score depends on the predictive model and its uncertainty. A deviation score depends on the reference window and its stability. Isolation Forest depends on the feature representation and the geometry by its partitions. An autoencoder depends heavily on how its training data can be reconstructed.

Still, equation (8) gives us a common place to put them together, and we can treat specific methods as instances. It lets us postpone the choice of algorithm and ask a more basic question: if a scoring function produces a large value, what would make the corresponding observation a true anomaly?

To express that distinction, let us have

As a latent ground truth indicator, where $Z_t=1$ means that the observation at time $t$ is truly anomalous. Let $z_t$ denote its realized value.

We are making a substantial assumption here. We assume that a true answer exists, even though the detector does not observe it at decision time. In many applications, the answer may remain uncertain afterward as well. For the moment, this hypothetical variable gives us something against which the detector’s decision can be evaluated. Later, it will also expose one of the more difficult parts of the formulation.

Adding “Real Time”

What does it mean for this detector to operate in real time?

There is always some elapsed time between an observation and a surfaced decision. The sensor has to produce the data, the data has to become available, and the detector has to compute a score. Adding up the latencies, it is quite unlikely to be real time. Sometimes, we may deliberately wait for a few additional observations because a short sequence provides stronger evidence than a single point, or a data point only becomes anomalous in a certain subsequence like a pattern anomaly. Let’s take the (necessary) delay into account to formulate the problem.

First, Let $\tau(t)$ be the time at which the detector makes its decision for the observation at time $t$. We can introduce an allowed delay $\Delta t$ and require

We can then land to the following definition.

This is a relative definition. It could be that $\Delta t$ is just a few milliseconds or seconds. But it could be that a decision made within several minutes could still create enough time for a useful response. A slowly developing process deviation might even permit a longer interval. The relevant deadline comes from the hypothetical action that the detection is meant to support.

The definition also creates a tension. Increasing $\tau(t)$ gives the detector more context and may improve the quality of the score. It also consumes more of the time available for action. Equation (11) places a boundary around that tradeoff without telling us where inside the boundary the decision should be made.

Adding the Cost of Being Wrong

Now we have an observation, a score, a binary decision, a hypothetical truth, and a time budget. We still need a way to evaluate quality and an objective to optimize. Here, let’s make our life easier by introducing a simplified loss function.

Let $c_t^{\mathrm{FN}} \geq 0$ be the cost of missing a true anomaly at time $t$, and let $c_t^{\mathrm{FP}} \geq 0$ be the cost of raising a false alarm. For the realized label $z_t$, we can define the loss as

If $z_t=1$, the false positive term becomes zero, and we pay a false negative cost when the detector does not alert. If $z_t=0$, the false negative term becomes zero, and we pay a false positive cost when it does alert.

This gives us one possible formulation of real-time anomaly detection:

The formulation asks us to choose a scoring function and threshold policy that minimize the average cost of errors while making each decision within the allowed delay. It is compact enough to fit on a few lines. Just by looking at it, however, we can see that several of its quantities are difficult to know.

The costs are one example. A false positive might consume an operator’s attention, interrupt production, or eventually cause user fatigue and lose trust. A false negative might lead to quality loss, equipment damage, or an unplanned shutdown. The cost may depend on variables that are unavailable at time $t$, and its full effect may only become visible much later. And, in practice, we could have an asymmetric cost, meaning that there exists a tradeoff of solving the problem (which we will omit its actual shape in here.)

Did we over-engineer the problem by adding all of this? Perhaps. Yet the cost terms remind us that two detectors with similar statistical accuracy can have different implications when their errors occur in different contexts. Even if we cannot estimate the costs precisely, the symbols give us a place to discuss that uncertainty.

What the Formulation Reveals

Let’s make the model a little more concrete with a hypothetical manufacturing process.

Suppose we are monitoring a production line that involves some chemical and physical processes. The line has sensors for temperature, pressure, flow rate, and machine speed. It also has controls and setpoints that influence how the process evolves. For simplicity, suppose our goal is to detect anomalies in flow rate because flow is related to throughput and product quality.

The process is only partially visible

The observed flow rate is generated by a process with an underlying state. That state could depend on upstream pressure, valve condition, material properties, the current recipe, recent control actions, and other variables. Some of them are measured. Some may be measured with error. Some is not measured at all.

In the notation above, $x_t$ is an observation produced through $\mathcal G$. If we learn $\mathcal F$ or $\mathcal G$ from historical data, we inherit the coverage and quality of that data. A model trained mostly on steady production may struggle during startup. A model trained during one product recipe may generalize poorly on a different recipe. Sensor noise could beinterpreted unintentionally as a meaningful process variation. A subtle and nuanced process change can be treated as noise and overly smoothed.

Different model families give us different ways to approximate these relationships. A simpler model can impose assumptions that make estimation more manageable. A more expressive model can capture richer relationships and may also fail to generalize well. Classic machine learning tradeoff, isn’t it?

A shortcut still contains assumptions

Could we skip the latent dynamics and apply an out-of-the-box anomaly detection method directly to the observations? We certainly can, and sometimes that could be a good way to solve the problem. However, these methods typically impose strong assumptions that might not fit the reality well.

For example, a MAD score requires us to decide which observations form the reference population. Isolation Forest requires a feature representation, which might include the current value, lagged values, rolling statistics, controls, or some combination of them. An autoencoder requires training data that represents the behavior we hope it will reconstruct well. Each method also needs a threshold that turns its score into a decision. And, each requires a heavy cycle of empirical study.

The true anomaly is still unknown

Now, suppose for a moment that we could model the dynamics of the production line almost perfectly. We know how temperature, pressure, valve commands, machine speed, and flow rate interact. An observation that is inconsistent with those dynamics would be unlikely to happen. Would it always produce $z_t=1$?

A low flow rate might be expected during startup, shutdown, or a product changeover. During steady production, the same rate could indicate a blockage or a failing pump. A sudden spike could reflect a process disturbance, a sensor fault, or a valid control response. A rare state might be harmless. Similarly, a small or statistically ordinary deviation could keeps compounding and leads to a costly failure.

The costs may only become clear afterward

The same uncertainty appears in $c_t^{\mathrm{FN}}$ and $c_t^{\mathrm{FP}}$. If an alert is raised during a critical production stage, inspecting the line would be expensive as we need to stop the production. If the alert prevents a larger quality problem, that expense would be justified. If no alert is raised, the consequence of the missed anomaly might depend on how long the condition persists and what happens downstream.

More evidence takes more time

Finally, consider the latency constraint. A single low flow-rate reading might be sensor noise. Several consecutive readings, combined with a pressure increase and an unchanged valve command, provide stronger evidence. Waiting gives the scoring function a richer history $h_{\tau(t)}$. Waiting also increases $\tau(t)-t$.

These difficulties do not make the formulation useless. The formulation helps us see them through explicitly.

Putting it together

We started with observations, introduced a state-space view, and then defined an anomaly score, a thresholded decision, a latent ground truth indicator, a time budget, and a cost-sensitive objective. The formulation is unlikely to be solved exactly. It can still serve as a reference: each proxy, algorithm, or evaluation procedure approximates some part of it and leaves other uncertainties unresolved.

What’s Next

The next question is how we might design a useful product around these abstractions. Such a product may need to help people express operating context, examine surfaced events, connect alerts to later outcomes, and gradually refine their understanding of an anomaly. In the next post, I will explore these questions through a concrete example and move the discussion from the theoretical model toward the product surface.