A generic, low resource usage, single input pre-processor for an anomaly detection algorithm

The general nature can be adapted for both analog or digital signals.

Typical signal changes

Spikes - a sudden change and recovery of the value.

Noise - semi/periodic fluctuations.

Step - a sudden large increase or decrease in the value without hysteresis recovery.

Drift - a slow & continuous change in value in the same direction (e.g., increasing or decreasing)

Static - no change in behaviour

Behaviour table

Each of the behaviors can be represented as amplitude, rate and direction changes.

Sampling considerations

The input signal is ideally as raw as possible; however, it may need to be preprocessed to better determine the 'intent' of the signal.
- Taking a number of samples and determining an appropriate average, minimum & maximum.
- Comparison of the current processed values against the previous processed values. This data could be stored as vector for a single parameter of time series data.
  - E.g., if a time series signal vector consists of: Min, Avg, Max then:
    - An increasing signal (or the opposite for decreasing) could be detected by comparing the current to the n-1, n-2... vectors and determining if:
      - The new maximum values are greater than the previous maximum value.
      - The new average value is greater than the previous average value.
      - The new minimum value is greater than the previous minimum value.
    - A step change or spike could be detected where:
      - The new minimum value is greater than the previous average value.
      - The new average value is greater than the previous average value.
      - A new maximum value is set.
    - Noise could be quantified by the samples range.
    - Drift where the rates of change are very slow but consistent in direction (e.g., increasing, or sometimes flat but not decreasing).
    - Static where the Min, Avg and Max values are very close.

Algorithm

Sample new data.
Check new value against thresholds.

If the data is a new maximum or minimum, update the max / min value, range & thresholds.

If new value is between threshold's (not above or below), decay the thresholds appropriately (self-modifying).
- The thresholds shall be determined as a function of the input signal (e.g., percentage of range) and used to classify the signals' instantaneous location.
Check for static data (e.g., data is not changing, or change results in the same classification).
Determine the location of the signal (above threshold, middle, below threshold).

If a change, compare current classification and previous location to determine direction.

6. A cycle (defined by two direction changes of a single variable) can be used to determine a frequency with a defined time period.

Further Classification

Known thresholds

System Modes e.g., startup - where a full cycle has not yet occurred. This condition is known and therefore a mask of such behavior shall occur and state indication e.g., waiting / training (where training in this scenario requires one or more cycles to occur).
Hardware failure e.g., sensor is flooded or failed, or power supply is not reliable.

Classification of undesired behavior (e.g., pre considered event / scenario detection)

E.g., a slow-moving signal (e.g., a low-rate water leak).
Abnormally high-water rate (e.g., a pipe rupture).
Long duration events (e.g., a sprinkler left on too long).
High water volumes (total usage over a period of time is abnormally high).

Anomaly detection (indication the system is not operating normally)

Min is very similar to max value (low range) and therefore vary small classification area (normal when first started but abnormal when otherwise).
- Abnormally long periods in one classification area (can be a genuine no water use scenario e.g., away). Indicates as abnormal but when couple with other information via a higher-level observer is ok.
Changes in the noise measurement.

Application - Water meter digitiser

A generalised version of the algorithm was adapted to operate on a low-cost micro controller and count the number of rotations of the spinning indicator of a water meter. It had to adapt to continuously changing conditions (hence adaptive thresholds resulting in adaptive classification regions) as to account for the ever-changing amount of condensation, dirt or flooding during a rain event (below ground pit).

Itron residential water meter

The small red dial, with metalic wedge is used to reflect light. One rotation indicates 1 liter of water consumption.

3D printed adaptor

An IR sensor adaptor was made to fit the manufacture's accessory clip. This held the IR sensor in the exact location to observer the spinning indicator.

Harsh and changing conditions

Sensor fitted and after a period of time was flooded. The adaptative thresholds ensured the processed data (water consumption data) was not affected.

Preprocessing code

The following is an application of the code running on a low power microcontroller. It has been running successful (recorded values match actual meter values).

An overview:

Read the input signal and calibrate.
Check if there is a new min. max value if so, update accordingly.
Decay the min / max if a new min max has not occurred for a set period of time.
Calculate new thresholds.
Calculate the range.
Check if the value is static.
Determine the IR states (above threshold / below threshold).
Determine if a cycle has occurred and if so, increment the counter.

Note, the code snip is from the functioning water meter reader. It was written very quickly as a test but worked so well, it has not been changed for years.

Single variable classification

Three defined regions:

1: Min to low limit.

2: Low limit to high limit.

3: High limit to Max

Note: If the value is not in region 1 or 3 then it is in region 2.

Raw input signal during water flow

Input signal (water in green), adaptive thresholds (limits) and recorded Min & Max values.

Processed signal

When the peaks and troughs of the input signal cross the threshold a new state is recorded. When a state transitions from one state and back, a cycle is completed, and a counter incremented.

This trend shows the slow nature of a tap left to flow at a minimum rate (dribble). This is shown by the slow change in the raw signal, followed by the tap at max rate and the much higher frequency of the oscillations.

A closer look at the typical amount of sensor noise when there is no flow.

A tap that is turned on, and then, turned off.

Adaptive thresholding. The high limits decay over a duration where there isn't any significant change in the input signal.

Improvements

Rather than multiplication use bit shifting (uses less resources) for threshold calculation.
Calculate the range / thresholds only if there is change.
Detect a slow leak if there is a long duration between cycles that is a typical of a slow running / short tap duration.
If the thresholds become too close, it could be a sensor error or failure.
Implement an observer: track typical daily maximum / minimum values, typical noise levels.

Jan 13, 2024

After weeks of periodic storms and heavy rain resulting in frequent flooding of the water meter pit (lowest point). The pictures are as it was found.

The following trends show how the code continued to adapt to the dirty conditions to read an accurate signal (with a lower SNR). At approx. 9:50 AM the photos were taken, the meter wiped clean, and the IR sensor reinstalled. Note, it took a few hours to re-adjust to the new amplitude and resume correct operation. An improvement may be a manual reset for when a known intervention event has occurred.

April 4th 2024 - Adaptation to a constantly changing environment

A periodic check after a few days of rain found significant amounts of deposits and moisture internal to the water meter. The adaptive nature of the algorithm displays the how it can track the minimum input voltage to ensure correct, continuous operation & digitisation of the analog water meter.

The Slow Water Leak

The normal sharp edges between high and low values began to soften over the last few days. A slow sinusoidal behavior was occurring with a period of roughly 2 hours or 0.5L/hour leak.

After searching around the house for a tap that was not quite off or a leaky washing machine, dishwasher, water filter etc. The leak was found.

A sun aged hose was found to not be turned off at the tap and was the cause of the slow leak. The sharp edges have now been restored.

Google Sites

Report abuse