Monday, November 27, 2017

Anomaly Detection - 1

On a broad level, anomalies can be classified as point anomalies, contextual anomalies and collective anomalies.

Here's some of techniques that can be used to detect anomalies

1.  Statistical Methods

Flag data points that deviate from common statistical properties of a distribution - mean, mode, median, quantile.

We can use rolling average or moving average to smoothen short term fluctuations. A low pass filter can be used by n-period simple moving average.

2.  Density Based Anomaly Detection

This technique assumes that normal data points will occur around a dense neighborhood and the abnormalities will be far away (based on k-nearest neighbors algorithm or local outlier factor).

3. Clustering (e.g. K-means clustering algorithm)

4. Support Vector Machine - Variants of SVM like One-Class SVM can be used for anomaly detection.



Further Reading: 

1.https://en.wikipedia.org/wiki/Anomaly_detection
2.http://www.nehalemlabs.net/prototype/blog/2013/04/05/an-introduction-to-smoothing-time-series-in-python-part-i-filtering-theory/
3.https://bugra.github.io/work/notes/2014-03-31/outlier-detection-in-time-series-signals-fft-median-filtering/

No comments:

Post a Comment