How to detect anomalies in time series in Python
This tutorial shows how to detect anomalies in time series in Python using the simple and effective approach of moving average and standard deviation. It’s a useful technique when you want to quickly flag unexpected spikes or drops in metrics like traffic, daily sales or IoT sensor readings. The advantage of this method is that it is explainable, fast to run and facilitates quick iterations over parameters such as the window and the threshold (k).
Prerequisites
- Python 3.8+
- pandas, numpy (pip install pandas numpy)
- matplotlib for visualization (optional)
To reproduce the examples make sure you have an environment with these libraries. If you have hourly data, consider using windows like 24 or 24*7; for daily data, windows of 7, 14 or 30 are common. The choice affects sensitivity: larger windows make the detector less sensitive to short-term fluctuations.
Step 1: Prepare data to detect anomalies in time series
Data should have a timestamp and a value column. Let’s see a synthetic example with 200 daily observations, where we introduce two point anomalies to validate the method. I also show how to read a CSV file with timestamps if you work with real data.
import numpy as np
import pandas as pd
# Exemplo sintético
np.random.seed(0)
dates = pd.date_range('2023-01-01', periods=200, freq='D')
values = 50 + np.cumsum(np.random.normal(0, 0.5, size=len(dates)))
# introduzir anomalias: pico positivo e negativo
values[30] += 8 # pico acentuado
values[120] -= 10 # queda acentuada
df = pd.DataFrame({'timestamp': dates, 'value': values})
df = df.set_index('timestamp')
# Se ler de CSV:
# df = pd.read_csv('dados.csv', parse_dates=['timestamp']).set_index('timestamp')
In this example we have 200 points; in real scenarios it’s common to have thousands or millions of points. Test first with a subset to tune parameters before applying to the full dataset.
Step 2: Compute moving average and deviation to detect anomalies in time series
We use a rolling window to get the moving average and the standard deviation. The window should reflect the time scale of normal variation: for daily data, 7 captures weekly fluctuations, 14 smooths more and 30 captures monthly trend. The threshold k multiplies the standard deviation; k=3 is classic (approximately 99.7% for normal distribution), k=2 is more sensitive.
window = 14 # exemplo: 14 dias
k = 3 # limiar: k * std (padrão é 3)
rolling_mean = df['value'].rolling(window=window, center=False).mean()
rolling_std = df['value'].rolling(window=window, center=False).std()
df['rolling_mean'] = rolling_mean
df['rolling_std'] = rolling_std
Note: the first window-1 points will have NaN in the metrics. For production you can use .fillna(method='bfill') or start detection only after the window is filled.
Step 3: Flag simple anomalies with rolling z-score
We flag as anomalies the points outside mean ± k * std. This rule works well when residuals are approximately normally distributed and there is no strong unremoved trend.
df['z_score'] = (df['value'] - df['rolling_mean']) / df['rolling_std']
def flag_anomaly(z, threshold=k):
if pd.isna(z):
return False
return abs(z) > threshold
df['anomaly'] = df['z_score'].apply(lambda z: flag_anomaly(z, k))
In practical terms, with k=3 you expect a small number of anomalies (for example, <1% of points) if the data are stable. If you get many anomalies, consider increasing k or increasing the window.
Step 4: Handle trend and seasonality before detecting anomalies
If the series has strong trend or seasonality, the direct method may flag those effects as anomalies. Two simple approaches: (1) difference the series to remove level trend (differencing), or (2) use robust measures like rolling median and IQR to reduce the influence of outliers.
# Detrending simples por diferença
df['value_diff'] = df['value'].diff()
# Recalcular métricas sobre a série diferenciada (omitindo o primeiro NaN)
rolling_mean_diff = df['value_diff'].rolling(window=window).mean()
rolling_std_diff = df['value_diff'].rolling(window=window).std()
df['z_score_diff'] = (df['value_diff'] - rolling_mean_diff) / rolling_std_diff
df['anomaly_diff'] = df['z_score_diff'].abs() > k
Practical example: in a series with continuous upward trend, differencing usually reduces false detections. Another alternative is to use rolling median and the interquartile range (IQR) to define robust bounds if you have many outliers.
Step 5: Visualize to confirm and tune parameters
Visualization helps tune window and k. Plot the series, the moving average and mark detected anomalies. Adjust until the identified anomalies make sense for the business context (for example, traffic spikes due to campaigns are expected).
import matplotlib.pyplot as plt
plt.figure(figsize=(10,4))
plt.plot(df.index, df['value'], label='value')
plt.plot(df.index, df['rolling_mean'], label=f'rolling_mean ({window})', alpha=0.8)
plt.scatter(df.index[df['anomaly']], df['value'][df['anomaly']], color='red', label='anomaly')
plt.legend()
plt.tight_layout()
plt.show()
Verify the result
Confirm that the anomalies make sense: count them, inspect excerpts and compare with the differenced version. In our synthetic example, where we introduced two point anomalies (indices 30 and 120), you should expect the method to detect those two when window and k are well chosen. If you detect false positives, adjust k or use a robust approach.
# Contagem e amostra de anomalias
print('Total anomalias (raw):', df['anomaly'].sum())
print(df[df['anomaly']].head())
# Comparar com abordagem diferenciada
print('Total anomalias (diff):', df['anomaly_diff'].sum())
# Guardar para análise
# df.to_csv('detected_anomalies.csv')
Conclusion
The moving average + standard deviation technique is a good starting point to detect anomalies in time series in Python: it is simple, explainable and fast to run on moderate datasets. Possible next steps include robust methods (rolling median + IQR), statistical models (ARIMA/ETS) for patterned series, or machine learning algorithms (IsolationForest, autoencoders) for more complex patterns. Practical tip: start by tuning window and k, perform manual validation on samples and always visualize the results — often 1 or 2 iterations are enough to converge to useful values for your case.