How to calculate error rate per service in Real-Time Analytics: step by step
I present how to calculate error rate per service in Real-Time Analytics, a crucial metric to monitor stability and trigger alerts. Knowing how to calculate the error rate in real time allows you to identify regressions and act before users are affected. In this guide I explain the rationale for each option (window, slide, smoothing) and give concrete examples that work in production.
Prerequisites
- Streaming event source (Kafka, Event Hubs or similar) with JSON events containing service_id, status and timestamp.
- Streaming processor (e.g.: Azure Stream Analytics, Apache Flink, Spark Structured Streaming or ksqlDB).
- Destination for metrics (e.g.: Time Series db, Prometheus, InfluxDB, or a Kafka topic for dashboards).
- Basic knowledge of SQL/KQL or the API of the chosen streaming engine.
Practical note: in environments with 5k–50k events/second it is advisable to use partitioning by service_id and ensure checkpointing. Setting a watermark (e.g.: 2 minutes) helps handle late events without inflating windows.
Step 1: Define the event and metric (why)
Before coding, clarify what an "error" is in your system. Typically, an error is considered an event with status = "error" or HTTP code >= 500. The error rate is: errors / total requests in a time window. We will calculate this rate per service in 1-minute windows with updates every 15s (sliding window) to get resolution and smoothness.
Why 1 minute and 15s slide? A 60s window reduces noise from short spikes; a 15s slide gives 4 updates per minute, enough for quick alerts. For APIs with very low latencies you can use 30s / 10s. In terms of practical thresholds, alerts are defined: warning if error_rate > 0.005 (0.5%) for 3 consecutive windows; critical if > 0.01 (1%). These numbers are guidelines and depend on the SLA.
Step 2: Stream ingestion and JSON parsing
Receive events from the topic and convert the JSON payload into columns: service_id, status, timestamp. Example with Spark Structured Streaming (PySpark):
# Exemplo em PySpark
from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col, to_timestamp
from pyspark.sql.types import StructType, StringType
schema = StructType() \
.add('service_id', StringType()) \
.add('status', StringType()) \
.add('timestamp', StringType())
spark = SparkSession.builder.appName('error-rate').getOrCreate()
raw = spark.readStream.format('kafka') \
.option('kafka.bootstrap.servers','kafka:9092') \
.option('subscribe','events') \
.load()
json_df = raw.select(from_json(col('value').cast('string'), schema).alias('j')) \
.select(col('j.service_id'), col('j.status'), to_timestamp(col('j.timestamp')).alias('ts'))
Tip: if you expect late events, use withWatermark('ts','2 minutes') before grouping.
Step 3: Classify events as error or success
Create a boolean column is_error that makes counting easier. This reduces common mistakes like forgetting to parse status or ignoring case. It also handles cases where the status field is empty.
from pyspark.sql.functions import when
events = json_df.withColumn('is_error', when(col('status').rlike('(?i)error|5\\d\\d'), 1).otherwise(0))
Practical example: if a service has 12,000 requests in a window and 60 errors, the rate is 60/12000 = 0.005 = 0.5%.
Step 4: Aggregate by sliding window and by service
Use a tumbling or sliding window. For a 1-minute sliding window with 15s slide in Spark:
from pyspark.sql.functions import window, sum as _sum, count as _count, when, lit
agg = events.withWatermark('ts','2 minutes').groupBy(
col('service_id'),
window(col('ts'), '60 seconds', '15 seconds')
).agg(
_sum(col('is_error')).alias('errors'),
_count(col('service_id')).alias('total')
)
result = agg.withColumn('error_rate', when(col('total') == 0, lit(0.0)).otherwise(col('errors') / col('total')))
Important: avoid division by zero (total = 0) and consider smoothing: apply EWMA to error_rate to reduce false positives (e.g.: alpha = 0.3). In terms of latency, with 15s micro-batches expect ~100–500ms of additional processing per batch depending on the cluster.
Step 5: Write metrics to destination
Export the results to a metrics system or topic for visualization. Avoid extra latency with very small writes per micro-batch. Example writing to another Kafka topic and setting a 15s trigger:
result.selectExpr('service_id', 'window.start as window_start', 'window.end as window_end', 'errors', 'total', 'error_rate') \
.selectExpr("to_json(struct(*)) AS value") \
.writeStream.format('kafka') \
.option('kafka.bootstrap.servers', 'kafka:9092') \
.option('topic', 'metrics.error_rate') \
.option('checkpointLocation', '/tmp/checkpoint_error_rate') \
.trigger(processingTime='15 seconds') \
.start()
Consider retention and compaction on the metrics topic; for dashboards in Grafana/Prometheus, write to the Time Series db with compatible resolution (e.g.: 15s). Don't forget to monitor the checkpoint directory and disk usage.
Verify the result
Confirm you have messages in the destination topic or in the metrics system with fields: service_id, window_start, window_end, errors, total, error_rate. Check edge cases: total = 0 (avoid division by zero), windows with low traffic and consistency with source logs. A simple Kafka consumer or a query in the Time Series db UI should show rates close to expected. Test with synthetic traffic: send 10k events with 100 errors (1%) and validate the metric.
Conclusion
Calculating error rate per service in Real-Time Analytics allows you to detect regressions quickly and feed alerts and dashboards. Next steps: normalize by request type, apply smoothing (e.g.: EWMA with alpha between 0.2 and 0.5) or create dynamic thresholds based on historical baseline. Final tip: always handle the total = 0 case, validate timestamps timezone to avoid misaligned windows, and size resources according to the expected throughput.