Skip to main content
Back to the lab

Academic Project · Big Data / Streaming

Real-Time Anomaly Detection

A Kafka and PyFlink prototype that flags unusually high click counts in simulated e-commerce traffic. It counts clicks per user in fixed one-minute event-time windows and prints an alert when a closed window contains more than 50 clicks. This is a volume-based heuristic, not a validated classifier of bots or attacks.

Apache KafkaPyFlinkDockerPythonpytestCount-Min Sketch

2026 academic project with Aka Elishama KADJO. Our joint report documents the pipeline, a separate Count-Min Sketch implementation, and local execution checks.

The Pipeline

1. Ingest

Kafka

Every click event (user_id, event, product_id, timestamp) goes into a Kafka topic. A Python producer simulates normal traffic. A separate anomaly simulator injects 10 000 rapid clicks for testing.

2. Process

Flink

PyFlink consumes the stream, assigns event-time watermarks, and uses exact SQL counts in 1-minute tumbling windows per user. Count-Min Sketch was implemented and tested separately; integrating it into Flink is proposed as future work.

3. Alert

Detection

When a window closes and a user has more than 50 clicks, Flink emits an anomaly alert with the user_id, window range, and click count.

Why Kafka + Flink

Kafka as the buffer

Kafka separates ingestion from processing and retains events on disk for replay, subject to its retention settings. Recovery also depends on the consumer offsets and Flink state that have been persisted.

Flink for stateful streaming

Flink provides keyed state, event-time windows, and watermarks for handling out-of-order data. Alerts are produced after windows close; the report does not measure end-to-end alert latency.

What happens when things break

If Flink crashes

Kafka can retain the backlog while Flink is unavailable. Restoring window state and offsets requires durable checkpoints. The report lists persistent checkpoint storage as production work still to add, so crash recovery is not demonstrated here.

If Kafka goes down

Producers can't send events. They'd need a local cache with retry logic, or accept temporary data loss. This is the weaker link in the chain.

Recorded validation and limits

Local execution

The report shows six Count-Min Sketch unit tests passing, a normal-traffic producer configured for approximately 10 events per second, and an anomaly simulator sending 10,000 clicks for user 1. Execution screenshots show Flink printing flagged windows and a running job.

What this establishes

These checks demonstrate the local prototype and its threshold rule. The producer rate is a test setting, not a throughput benchmark. The report provides no measured latency, precision, recall, or production-load evaluation.

The configured watermark allows 5 seconds of out-of-order arrival; events arriving after their window has closed can be dropped. A burst split across two fixed windows can also evade the threshold. Sliding windows, an alert sink, and persistent checkpoint storage are proposed improvements.

Source: project report, sections 4–5 and execution screenshots

Project Report

Full report covering the architecture, event time vs processing time, Count-Min Sketch implementation, fault tolerance, and execution screenshots.