Academic Project · Big Data / Streaming
Real-Time Anomaly Detection
A Kafka and PyFlink prototype that flags unusually high click counts in simulated e-commerce traffic. It counts clicks per user in fixed one-minute event-time windows and prints an alert when a closed window contains more than 50 clicks. This is a volume-based heuristic, not a validated classifier of bots or attacks.
2026 academic project with Aka Elishama KADJO. Our joint report documents the pipeline, a separate Count-Min Sketch implementation, and local execution checks.
The Pipeline
1. Ingest
Kafka
Every click event (user_id, event, product_id, timestamp) goes into a Kafka topic. A Python producer simulates normal traffic. A separate anomaly simulator injects 10 000 rapid clicks for testing.
2. Process
Flink
PyFlink consumes the stream, assigns event-time watermarks, and uses exact SQL counts in 1-minute tumbling windows per user. Count-Min Sketch was implemented and tested separately; integrating it into Flink is proposed as future work.
3. Alert
Detection
When a window closes and a user has more than 50 clicks, Flink emits an anomaly alert with the user_id, window range, and click count.
Why Kafka + Flink
Kafka as the buffer
Kafka separates ingestion from processing and retains events on disk for replay, subject to its retention settings. Recovery also depends on the consumer offsets and Flink state that have been persisted.
Flink for stateful streaming
Flink provides keyed state, event-time windows, and watermarks for handling out-of-order data. Alerts are produced after windows close; the report does not measure end-to-end alert latency.
What happens when things break
If Flink crashes
Kafka can retain the backlog while Flink is unavailable. Restoring window state and offsets requires durable checkpoints. The report lists persistent checkpoint storage as production work still to add, so crash recovery is not demonstrated here.
If Kafka goes down
Producers can't send events. They'd need a local cache with retry logic, or accept temporary data loss. This is the weaker link in the chain.
Recorded validation and limits
Local execution
The report shows six Count-Min Sketch unit tests passing, a normal-traffic producer configured for approximately 10 events per second, and an anomaly simulator sending 10,000 clicks for user 1. Execution screenshots show Flink printing flagged windows and a running job.
What this establishes
These checks demonstrate the local prototype and its threshold rule. The producer rate is a test setting, not a throughput benchmark. The report provides no measured latency, precision, recall, or production-load evaluation.
The configured watermark allows 5 seconds of out-of-order arrival; events arriving after their window has closed can be dropped. A burst split across two fixed windows can also evade the threshold. Sliding windows, an alert sink, and persistent checkpoint storage are proposed improvements.
Source: project report, sections 4–5 and execution screenshotsProject Report
Full report covering the architecture, event time vs processing time, Count-Min Sketch implementation, fault tolerance, and execution screenshots.