Your data team just got a new SLA: surface fraud signals within 500ms of a transaction.
Right now you're running nightly Spark batch jobs. The business wants "real-time." Your team knows Spark. Someone already opened a PR adding Spark Structured Streaming.
The transaction volume: 8,000 events/sec peak. You're on AWS. The fraud model runs in Python. The output feeds a DynamoDB table the API reads from.
You need to redesign the pipeline. What do you pick?
A) Kafka Streams — event-by-event processing, stateful operators, sub-10ms latency. Lives inside your app JVM.
B) Apache Flink — true streaming engine, exactly-once semantics, built for high-throughput stateful processing.
C) Spark Structured Streaming — micro-batch under the hood, 100ms–5s windows, same API your team already knows.
D) Keep the batch job, drop the window to 1 minute — "near real-time" at zero migration cost.
Three of these can hit sub-500ms. One of them cannot — no matter how you tune it.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
Drop your answer 👇
