Your payment service just charged a customer.
It writes to the DB. Now it needs to tell the notification service: "Send the confirmation email."
The HTTP call times out. Did it arrive? You don't know. You retry. Customer gets two emails.
You've just hit the Two Generals Problem. It's not a bug. It's a proof.
No protocol over an unreliable channel can guarantee both sides agree on the final message. Not HTTP. Not TCP. Not your retry loop. The uncertainty is mathematically irreducible.
Here's the setup:
PaymentService (Node.js, PostgreSQL) → NotificationService (Go)
~40ms p99 latency, occasional 504s under load.
You need to send exactly one confirmation email per payment — no double-sends, no missed sends.
What do you build?
A) Retry with exponential backoff until NotificationService returns 200. If you keep retrying until you get an ACK, you know it arrived.
B) Wrap both in a distributed transaction (2PC) — PaymentService and NotificationService commit together or neither does.
C) Outbox pattern — PaymentService writes the notification event to an outbox table in the same DB transaction as the payment. A relay process delivers it separately.
D) Push to SQS with at-least-once delivery. NotificationService deduplicates on a stable idempotency key. Accept you might send twice, but never miss.
One of these is a trap that senior engineers fall into every time. One of them doesn't solve the fundamental impossibility at all. And one is what you actually ship.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
