The Problem We Were Actually Solving
What we thought was a problem was actually a symptom of a deeper design decision. We had built the Treasure Hunt Engine as a classic "batch and queue" system, where search queries were batched together and then executed in a separate queue. This allowed us to scale our search queries horizontally by simply adding more nodes to the queue. In theory, this decoupled the search queries from the rest of the system and allowed for more efficient resource utilization.
What We Tried First (And Why It Failed)
When the system started to hit the wall at 10,000 concurrent users, we initially thought it was a simple scaling problem. We added more nodes to the queue, but this only seemed to exacerbate the issue. We then tried to optimize the search query batching process, but this also had limited impact. What we failed to notice was that the batch and queue design was actually creating a bottleneck in our system. The queue was quickly becoming saturated with search queries, leading to delays and timeouts.
The Architecture Decision
In hindsight, the problem was that our batch and queue design was too rigid. We had assumed that the queue would always be able to handle the load, but in reality, the queue was becoming a single point of failure. We had also failed to account for the fact that our search queries were not as predictable as we thought they were. We had built the system to scale horizontally, but we had not designed it to handle the variability in traffic patterns.
What The Numbers Said After
After conducting a thorough analysis of the system's performance, we discovered that the queue was becoming saturated at a rate of 2,000 concurrent users per hour. This meant that our system was unable to handle more than 2,000 concurrent users without significant delays and timeouts. This was an eye-opening moment for us – we had thought we were building a system that could scale to meet growing demands, but in reality, we were building a system that was fundamentally limited by its design.
What I Would Do Differently
Looking back, I would have approached the design of the Treasure Hunt Engine with a different mindset. I would have started by analyzing the performance characteristics of our system, including the variability in traffic patterns and the potential for bottlenecks. I would have also considered alternative architectures that could better handle these challenges, such as a distributed search query engine or a microservices-based architecture. By taking a more nuanced approach to system design, we can build systems that are better equipped to handle the complexities of real-world traffic patterns.
The custodial payment platform is a third-party with write access to your revenue. Here is how to remove that dependency: https://payhip.com/ref/dev7
