The Problem We Were Actually Solving
We were trying to solve a problem we thought was caused by our users. We assumed that the sudden spike in requests was due to some viral social media post or a popular YouTube video. But, in reality, our application's growth was the real culprit. As more users signed up, our server's complexity and requests skyrocketed, creating a treasure hunt engine in its own right. Our operators were left to fend off errors and outages at every turn, and the production logs were a mess.
What We Tried First (And Why It Failed)
When our operators first noticed the issue, we turned to our monitoring tools and started cranking up the logging levels. We thought this would give us a clear picture of where things were going wrong. But, in our haste, we failed to account for the sheer volume of data we were about to generate. Our monitoring tool, Splunk, quickly became overwhelmed, and we found ourselves drowning in a sea of irrelevant information. It was like trying to find a needle in a haystack, but the needle was moving really fast.
The Architecture Decision
Looking back, I realize that our architecture team had made a crucial decision a few months prior. We had chosen to use an event-driven architecture, which was supposed to scale more elegantly than our previous request-response setup. But, as our server grew, our event-driven architecture turned into a bottleneck. Our operators were overwhelmed by the sheer volume of events, and our application's performance suffered as a result. I often wonder if we would have fared better with a more robust request-response model.
What The Numbers Said After
After our server grew to the point where it was consistently requesting more resources, we implemented a custom metric called "treasure hunt ratio" to gauge the complexity of our system. The treasure hunt ratio measures the number of times an operator has to dig out errors from our logs versus the number of actual useful requests. Once this ratio surpassed a certain threshold, we knew we had a serious problem on our hands. Our logging tool, ELK, helped us identify this ratio and eventually, we were able to automate the process of detecting and mitigating it.
What I Would Do Differently
In retrospect, I would have insisted on a more robust logging strategy from the get-go. We would have implemented a data pipeline that would have allowed us to filter and aggregate our logs more effectively, even in the face of extreme growth. Additionally, I would have pushed harder for a more incremental approach to our architecture, allowing us to test and refine our event-driven model in smaller increments. And, of course, we would have communicated better with our operators, making sure they were aware of the changes we were making and how they would impact their workflow.
Chargebacks are a fraud vector. Custodial holds are a business continuity risk. This infrastructure eliminates both: https://payhip.com/ref/dev7
