How I DDoS-ed us with best intentions?

Oh, look - lots of users are onboarding. Awesome! A couple of hours later ... App is slow, ECS tasks for the API are being killed, and database usage is off the charts. What's going on?

This is a true story of how I DDoS-ed us with the best intentions.

Health checks

If you've ever used AWS ECS with a load balancer, you know that the API must implement a health check endpoint. The load balancer periodically calls the endpoint. Once it gets enough success response back, it considers the task healthy, and it starts routing the requests to it.

What do you do when you want to expose the basic status page for your app? You add a periodic call to the same health check endpoint. It's all good until ... You write a migration that acquires access exclusive lock on your users table. And you end up with a locked database. No one can really use the app, but the status says it's up.

Improved health checks

So you decide to improve the health check endpoint. You want the call to fail if you can't access the users table. You go and pick the first method, which is listing all users created in the last 7 days. Usually that's around 50 or so. The next time you try to apply an overly eager migration, you see a failed health check, you stop it, you fix it, re-apply it, and everyone's happy.

Snowballing

Everything is great until, all of a sudden, 1000 users sign up in two days. All of a sudden, everything is slow. You see API tasks being auto-scaled up to the max, and you see the database struggling. You start debugging: - The number of requests is roughly the same - If you re-deploy, things are better for some time, but then they derail again - Logs are not showing anything special

And then you remember to check the health check endpoint. Yep, 1000 user rows are fetched from the database every couple of seconds. From each task that's running. The more tasks are running, the worse it gets.

Learnings

First of all, you should never run unbounded queries inside the app. You should always paginate. Even if you actually need all rows. It's fine to have a large page size, but there should always be a page size. You should not simply try to fetch 100k records at once. If you allow that, a single query can choke the whole system.

Second, you should separate load balancer health check from status health check. The load balancer only needs to know that the app is up and running. There's no need for it to know the database state. These requests are too frequent, and they hit every running task. Status check should touch the database to ensure the app is actually responding. Instead of listing all users created in the past 7 days, one can list the last 10 users. This way you check that you can actually get data from the database while keeping the load small and predictable.

  @app.get("/health")  # for the load balancer
  def health():                                                                                                                                                                                                                          
      return {"ok": True}                                                                                                                                                                                                              

  @app.get("/health-status")  # for the public status page                                                                                                                                                                                      
  def status(db: Session = Depends(get_db)):       
      db.execute(text("SELECT email FROM users LIMIT 10"))                                                                                                                                                                                    
      return {"ok": True}                                                                                                                                                                                                              

Conclusion

You can bring your system down even if your intentions are the best. I keep relearning that over and over. You should always try to draw reasonable boundaries between different use cases inside your system. Many times reuse seems like an obvious option, but the "duplication" is only accidental. Also, it's important to have good observability of the system. This way you can quickly find the root cause when something goes wrong.

Happy scaling!