Avoiding the Database Stampede

This is not an article about "Cache is King" and proposal to use nosql tactics like memcached. It is more about illustration of a database stampede (db stampede) that occurs while using the traditional memcached pattern in a high-concurrency situations.

Below is the illustration of traditional memcached pattern:

A team at Go Daddy has discovered a problem in the Load Test Lab in September 2010. They ran a load profile of some 500 users doing various tasks against the Community application (putting some good old fashioned load onto the system). It was holding up fine, except that every 5 minutes, there was a blip in response time.

They received 500 threads of apache/php all reaching the same conclusion, at the same time - memcached data is expired, refer database. The database went from idle to pegged almost instantly.

They concluded this solution:

The Web servers don't go back to the database. It won't even know that it's an option anymore. Instead, using an independent thread, kicked off from "cron" every minute, to run queries on the database and populate memcached. Set the TTL for several hours, so in the rare event the db goes down, the web servers still have data to use.

This might look like a bad idea, because you shouldn't rely on memcached having the value stored. By default, memcached will roll key pairs out of the cache to make room for new key pairs. This issue can be overcome by managing the memcached instance(s) and making sure you don't reach your pre-configured memcached pool size.

As soon as this pattern change were implemented, the response time blips disappeared. This pattern is proven and running incident free for over a year, on sites that takes a considerable amount of traffic.

Source: godaddy.com