Memcache User Guide

Memcache is a technology that helps web apps and mobile app backends in two main ways: performance and scalability.

Table of Contents

What is memcache?

Memcache is an in-memory, distributed cache. The primary API for interacting with it are SET(key, value) and GET(key) operations.  Memcache is essentially a hashmap (or dictionary) that is spread across multiple servers, where operations are still performed in constant time.

The most common usage of memcache is to cache expensive database queries and HTML renders such that these expensive operations don’t need to happen over and over again.

Performance

Memcache reduces page load time, giving your user a faster, better experience. Remember, speed is a feature.  Without memcache, each page load will need to perform database queries and  HTML rendering over and over again.  Often these operations take several hundred milliseconds, but sometimes they can take seconds depending on complexity and server load.  Memcache queries return in milliseconds, which is usually 100x faster than a database query or a complex render (again, depending on load and complexity). This allows any database queries or renders that are common to some of your users to be shared. Even operations unique to a user can be useful to cache if the same result will be displayed to them on every page.

If you’re starting with an empty cache, the first page load will execute in the same time as it would without memcache. However, after expensive operations have been cached during the first page load, that same page load will be drastically faster each subsequent time until your key is either expired or evicted from the cache.  More on eviction and expiration later.

Scalability

An app’s scalability is a function of throughput — the more throughput your app has, the better it will scale.  Memcache gives your app more throughput because of two related reasons. The first is simple, by avoiding slower database queries, page renders or web API calls your app can handle a request faster and so execute more per second. The second component of scalability is by hitting the cache you avoid touch the database and disk. This is generally highly relied upon to scale out the database. Without memcache, many apps would be hitting their database too often, making it a bottleneck for their app. While they could try to directly scale the database with say master / slave replication, these are complicated technologies to setup and use and aren’t as effective as simply putting memcache in between your app and the database for a lot of use cases.

How is memcache so fast?

Memcache is so fast for two reasons: the cache is entirely in-memory; and operations are performed in constant time.  Memcache avoids writing any data to disk, which results in faster operations and no fault tolerance.  If the server running memcache is restarted, the keys on that server will be lost, which is fine for the use case of memcache, as a temporal caching layer and not a definitive data store. The use of constant time operations is simply a deployment of all those good old tricks and tips for a computer science algorithms & data structures class. O(1) is a nice property to have.

Eviction, Expiration, and Memory Limits

Besides getting and setting keys, eviction and expiration are major components of memcache, too.  When a key is SET in the cache, an expiration is set along with the key.  Most memcache clients have a default expiration which can be optionally overwritten on a per-key basis.

Memcache doesn’t evict keys when their expiration time expires, as doing so isn’t possible while still guaranteeing O(1) operation times. Instead expiration is more of a way to say how long until a key should be considered stale. When a GET is performed, Memcache checks if the key’s expiration time is still valid before returning it.

A key is evicted from the cache when the total cache size has reached the limit.  Most memcache implementations offer a least recently used (LRU) eviction policy, which evicts the key that was least recently used when a new key is added to the cache and the cache has reached its limit.

In general, the higher your cache limit, the fewer evictions you’ll have, and the higher your hit-rate will be, ultimately resulting in better performance and scalability.

How can I check query and render times?

Most web frameworks provide a mechanism to log page load operations.  For example, in Rails, the development log looks something like this:

...
Application Load (16.3ms)  SELECT `customers`.* FROM `applications`
Rendered customers/_list.html.erb (185.8ms)
...

In the log above we see that the SELECT statement ran in 16.3ms, and the render of customers/_list.html.erb ran in 185.8ms.

In addition to logs, tools like New Relic analyze performance in real time and give you reports about page load times.

Getting Started

Now we’ll get a little deeper into the usage of memcache by showing some example code, along with detailed explanations.  We’ll give you an idea about what it takes to write memcache code to help your app scale. We’ll use pseudocode to demonstrate.

Step 1: Connect to Memcache

Connecting to memcache is simple — all that’s involved is a set of servers, a username, and a password (although if you’re running your own memcache you won’t need a username and password). The details of connecting to memcache are language- and framework-specific.  Each language has its own set of memcache clients, and each client has a different configuration interface. We won’t cover each language, framework, and client in this post. However, our Heroku docs contain detailed instructions for connecting to memcache in Ruby, Rails, Django, PHP, and Java.

The reason that you use a list of servers when connecting and not a single server when connecting s that standard memcache puts some of the intelligence of the design into the client. So the distributed cache component of memcache is actually handled by clients, not servers. (As a quick side note, Memcachier works differently than this, giving you a single big-ass cache in the sky so you only need one server to connect to and never need to change your configuration to add more cache space). We previously covered some of the computer science behind all of this in a blog post on consistent hashing.

Step 2: Test the Connection

Before you start writing application code, make sure your connection to memcache is working.  You can do this in a console (for example, irb in Ruby) or in a test application.  Start by establishing a connection and trying a get and set operation.  Something like this (again, we’ll show you pseudocode here):

cache = Cache.connect(SERVERS, USERNAME, PASSWORD)
cache.set("foo", "bar")
print cache.get("foo")
 > "bar"

In the above example, we’re establishing a connection, setting the “foo” key to the value “bar”, and printing the results of a get with the “foo” key.

Step 3: Write Application Caching Code

Once your client is configured properly and you’ve verified that you can get and set, now you should start writing some caching code.  Start with a small, isolated part of your code to familiarize yourself with memcache.  For example, if you’re building a friend-based app, perhaps a good starting point is with a list of friends and a user profile.  Below we’ll demonstrate each of these scenarios: one with a database query, the other with a rendered view.

Database Queries

The below code shows how you can use memcache to cache the results of a database query:

...
user = current_logged_in_user
friend_list = cache.get("friends_for_user_" + user.id)
if (friend_list == null) {
  friend_list = Friend.for_user(user.id) # 'SELECT * FROM friends WHERE user_id = <user.id>'
  cache.set("friends_for_user_" + user.id, friend_list)
}
...

The code starts by checking the cache to see if the user’s friends are already in the cache.  We’ve chosen a descriptive key name — “friends_for_user_<user_id>” — to store the friend list.  You can name your keys whatever you want, but descriptive names are ideal.

Next, if the friend list is cached, nothing else needs to happen.  However, if the friend list is not cached, we’ll need to fetch the list from the database and store the results in the cache.  The friend list is stored in the cache so the next time this code runs, the database query won’t need to be executed.

Rendered Views

In addition to caching database queries, you probably want to start caching rendered views, too.  The below code shows how you can use memcache to cache the results of a rendered view:

...
user = current_logged_in_user
profile_view = cache.get("profile_for_user_" + user.id)
if (profile_view == null) {
  profile_view = render("profile", user)
  cache.set("profile_for_user_" + user.id, profile_view)
}
http_respond(profile_view)
...

The code starts by checking the cache to see if the user’s profile view is already in the cache.  Again, we choose a descriptive key name.

Next, if the profile view is cached, nothing else needs to happen except that the view is sent with the HTTP response.  However, if the profile view is not cached, we’ll need to render the view and store the rendered view in the cache.  The rendered view is stored in the cache so the next time this code runs, the view won’t need to be rendered again.

Note that frameworks such as Django and Rails have their own helper functions for caching rendered views.  You should read your framework’s caching documentation to learn how to utilize these helper methods.

Step 4: Expirations and Deletions

The code we’ve seen so far for caching database queries and rendered views is all well and good until data changes.  In the above examples, our cache will be out of date if a user adds a new friend or changes their profile information.  There are two strategies for avoiding an out-of-date cache: expirations and deletions.  Each of these will be covered in detail below:

Expirations

Keys, at the time they’re set, are given an expiration time — or a time when the key will be automatically removed from the cache.  Most clients allow for a global expiration to be set, along with a mechanism to set the expiration on a key-by-key basis.  Again, when a key expires it is automatically removed from the cache.

Expiration times can be strategically set according to the type of data being stored.  For example, if you cache algebra equations and their results, you can set an infinite expiration time (0) because the result will never change (2+2 always equals 4).  As for a user’s profile or list of friends, you could set an expiration of a few minutes — the logic being that these keys won’t change more frequently than a few minutes.  However, relying on an expiration time in this case will result in a bad user experience.  You don’t want a user adding a friend, then having to wait a few minutes for that friend to show up in their friend list.  This is where deletions come in.

Deletions

Deletions, at least in the web and mobile space, are the most common way to keep your cache up-to-date.  A deletion strategy is one where a particular key is deleted whenever its value is known to be out of date.  In our friend and profile example, we would delete the profile_for_user_X key when a user updated their profile, and the friends_for_user_X key when the user added or removed a friend.

Deletion is the best way to keep a cache up to date.  However, the code can get ugly very quickly.  Without careful consideration it’s easy to complicate most of your logic with deletion calls whenever you know a key needs to be deleted.  For example, we could put the call to delete the friend list in the friendship controller:

...
user = current_logged_in_user
Friend.create(user.id, new_friend_id)
cache.delete("friends_for_user_" + user.id)
cache.delete("friends_for_user_" + new_friend_id)
...

The above example will properly delete out-of-date values from the cache. But now our controller code is polluted with cache calls. A better strategy for deleting keys is to put the deletion code into the model itself so that it is triggered when the model changes. This allows you to centralize the logic for cache invalidation, avoiding duplication of code and complication of controllers.

Rails, for example, has excellent support for this style with its ActiveRecord::Callbacks methods.  Other frameworks and languages have great support for such model triggers, too.  To go along with our pseudocode example, an improvement on the above deletion code would look something like the below — instead of logic going in the controller, it goes in the model.

class Friend {
  after_create: delete_cache()
  after_delete: delete_cache()

  ...

  function delete_cache() {
    cache.delete("friends_for_user_" + this.user_one_id)
    cache.delete("friends_for_user_" + this.user_two_id)
  }
}

Notice how the above cache deletion logic is conveniently tucked away in our model instead of our controller. This greatly simplifies our controller code and makes our app far easier to maintain.

There’s still one more improvement we can make to this code.  Instead of just deleting each key when we know its value is out of date, we could set the value of each key to the new, up-to-date value.  Something like this:

class Friend {
  after_create: refresh_cache()
  after_delete: refresh_cache()

  ...

  function refresh_cache() {
    cache.set("friends_for_user_" + this.user_one_id, Friend.for_user(this.user_one_id))
    cache.set("friends_for_user_" + this.user_two_id, Friend.for_user(this.user_two_id))
  }
}

With the above code, our cache will be up-to-date, our controller code won’t be ugly, and our cache will be warm immediately, at least for the two new friends.

Certain frameworks such as Rails have Sweepers, which are classes that observe changes to a model and expire the cache as needed.  Sweepers are yet another improved way to write maintainable code, but not all frameworks have them.

Step 5: Be Strategic About What to Cache

As you can tell from the above examples, writing cache code can be messy. Your cache is also generally far smaller than your database.  For this reason, you should be strategic about what you cache.  Not all database queries and rendered views will be expensive or frequently accessed, so pick the right ones.  You should profile your app, either by examining the logs or by using a profiling tool like New Relic, to understand where your application spends the majority of its time.  Chances are good that only a few complex queries and views will be slowing down your app, and you should start caching these expensive operations first.

The Impact of a Larger Memcache

A larger memcache can sometimes enable better performance and scalability.  But not always.  Some of our customers have benefited from a larger cache, while others haven’t.  Below we’ll explain how to determine if you need a larger cache and how a larger cache improves performance.

Do I need a larger cache?

Before considering increasing your memcache, you should understand how much memcache you’re using.  Memcache provides a stats function (in Rails it’s Rails.cache.stats) that returns the number of bytes and the number of objects in your cache.

You shouldn’t consider increasing your cache size if you aren’t already using most of your cache.  You’ll usually never see exactly 100% usage, though. [1]  So consider increasing your cache size if your usage is consistently above 90%.  Increasing your cache size with less usage won’t change performance at all.

What does a larger memcache do for me?

A larger cache will not affect latency for individual get and set operations.  Every operation in memcache is always completed in constant time.

However, a larger cache enables you to keep more keys in memory, which will increase your overall cache hit rate.  More database queries and rendered HTML will fit in your cache, which means that more page loads will be faster, too.  An increased cache will improve your app’s average page load time.  Consider the following diagram:

Memcache has a least recently used eviction algorithm, which keeps popular keys in the cache in favor of less popular keys.  Your average page load time will increase as you increase memcache because more and more of your long tail will be cached.

Code Examples

We have a number of real code examples, showing both how to connect to memcache and get/set keys.  Take a look:

What else?

This guide is, and always will be, a work in progress. We’ll continue to improve it. In the meantime, if there’s anything we can do to help, definitely let us know.

[1] Memcache cannot guarantee that your cache will be 100% used for two main reasons.  First. the sum of each value size, in bytes, may not equal the memcache limit.  And second, your cache is spread across multiple cache servers.  A cheap, less accurate, algorithm must be used to calculate cache limits in order to guarantee low latencies.