Redis is a RAM based key-value store. RAM is expensive. Hard disks (even SSD) are slow. It’s the truth, we know.
A few months ago people tried to use Redis instead of MySQL (or similar SQL DBs) as main datastore. When you do, is easy to clash against the memory limit. As we learned from operative systems, first solution is to use your disk space to “enlarge” your RAM. Redis versions from 2.0 to 2.6 offer a Virtual Memory implementation.
Virtual Memory seems to be really useful in many cases. If only a small part of your keys get the vast majority of accesses you can efficiently keep only that part of keys into RAM and leave the remaining part on disk.
To enable Virtual Memory you can switch it on using
vm-enabled yes and set the memory limit using
vm-max-memory. Additionally you can fine tune the configuration using
vm-page-size for swap file and
vm-max-threads for concurrency.
Anyway since version 2.4 Virtual Memory is deprecated. This is the official note about it:
Redis VM is now deprecated. Redis 2.4 will be the latest Redis version featuring Virtual Memory (but it also warns you that Virtual Memory usage is discouraged). We found that using VM has several disadvantages and problems. In the future of Redis we want to simply provide the best in-memory database (but persistent on disk as usual) ever, without considering at least for now the support for databases bigger than RAM. Our future efforts are focused into providing scripting, cluster, and better persistence.
The alternative is Redis cluster. It will be a “distributed and fault tolerant implementation of a subset of the features available in the Redis stand alone server”. At the moment is a work in progress. There are some client-side implementation (for Node.js, for Ruby and more) but not yet an official, standalone version.
Virtual memory deprecation and Redis cluster long developing time make me think about a simple idea:
Redis is not ready to be the main datastore for a huge dataset, not yet.
More about Redis scaling
TL;DR: the outcome of this test was expected and Redis is an in-memory system 🙂