Virtual memory and Redis cluster: to scale Redis is hard

Redis is a RAM based key-value store. RAM is expensive. Hard disks (even SSD) are slow. It’s the truth, we know.

A few months ago people tried to use Redis instead of MySQL (or similar SQL DBs) as main datastore. When you do, is easy to clash against the memory limit. As we learned from operative systems, first solution is to use your disk space to “enlarge” your RAM. Redis versions from 2.0 to 2.6 offer a Virtual Memory implementation.

Virtual Memory seems to be really useful in many cases. If only a small part of your keys get the vast majority of accesses you can efficiently keep only that part of keys into RAM and leave the remaining part on disk.

To enable Virtual Memory you can switch it on using vm-enabled yes and set the memory limit using vm-max-memory. Additionally you can fine tune the configuration using vm-pages and vm-page-size for swap file and vm-max-threads for concurrency.

Anyway since version 2.4 Virtual Memory is deprecated. This is the official note about it:

Redis VM is now deprecated. Redis 2.4 will be the latest Redis version featuring Virtual Memory (but it also warns you that Virtual Memory usage is discouraged). We found that using VM has several disadvantages and problems. In the future of Redis we want to simply provide the best in-memory database (but persistent on disk as usual) ever, without considering at least for now the support for databases bigger than RAM. Our future efforts are focused into providing scripting, cluster, and better persistence.

The alternative is Redis cluster. It will be a “distributed and fault tolerant implementation of a subset of the features available in the Redis stand alone server”. At the moment is a work in progress. There are some client-side implementation (for Node.jsfor Ruby and more) but not yet an official, standalone version.

Virtual memory deprecation and Redis cluster long developing time make me think about a simple idea:

Redis is not ready to be the main datastore for a huge dataset, not yet. 

More about Redis scaling

[2013-03-09 UPDATE] @olinicola advises me a post by @antirez about to use Redis in memory and to swap on SSD. His response is the same:

TL;DR: the outcome of this test was expected and Redis is an in-memory system 🙂

Performance driven data modeling using MongoDB – Part 1

This week my problem was to modelize a semi-relational structure. We decided to use MongoDB because (someone says) is fast, scalable and schema-less. Unfortunately I’m not a good MongoDB designer yet. Data modeling was mostly easy because I can copy the relational part of the schema. The biggest data modeling problem is about m-to-m relations. How to decide if embed m-to-m relations keys into documents or not? To make the right choice I decided to test different design solutions.

Foreign keys emdedded:

class A
  include Mongoid::Document
  field :name, type: String
  has_and_belongs_to_many :bs
end

class B
  include Mongoid::Document
  field :name, type: String
  has_and_belongs_to_many :as
end

def direct(small, large)
  small.times do |i|
    a = A.new
    a.name = "A#{i}"
    large.times do |j|
      b = B.create(name: "B#{j}")
      a.bs << b
    end
    a.save
  end
end

Foreign keys into an external document:

class C
  include Mongoid::Document
  field :name, type: String
  has_many :rels
end

class D
  include Mongoid::Document
  field :name, type: String
  has_many :rels
end

class Rel
  include Mongoid::Document
  belongs_to :c
  belongs_to :d
end

def with_rel(small, large)
  small.times do |i|
    c = C.new
    c.name = "C#{i}"
    large.times do |j|
      d = D.create(name: "D#{j}")
      Rel.create(c: c, d: d)
    end
  end
end

I tested insert time for a database with 10 objects related to a growing number of other objects each iteration (from 100 to 5000).

def measure(message, &block)
  cleanup
  start = Time.now.to_f
  yield
  finish = (Time.now.to_f - start).to_f
  puts "#{message}: #{"%0.3f" % finish}"
end

(1..50).each do |e|
  measure "10 A embeds #{e*100} B each one" do
    direct(10, e*100)
  end
  measure "10 A linked to #{e*100} B with extenal relation" do
    with_rel(10, e*100)
  end
end

Results are really interesting:

Number of relation for each element Insert time embedding relation key Insert time with external relation
100 0.693 1.021
200 1.435 2.006
300 1.959 2.720
400 2.711 3.587
500 3.477 4.531
600 4.295 5.414
700 5.106 6.369
800 5.985 7.305
900 6.941 8.221
1000 7.822 8.970
1200 12.350 13.946
1400 14.820 15.532
1600 15.806 17.344
1800 18.722 18.372
2000 21.552 20.732
3000 36.151 29.818
4000 56.060 38.154
5000 82.996 47.658

As you can see when number of embedded relation keys go over 2000, the time grow geometrically.

I know, this is not a real case test so we can’t say that using embedded relation is worse than using external. Anyway is really interesting observe that limits are always the same in both SQL and NoSQL world: when you hit a memory limit and need to go to disk, performances degrade.

In coming post I’m going to analyze reading performances.

Persistence in the Amazon AWS Cloud

I’m developing a new project which require a data structure not yet well defined. We are evaluating different solutions for persistence and Amazon AWS is one of the partners we are considering. I’m trying to recap solutions which it offers.

Amazon Relational Database Service (RDS)

Relational Database similar to MySQL and PostgreSQL. It offers 3 different engines (with different costs) and each one should be fully compatible with the protocol of the corresponding DBMS: Oracle, MySQL and Microsoft SQL Server.

You can use it with ActiveRecord (with MySQL adapter) on Rails or Sinatra easily. Simply replace you database.yml with given parameters:

production:
  adapter: mysql2
  host: myprojectname.somestuff.amazonaws.com
  database: myprojectname
  username: myusername
  password: mypass

Amazon DynamoDB

Key/Value Store similar to Riak and Cassandra. It is still in beta but Amazon released a paper (PDF) about its structure a few year ago which inspire many other products.

You can access it using Ruby and aws-sdk gem. I’m not an expert but this code should works for basic interaction (not tested yet).

require "aws"

# set connection parameters
AWS.config(
  access_key_id: ENV["AWS_KEY"],
  secret_access_key: ENV["AWS_SECRET"]
)

# open connection to DB
DB = AWS::DynamoDB.new

# create a table
TABLES["your_table_name"] = DB.tables["your_table_name"].load_schema
  rescue AWS::DynamoDB::Errors::ResourceNotFoundException
    table = DB.tables.create("your_table_name", 10, 5, schema)
    # it takes time to be created
    sleep 1 while table.status == :creating
    TABLES["your_table_name"] = table.load_schema
  end
end

After that you can interact with table:

# Create a new element
record = TABLES["your_table_name"].items.create(id: "andrea-mostosi")
record.attributes.add(name: ["Andrea"])
record.attributes.add(surname: ["Mostosi"])

# Search for value "andrea-mostosi" inside table
TABLES["your_table_name"].items.query(
  hash_value: "andrea-mostosi",
)

Amazon Redshift

Relational DBMS based on PostgreSQL structured for a petabyte-scale amount of data (for data-warehousing). It was released to public in the last days and SDK isn’t well documented yet. Seems to be very interesting for big-data processing on a relational structure.

Amazon ElastiCache

In-RAM caching system based on Memcached protocol. It should be used to cache any kind of object like Memcached. Is different (and worse IMHO) than Redis because doesn’t offer persistence. I prefer a different kind of caching but may be a good choice if your application already use Memcached.

Amazon SimpleDB

RESTFul Key/Value Store using only strings as data types. You can use it with any REST ORM like ActiveResource, dm-rest-adapter or, my favorite, Her (read previous article). If you prefer you can use with any HTTP client like Faraday or HTTParty.

[UPDATE 2013-02-19] SimpleDB isn’t listed into “Database” menu anymore and it seems no longer available for activation.

Other DBMS on markerplace

Many companies offer support to theirs software deployed on EC2 instance. Engines include MongoDB, CouchDB, MySQL, PostgreSQL, Couchbase Server, DB2, Riak, Memcache and Redis.

Sources

Intersect a SET with a ZSET

Redis‘s SET and ZSET (sorted sets) are a really powerful structure. The only limits are about set operation you can perform. Using Redis you can’t obtain the intersection (or the union) between two sorted set or between a SET and a ZSET. You can use SINTER to intersect a group of SET or SUNION for union. Unfortunately there is no direct way for ZSET.

In our use case, we had to intersect a ZSET (a sorted rank) and a SET (a group of categorized items) to find the rank of the element inside selected category.

After a successful search on Google I found a way on StackOverflow (view below link): use ZINTERSTORE. It’s really simple: act like SINTER but store results into a new ZSET. It has a quite expensive memory footprint but is ok if you frequently reuse the result (is like a cache and you can set expire time using EXPIRE).

Source
http://stackoverflow.com/questions/10500695/redis-how-to-intersect-a-normal-set-with-a-sorted-set

Delete a huge amount of files without killing your server

Many high-traffic web applications take advantages from caching systems. HTML cache is easy and powerful. IMHO best solution is serving it using something like nginx and Varnish. Many people use custom solutions (for example a WordPress plugin) which produce an HTML snapshot of the page and save it to disk.

If you website il quite large in a flash you get a huge amount of small files stored on your disk. Clean cache is not easy as you would hope.

First solution is to use:

rm -Rf [path]

But this is dangerous because IO wait is terrible (even with SSD) and load average of you machine rise up to 50 in a minute.

find can help you:

find [path] -type f -print -delete

Unfortunately, after a while, load average rise again. Rising is slower but is still dangerous.

Solution is to use ionice. It enable you to limit priority of your process and avoid load average rise.

ionice -c 3 find [path] -type f -print -delete

Thanks to @dani_viga for the tips! He saves my day 🙂

More about ionice