Hadoop would not be real without this paper. MapReduce is the most famous and still most used processing paradigm for big data. It is not suitable for everything and there are several improvements (Dryad, Spark, …) but Google, Facebook, Twitter and many other has million rows of code deployed into their systems.


google_logoTitle: MapReduce: Simplified Data Processing on Large Clusters (PDF), December 2004
Authors: Jeffrey Dean and Sanjay Ghemawat

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.


Check out the list of interesting papers and projects (Github).

Last week I started to play with my RaspberryPi (thanks @dani_viga). It seems simple and funny. I’d like to run an Hadoop cluster like LinkedIn or others geeks but first of all I need to setup the system.

My RaspberryPi kit includes the board, an USB-microUSB cable (as power supply), an SD card (16 GB) and an ethernet cable.

raspberry_kit

pidora_logoAs Linux distribution I choose Pidora because I like CentOS and this a Fedora/RedHat-like distribution. Base image can be downloaded from official page. Then you have to burn the image on the SD card. It seems that RaspberryPi can boot only from SD card. I can’t find any guide to boot it from USB key without SD card. Anyway is possible to use it as main drive. To burn it on SD on Mac o Linux you can use dd

dd if=/path/to/image/pidora-18-r1c/pidora-18-r1c.img of=/dev/disk3 bs=1m

First parameter is the path of the downloaded image, the second one is the endpoint of the SD drive (you can find it using diskutil list).

After this I simply plugged USB to my Mac and ethernet to a connection. After a dozen of seconds lights start blinking!

raspberrypi_running

System is already configured and on port 22 runs SSHd. If you try to connect to the IP using root as username and raspberrypi as password the RPi answers!

raspberrypi_shell

Next week I’ll try to install a LEMP stack on it. FUN! 😀