Paper: MapReduce: Simplified Data Processing on Large Clusters

Hadoop would not be real without this paper. MapReduce is the most famous and still most used processing paradigm for big data. It is not suitable for everything and there are several improvements (Dryad, Spark, …) but Google, Facebook, Twitter and many other has million rows of code deployed into their systems.

google_logoTitle: MapReduce: Simplified Data Processing on Large Clusters (PDF), December 2004
Authors: Jeffrey Dean and Sanjay Ghemawat

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.

Check out the list of interesting papers and projects (Github).

  • Gabe

    Thanks for such an informative blog! MapReduce is the heart of hadoop.
    It is this programming paradigm that allows for massive scalability
    across hundreds or thousands of servers in a Hadoop cluster. The MapRedue concept is fairly simple to understand for those who are familiar with clustered scale-out data processing solutions.The term MapReduce actually refers to two separate and distinct tasks
    that Hadoop programs perform. The first is the map job, which takes a
    set of data and converts it into another set of data, where individual
    elements are broken down into tuples (key/value pairs). The reduce job
    takes the output from a map as input and combines those data tuples into
    a smaller set of tuples. As the sequence of the name MapReduce implies,
    the reduce job is always performed after the map job. More at