Processing Data At Scale With MapReduce
In the current market landscape, organizations must engage in data-driven decision-making to maintain competitiveness and foster innovation. As a result, an immense amount of data is collected on a daily basis.
Although the challenge of data persistence has largely been resolved, thanks to the widespread availability and affordability of cloud storage, modern organizations continue to grapple with the efficient and effective processing of massive amounts of data.
Over the past few decades, numerous Programming models have emerged to address the challenge of processing big data at scale. Undoubtedly, MapReduce stands out as one of the most popular and effective approaches.
What is MapReduce
MapReduce is a distributed programming framework originally developed at Google by Jeffrey Dean and Sanjay Ghemawat, back in 2004 and was inspired by fundamental concepts of functional programming. Their proposal invloved a parallel data processing model consisting of two steps; map and reduce.
In simple terms, map step invovles the division of the original data into small chunks such that transformation logic can be applied to individual data blocks. Data processing can therefore be applied in parallel across the created chunks and finally, the reduce step will then aggregate/consolidate the processed blocks and return the end result back to the caller.
How does MapReduce algorithm work
Even though MapReduce algorithm has been widely known as a two-step process, it actually invovles three distinct stages.
1. Map: In this very first step, the data is split into smaller chunks and distributed across multiple nodes that are usually part of a cluster of processing units. Each chunk created is then assigned to a mapper. The input to the mapper is a set of
As an example, let's consider the following example where the input text is first split across three mappers and the input is provided in the form of key-value pairs.
2. Shuffling: Now in this step, the algorithm will shuffle the data such that reocrds with the same key are allocated to the same worker node. This is usually the most expensive operation that is performed throughout the lifecycle of a MapReduce process.
3. Reduce: In this final step, each reducer will accept as an input the output of the corresponding mapper, which is in the form of a
MapReduce and Hadoop
MapReduce is part of the Apache Hadoop framework that is used to access data stored in Hadoop Distributed File System (HDFS). Hadoop consists of four basic modules:
- Hadoop Distributed File System (HDFS): This ia a distributed file system that can store large datasets in a fault-tolerant fashion
- Yet Another Resource Negotiation (YARN): This is the node manager that monitors cluster and resources. It also acts as the scheduler of jobs
- MapReduce
- Hadoop Common: This is a module that provides commonly used Java libraries
Previously, we mentioned how mappers and reducers run on individual nodes within a cluster of computers. In fact, these worker nodes are part of the Hadoop framework that decides the amount of mappers required to be used in each case, depending on the volume of the input size.
Be design, Hadoop offers fault-tolerance. In the event of a node failure, Hadoop will rerun the task on another mapper node and generate the output required.
Final Thoughts
MapReduce has been a groundbreaking concept in distributed computing, empowering numerous organizations to process vast volumes of data and extract valuable insights.
Familiarizing oneself with this concept is crucial, particularly when utilizing technologies like Spark that leverage the MapReduce framework.
