NEXT GENERATION HADOOP MAPREDUCE
Chris Douglas (Yahoo!)
Monday, March 7, 2011
ABSTRACT
The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization.
For more information, see:
developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
BIOGRAPHY
Chris Douglas has been a member of the Yahoo! Hadoop team since 2007. He has worked on teams focused on HDFS, performance and utilization, and MapReduce. [He is probably better known as the user login "chrisdo"; the one that shows up in the compiled by line in Apache Hadoop 0.20.2.]
