Optimizing and Enhancing Cloud MapReduce for Processing Stream Data Using Pipelining (CAPS-CMR)


Team

  • Rutvik Karve
  • Devendra Dahiphale
  • Amit Chhajer

  • Mentor

    Chirag Jog and Amod Jaltade


    Year:

    2011-2012


    Synopsis

    Cloud MapReduce (CMR) is a framework for processing large data sets of batch data in cloud. The Map and Reduce phases run sequentially, one after another. This leads to:
    1. Compulsory batch processing
    2. No parallelization of the map and reduce phases
    3. Increased delays.

    The current implementation is not suited for processing streaming data. We propose a novel architecture to support streaming data as input using pipelining between the Map and Reduce phases in CMR, ensuring that the output of the Map phase is made available to the Reduce phase as soon as it is produced. This 'Pipelined MapReduce' approach leads to increased parallelism between the Map and Reduce phases, thereby
    1. Supporting streaming data as input
    2. Reducing processing time by > 33%
    3. Enabling the user to take 'snapshots' of the approximate output generated in a stipulated time frame
    4. Supporting cascaded MapReduce jobs.
    This cloud implementation is light-weight and inherently scalable.


    Publications/Talks

  • Optimizing Cloud MapReduce for Processing Stream Data Using Pipelining
  • An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications
  • Presented at WIVCC 2012 at IIT Bombay

  • Project Achievements

    Project Competitions

  • Grabbed 2nd prize at IIT Kanpur
  • Won 2nd prize in Concepts at Pune Institute of Computer Technology
  • 1st Prize at Army Institute of Technology
  • Won 1st Prize at Singhad College of Engineering
  • Won 1st Prize at Modern College of Engineering
  • 1st Prize at Singhad Institute of Technology
  • Achieved 1st prize at Vishwakarma Institute of Technology
  • Made to 2nd spot at Maharashtra Institute of Technology
  • 2nd prize at AISSMS, Pune
  • Won 2nd prize at Bharati Vidyapeeth
  • Paper Presentation

  • First prize at Pratibha at Pune Institute of Computer Technology
  • 2nd Prize at Vishwakarma Institute of Technology