INTRODUCTION FOR MAP REDUCE


Hadoop had two main indispensable components HDFS and Map Reduce. About Hadoop eco system and HDFS, we already discussed in another blog with detailed explanation if you are not gone through, check out those blogs (hdfsblog-coming soon, hadoopecosystem-coming soon). It will  be helpful to understand more about Hadoop Technology.

    HDFS is the distributed file system for Hadoop and map reduce is distributed processing engine where can we do cleansing, transformation etc….

TechnicalTerms You need to Know Before stepping into Big Data Processing

Map Reduce does : 

 It is basically a distributed processing engine with batch processing technique. Mostly used for data processing.

Programming language map reduce supports:

Java, Python, C++, Ruby, Perl are some major languages supported by map reduce.
by default, java. java will give better performance than any other language in map reduce.

Advantages of map reduce:

    it is parallel processing ( distributed processing). If you use Hadoop/map reduce frame work, it will take care below functions. Else you need to write code for and manage those things.

  • cluster management
  • resource allocation
  • cluster monitoring
  • scheduling
  • Execution
  • Speculative Execution{L}

Abstraction of Map Reduce:

Applications run on top of map reduce. 
  • Hive - SQL (facebook),
  • Pig - pig Latin (yahoo)
  • Sqoop - data migration
  • oozie - schedule
  • Mahout - Machine Learning Library
  • flume - monitoring etc...
NOTE : Hive and Pig is not a alternative for Map Reduce. Instead, those are alternative for java which is used to communicate with Map reduce by developers. Hive helps us to use SQL instead java and Pig helps us to use Pig Latin script instead java on in map reduce job writing.

Alternative for map reduce:

Open source : Spark, Flink, Dask, Ray

Paid : Amazon redshift, azure data analytics services, bigquery etc…


Daemons of map reduce : 

job tracker and task tracker - Hadoop V1
Resource Manager and Node Manager - Hadoop V2

Who decides the mapper count?
based on no of blocks of data = no of mappers.
it is changeable by changing the configuration like run 2 mapper for each block ( if we had more RAM or for two block 1 map due to RAM constrain) 
no of node != no of map

map and reduce will took only data in Key value pair format as a input and output whether the format can be a custom or built-in. 
below are important built-in formats
  1.  text input and text output format
    ex, 
  2. key value input and output format

hearderheader2


serialize and deserialize 

Comments

Post a Comment

Popular posts from this blog

Technical Terms You need to Know Before stepping into Big Data Processing

  List Of Technical Terms  Cluster :  group of computers or servers. Parallel processing :  group of computers or servers used to do same task.   Like Multiple developers hired to develop same application by splitting their tasks. Similar to that if we have 320 MB data to process and have 5 machines, Hadoop will split it into five 64 MB chunks then five parallelly processes by 5 machines. It   will reduce 4 times processing time. Concurrency :  Single thread or process will handle multiple works or tasks in a time period. Example : Technically – a process got task to retrieve 100 accounts data through API. API will accept only one account per request, process should make 100 API call to complete that task.   Normally, a process will call API for account1 and wait ideally until get response. In concurrent scenario, meanwhile it waiting for Api response for Account1, it will send requests for other accounts also to API. Suppose, process t...
Read more