Prashant Sharma

Hadoop

An Elephant can't jump. But can carry heavy load.

A 20 page introduction to hadoop and friends.

Prashant Sharma

Table of Contents

1. INTRODUCTION...........................................................................................................................................5

1.1 What is distributed computing?....................................................................................................................5

1.2 What is hadoop? (Name of a toy elephant actually).....................................................................................5

1.3 How does Hadoop eliminate complexities?...................................................................................................5

1.4 What is map-reduce?.....................................................................................................................................6

1.5 What is HDFS?..............................................................................................................................................6

1.6 What is Namenode?.......................................................................................................................................6

1.7 What is a datanode?......................................................................................................................................6

1.8 What is a Jobtracker and tasktracker?.........................................................................................................7

2. HOW MAP-REDUCE WORK?......................................................................................................................7

..............................................................................................................................................................................7

...............................................................................................................................................................................7

2.1 Introduction...................................................................................................................................................8

2.2 Map-reduce is the answer.............................................................................................................................8

2.3 An example program which puts inverted index in action using Hadoop 0.20.203 API...........................8

2.4 How Hadoop runs Map-reduce?..................................................................................................................11

2.4.1 Submit Job..................................................................................................................................................11

2.4.2 Job Initialization.........................................................................................................................................11

2.4.3 Task Assignment........................................................................................................................................12

2.4.4 Task Execution...........................................................................................................................................12

3. HADOOP STREAMING..............................................................................................................................12

3.1 A simple example run..................................................................................................................................13

3.2 How it works?..............................................................................................................................................13

3.3 Features........................................................................................................................................................13

4. HADOOP DISTRIBUTED FILE SYSTEM ................................................................................................13

4.1 Introduction.................................................................................................................................................13

4.2 What HDFS can not do?..............................................................................................................................14

4.3 Anatomy of HDFS !.....................................................................................................................................14

4.3.1 Filesystem Metadata...................................................................................................................................14

4.3.2 Anatomy of write........................................................................................................................................15

4.3.3 Anatomy of a read......................................................................................................................................15

4.4 Accessibility..................................................................................................................................................15

4.4.1 DFS shell....................................................................................................................................................15

4.4.2 DFS Admin.................................................................................................................................................15

4.4.3 Browser Interface........................................................................................................................................16

4.4.4 Mountable HDFS........................................................................................................................................16

5. SERIALIZATION.........................................................................................................................................16

5.1 Introduction.................................................................................................................................................16

5.2 Write your own composite writable............................................................................................................17